Title: \(\varepsilon\)-Optimally Solving Two-Player Zero-Sum POSGs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Zero-Sum Partially Observable Stochastic Games

Transition-Independent Stochastic Games

Planning

Linear Programming
Abstract: We present a novel framework for \(\varepsilon\)-optimally solving two-player zero-sum partially observable stochastic games (zs-POSGs). These games pose a major challenge due to the absence of a principled connection with dynamic programming (DP) techniques developed for two-player zero-sum stochastic games (zs-SGs). Prior attempts at transferring solution methods have lacked a lossless reduction—defined here as a transformation that preserves value functions, equilibrium strategies, and optimality structure—thereby limiting generalisation to ad hoc algorithms. This work introduces the first lossless reduction from zs-POSGs to transition-independent zs-SGs, enabling the principled application of a broad class of DP-based methods. We show empirically that point-based value iteration (PBVI) algorithms, applied via this reduction, produce \(\varepsilon\)-optimal strategies across a range of benchmark domains, consistently matching or outperforming existing state-of-the-art methods. Our results open a systematic pathway for algorithmic and theoretical transfer from SGs to partially observable settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to solve two-player zero-sum partially observable stochastic games (zs-POSGs) up to $\\varepsilon$-optimality. It introduces the first principled and lossless reduction from zs-POSGs to transition-independent zero-sum stochastic games (zs-SGs), marking the first such reduction to adversarial games. The core technical contribution is a novel hierarchy-of-planners framework that enables this reduction. This, in turn, allows the use of a broad class of dynamic programming-based methods, leading to both theoretical guarantees and scalable planning algorithms. The authors also adapt point-based value iteration methods and evaluate their approach across several benchmark domains, including matching pennies and pursuit-evasion, demonstrating performance that matches or surpasses existing state-of-the-art methods such as HSVI and CFR+.\n\nSTRENGTHS AND WEAKNESSES:\nThe main strength of this paper lies in the hierarchy-of-planners approach. The hierarchy comprises four planners:\n\n* **Focal planner**: Knows only the focal policy and nothing else.\n* **Uninformed planner**: Has access to the opponent's policy and the focal planner's data.\n* **Informed planner**: Knows the public signals and the uninformed planner's data.\n* **Marginal planner**: Has access to the opponent's history and the informed planner's data.\n\nWhile only the focal planner is executed, the other planners provide important theoretical insights into the structure of the value functions. This hierarchy also reveals new structural properties of zs-POSGs, such as optimality equations, strategy selection rules, and the uniform continuity of value functions. The proposed reduction to transition-independent zs-SGs preserves key elements such as the value, equilibrium structure, and information constraints.\n\nThe hierarchy of planners also enables practical computation. The uniform continuity of the value function, established at the level of the marginal planner, allows for the use of point-based representations. This, in turn, makes it possible to apply point-based value iteration methods. A key component of the approach is the definition of action-value functions over the uninformed occupancy states, from which greedy decision rules are extracted via linear programming.\n\nAlthough I did not identify any major weaknesses, I believe the presentation could be improved by including some of the diagrams from the supplementary material in the main body. Currently, the paper's main text feels somewhat dense due to the heavy concentration of technical content.\n\nQUESTIONS:\nAre there any decentralized algorithms for zero-sum stochastic games that could potentially be applied to your reduced game? Do you think such an approach might improve the performance, particularly in terms of its dependence on the planning horizon?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method to simplify a zero-sum two player Stackelberg game by reformulating the primal construct of the game to allow for efficient representations and approximation of the players' value function. The authors provide the conditions and definitions for when this simplification is possible. Provide the methodology for this reduction, and theoretical guarantees regarding the algorithms performance, accompanied by empirical results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors use rigorous methods to convert a Stackelberg zs-POMDP into a more tractable representation. This simplification could have strong ramifications in multi-agent optimization and game theory.\n\nWeaknesses:\n- As a zero-sum game, the mini-max von Neumann theorem applies, it is not overly challenging to show that Theorem 3.4, and subsequently 4.1, holds. This is merely an extension from normal form to extensive form games, and the minimax theorem would still apply to sequential games.\n- In the empirical experiments, there should be standard error measurements on the empirical estimates.\n- This paper in general needs a bit more polish, the empirical results in Table 1 contains some unconvincing results (i.e. the new method poses no significant advantage compared to the baseline CFR.)\n- Other Stackerlberg game solver baselines could also be employed here, such as KKT [1] reformulations of the problem, and gradient descent methods [2]. It was not considered.\n\n[1] Allende, Gemayqzel Bouza and Georg Still (2013). “Solving bilevel programs with the KKTapproach”. In: Mathematical programming 138, pp. 309–332.\n[2] Liu, Risheng et al. (2021). “Towards gradient-based bilevel optimization with non-convex followers\nand beyond”. In: Advances in Neural Information Processing Systems 34, pp. 8662–8675.\n\nQUESTIONS:\n- Comparatively speaking, what is the more inefficient LP formualtion of the problem and what was the complexity reduction compared if someone were to brute-force the solution?\n- Where is the source-code to reproduce these experiments?\n- How come the CFR baseline is not presented in 3/4 plots?\n\nLIMITATIONS:\n- I don't see any major limitations of this work, my critique largely lies in the novelty of the methods. I do not believe the LP formulation is game-changing in terms of representing the Stackelberg ZS-POMDP efficiently. It provides a nice construct, good theory, but a more impactful research direction would be to look into kernel methods for non-lienar representations or dual-representations of the MIP/LP format, such that we can transform even non-linear problems into linear problems. Thus the technical challenge of the work is not to high, as I mentioned before, the von Neumann mini-max applies in this setting leading to simple reformulations, whereas in general sum games, a greater technical challenge exists.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors tackle the problem of computing approximate equilibria to 2p0s POSGs, and present a novel framework that is lossless, in the sense that it preserves value functions, equilibria, and information. Based on this reduction, they apply PBVI algorithms, to compute $\varepsilon$ optimal equilibria in a variety of games.

STRENGTHS AND WEAKNESSES:
The paper is well-written and tackles an important problem of equilibrium computation in 2p0s POSGs. Their lossless framework allows application of techniques in “vanilla” Markov games to the 2p0s POSGs (PBVI algorithms are the only ones that can be applied). 
In terms of weaknesses, given my background (or lack thereof), I am not well equipped to assess originality beyond the presentation of the authors. I find the math typesetting a bit odd and in some parts hard to read. It seems that many different typesetting styles are being combined, and personally, that, combined with the color subscripts, makes it harder to read rather than easier because it is distracting. The authors should consider shortening exposition in certain portions to make the graphs and tables in the experiment section larger. While the trends are clear in the graphs, and the figures in the table can be read, they are both quite small, the table especially.

QUESTIONS:
Is there a reason why in the experiments why you don’t compare to the annealing approach of Sokota et al.?

LIMITATIONS:
Yes, the authors have addressed limitations.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present a framework that introduces the first lossless reduction from zs-POSGs to transition-independent zs-SGs, enabling the principled application of a wide range of dynamic programming (DP) methods.\n\nThe paper begins by introducing the problem and detailing the limitations of existing reductions. The authors then reformulate zs-POSGs as transition-independent zs-SGs by decentralizing planning into per-player occupancy sets, thereby preserving value functions, equilibria, and information constraints.\n\nNext, they show how transition-independent zs-SGs can be solved via a hierarchy of planners, which enables more efficient value function decomposition and planning.\n\nFinally, the authors demonstrate how Point-Based Value Iteration (PBVI) can be applied over the reduced zs-SG and validate their theoretical results empirically on a suite of challenging benchmark domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The problem is very well motivated. I appreciate that the authors clearly articulate the desirable criterion of losslessness early on, which helps frame the technical goals precisely and underscores why prior approaches fell short.\n2. Although I am not deeply familiar with all related literature, the contributions appear to be technically solid and substantial. Sections 3, 4, and 5 each introduce significant and novel technical elements, including the construction of the reduction, the hierarchy of planners, and the uniform continuity results that underpin the approximation guarantees.\n3. Beyond the strong theoretical results, the authors thoughtfully address practical considerations around scalability. The proposed pruning strategies for point-based value iteration are well-motivated, and the experimental evaluation is convincing, showing consistent improvements in exploitability and runtime over prior baselines.\n\nWeakness:\n1. While the theoretical development is elegant, the presentation is quite dense. I think additional figures, diagrams, or illustrative examples would help make the key constructions (particularly the planner hierarchy and the occupancy set definitions) more accessible to readers who are less familiar with the technical background.\n2. One potential concern is memory usage. Because the approach relies on sampling and storing sets of marginal and informed occupancy states, I wonder whether the memory footprint grows significantly with the horizon and problem size, perhaps more so than some regret-minimization approaches. It would be helpful if the authors could clarify this point or provide comparative memory usage statistics.\n\nQUESTIONS:\n1. I am curious whether you have considered comparing your approach to recent deep reinforcement learning (deep RL) based methods for imperfect-information games, such as Neural Fictitious Self-Play or other policy optimization frameworks (e.g., actor-critic variants or regret minimization with function approximation). While I understand that these methods often lack the same theoretical guarantees and lossless reductions, they have shown practical success on some large-scale settings. Do you see your framework as complementary to these approaches, or would you expect it to outperform them in terms of exploitability or sample efficiency? It would be helpful to hear your perspective on how your method compares empirically or conceptually to deep RL-based baselines, and whether you envision integrating neural approximators into your framework in the future.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}