Title: Thresholds for sensitive optimality and Blackwell optimality in stochastic games
Venue: NeurIPS 2025
Accepted: yes
Keywords: Stochastic games

Blackwell optimality

mean-payoff optimality
Abstract: We investigate refinements of the mean-payoff criterion in two-player zero-sum perfect-information stochastic games. A strategy is *Blackwell optimal* if it is optimal in the discounted game for all discount factors sufficiently close to $1$. The notion of *$d$-sensitive optimality* interpolates between mean-payoff optimality (corresponding to the case $d=-1$) and Blackwell optimality ($d=\infty$). The *Blackwell threshold* $\alpha_{\sf bw} \in [0,1[$ is the discount factor above which all optimal strategies in the discounted game are guaranteed to be Blackwell optimal. The *$d$-sensitive threshold* $\alpha_{\sf d} \in [0,1[$ is defined analogously. Bounding $\alpha_{\sf bw}$ and $\alpha_{\sf d}$ are fundamental problems in algorithmic game theory, since these thresholds control the complexity for computing Blackwell and $d$-sensitive optimal strategies, by reduction to discounted games which can be solved in $O\left((1-\alpha)^{-1}\right)$ iterations. We provide the first bounds on the $d$-sensitive threshold $\alpha_{\sf d}$ beyond the case $d=-1$, and we establish improved bounds for the Blackwell threshold $\alpha_{\sf bw}$. This is achieved by leveraging separation bounds on algebraic numbers, relying on Lagrange bounds and more advanced techniques based on Mahler measures and multiplicity theorems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to find d-sensitive and Blackwell optimal strategies in two-player zero-sum perfect-information stochastic games. It provides new bounds on the d-sensitive threshold ( $\\alpha_d$ ) and the Blackwell threshold $\\left(\\alpha_{B w}\\right)$. These thresholds are important because they determine the complexity of finding these strategies by solving standard discounted games.\n\nThe paper's main contributions are the first bounds for $\\alpha_d$ when $d>-1$ and improved bounds for $\\alpha_{B w}$ . The authors use three techniques from algebraic number theory to derive their results: Lagrange bounds, Mahler measures, and root multiplicity bounds. They present results for both deterministic and stochastic games, showing which technique is best for different parameter settings.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The paper provides the first explicit bounds for the d-sensitive threshold $\\alpha_d$ (for $d>-1$ ), which is a new contribution to the field.\n2. The use of advanced tools from algebraic number theory (Lagrange bounds, Mahler measures, multiplicity theorems) is innovative and leads to stronger results than prior work.\n3. Clear Improvements: The new bounds on the Blackwell threshold $\\alpha_{B w}$ improves over existing results, particularly in their dependence on the number of states, $n$.\n4. The paper is well-organized. It clearly presents the problem, the methods, and the results. The summary tables are helpful for comparing the different bounds.\n\n### Weaknesses\n1. The improved bounds still lead to algorithms with superpolynomial complexity. Therefore, the work does not yet yield efficient, practical algorithms for finding Blackwell optimal strategies.\n2. The bound on $\\alpha_d$ in the general stochastic case requires a unichain assumption, which limits the result's applicability.\n3. The work is entirely theoretical, with no experiments to indicate how tight the bounds are in practice.\n\nQUESTIONS:\n### Questions\n1. The multiplicity-based method fails for stochastic games (Remark 4.5). Can you provide more intuition on why the analysis doesn't carry over from the deterministic case?\n2. The paper offers three different bounds. Is this necessary due to the nature of the problem, or could a unified theory combine these techniques for a single, stronger bound?\n3. What are the main obstacles to extending the $\\alpha_d$ bound to general multi-chain games without the unichain assumption?\n4. Does your analysis offer any insight into what a \"hard\" game instance (i.e., one requiring a very high discount factor) might look like?\n\nLIMITATIONS:\nThe paper correctly identifies its main limitations.\n1. The bounds do not yet lead to polynomial-time algorithms.\n2. The $\\alpha_d$ bound for stochastic games is limited to the unichain case.\n3. The analysis is entirely theoretical, lacking empirical validation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study two‑player zero‑sum perfect‑information stochastic games and ask how close the discount factor \n$\\alpha$ needs to be to $1$ before the optimal discounted strategies are already optimal for more farsighted criteria.\nIn particular, they consider the notion of Blackwell optimality – strategies that remain optimal for every $\\alpha$ sufficiently close to $1$, and an intermediate notion called d‑sensitive optimality that interpolates between mean‑payoff (for $d=-1$) and Blackwell optimality (for $d$ approachng infinity).\nThe authors use heavily algebraic tool such as Lagrange separation bound and Mahler‑measure bounds to derive new thresholds on $\\alpha$ that yield substantial quantitative improvements compared to the bounds in the prior works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths. \n1. The paper uses sophisticated algebraic methods such as Lagrange bounds and Mahler‑measure techniques. \n2. The explicit bounds improve previous results by up to a factor of $Ω(n)$, where $n$ is the size of the game, for the Blackwell optimality threshold, and, for the first time, bound the threshold for $d$-sensitive optimality for general values of $d$. This tightens the best-known estimates for the complexities of computing Blackwell/$d$-sensitive optimal strategies.\n3. The paper is technical in nature: even with the connection between Blackwell threshold and classic polynomial root separation results, to apply these results, non-trivial analysis is needed to control the coefficients of the polynomial showing up in differences of discounted value functions. \n\nWeaknesses.\n1. The paper does not construct explicit games that witness a gap between their upper bounds and information‑theoretic lower bounds. It will be nice if the authors can provide even some sub-optimal game construction to show how far we are from the optimal bounds for these thresholds.\n2. For stochastic game, the need for unichain structure limits applicability; relaxing this assumption is an open direction acknowledged by the authors.\n\nQUESTIONS:\n1. Does author have some guess on whether the fact that some bounds are exponentially close to $1$ are in inherent or limitations of the analysis.\n2. Adding game instances where thresholds are close to $1$ would help calibrate how tight the theory is.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper obtains bounds on the d-sensitive ($\alpha_d$) and Blackwell thresholds ($\alpha_{BW}$) for stochastic games. The bounds improve upon existing results. Deterministic and general stochastic games are considered.

STRENGTHS AND WEAKNESSES:
Strengths :

1. A thorough analysis of the different regimes of the stochastic games parameter is provided.
2. A variety of techniques are used to show the bounds.


Weaknesses:

1. The comparison tables can be improved - the results of MDP should be presented separately from those of stochastic games. 
2. Since the MDPs may be such that the reward function depends on both the current and next states maybe a clarification for the definition of MDPs considered can be detailed, along with its affects on the results on $\alpha_{bw}$.

QUESTIONS:
1. The clarification regarding MDPs mentioned in the weaknesses section can be clarified.
2. Is there an assumption of smoothness on the rewards function in either the MDP or the stochastic games? Or is it not needed due to the finite state space?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies refinements of the mean-payoff criterion in two-player zero-sum perfect-information stochastic games, focusing on Blackwell optimality and its generalization through d-sensitive optimality. The authors introduce new bounds on the d-sensitive threshold \\alpha_d​ that interpolates between mean-payoff and Blackwell optimality. The paper employs the methods and toolkits from separation bounds on algebraic numbers and shows their results can improve bounds for the Blackwell threshold. These thresholds are critical for understanding the algorithmic complexity of computing optimal strategies, as solving discounted games typically requires O(1/(1−α)) iterations.\n\nSTRENGTHS AND WEAKNESSES:\nStrength \n\n- The paper did a reasonably good job introducing to me the concepts of sensitive optimality and Blackwell\nOptimality. The techniques of separation bounds, algebraic roots, Mahler measures, and multiplicity theorems from algebraic number theory are completely new to me, and I am surprised at how it can be used in a learning problem. There seems to be many new ideas in the analysis of this problem (especially over another neurips paper [GCP23]), though The technical analysis is well beyond my comfort zone, so I cannot make a good judgement call.\n- The problem is well-motivated to generalize this fundamental concept of Blackwell optimality to sensitive optimality. The cited statement of “one of the pressing questions in reinforcement learning” is a good showcase. \n-  The paper clearly acknowledged the limitations of their results.\n\nWeakness\n\n- I am not sure if the problems studied by this paper fits into the theme of Neurips, or its result might be interesting to a general audience at this conference. \n- A notation table can help us quickly parse through your technical results.\n- Typos: the left/right square brackets throughout this paper are sometimes not matched\n\nQUESTIONS:\n- The results seem to be more technically heavy than a usual Neurips paper, would it be more suitable for a more theoretical audience to review such as COLT or ALT?\n- Perhaps I am not the right reviewer for this paper, but I wonder what could be the insights and takeaways for practitioners in their design and analysis of real-world RL algorithms?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}