Title: Optimism Without Regularization: Constant Regret in Zero-Sum Games
Venue: NeurIPS 2025
Accepted: yes
Keywords: optimistic fictitious play

learning in games

zero-sum games
Abstract: This paper studies the *optimistic* variant of Fictitious Play for learning in two-player zero-sum games. While it is known that Optimistic FTRL -- a *regularized* algorithm  with a bounded stepsize parameter -- obtains constant regret in this setting, we show for the first time that similar, optimal rates are also achievable *without* regularization: we prove for two-strategy games that Optimistic Fictitious Play (using *any* tiebreaking rule) obtains only *constant regret*, providing surprising new evidence on the ability of *non*-no-regret algorithms for fast learning in games. Our proof technique leverages a geometric view of Optimistic Fictitious Play in the dual space of payoff vectors, where we show a certain energy function of the iterates remains bounded over time. Additionally, we also prove a regret *lower bound* of $\Omega(\sqrt{T})$ for *Alternating* Fictitious Play. In the unregularized regime, this separates the ability of optimism and alternation in achieving $o(\sqrt{T})$ regret.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper demonstrates that constant regret can be achieved with Optimistic Fictitious Play (OFP) without regularisation in two-player zero-sum games, and the key contribution is identifying that constant regret is possible by associating primal variables with enumerated game states, represented by vertices of the joint simplex. The authors leverage geometric insights and a Lyapunov-esque energy-function framework to establish a constant (joint) regret bound. Experimental evidence is provided suggesting that OFP enjoys the same constant-regret properties in higher-dimensional games.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The presented results constitute the first O(1) regret bound for fictitious play without regularisation. This is already a result of interest.\n* The analysis utilizing energy functions is elegant, insightful, and affords some interesting tools to build upon, and potentially re-apply to the study of other approaches to the problem. In particular, I would foresee the energy function argument being applied to other related algorithmic approaches to two-player zero-sum games.\n* While analyzing this simple algorithm, the authors provide a series of intriguing observations that elegantly distinguish optimistic fictitious play from standard fictitious play. The observation of the existence of the invariant and the analysis stemming from the game-state representation are particularly interesting. \n* Experiments in higher-dimensional settings are compelling and strongly support the hypothesis that findings from the 2x2 setting extend to higher dimensions.\n* The story of the paper is extremely clear, and the writing and communication of key ideas is concise, intuitive and easy to follow.\n\nWeaknesses:\n* Despite strong evidence that higher-dimensional generalizations are plausible, the paper’s rigorous theoretical results remain limited explicitly to 2x2 settings.\n* The notation occasionally becomes cumbersome, making the proofs somewhat difficult to parse at certain points. However, given that the proof encompasses numerous intricate observations and ideas (and since it’s unclear how notation could be significantly simplified), this weakness is relatively minor.\n* The simplicity of the presented method raises questions about its broader applicability in more generic or practical scenarios (see final question).\n\nQUESTIONS:\n* Upon reviewing the proof, I don't see any obvious obstacles precluding direct generalization to higher-dimensional games. Is the primary bottleneck in extending results from 2x2 to higher-dimensional settings solely related to a generalization of Assumption 1 and Propositions 4.1 and 4.2? Would addressing these issues require only a more careful analysis of the relevant subspace dimensions, or might it necessitate different mathematical tools?\n* Do you have theoretical insights or intuitive explanations for why constant regret empirically holds in larger-dimensional games despite the theoretical analysis currently covering only 2×2 settings?\n* In some practical cases, the ratio $a_{\\max}/a_{\\min}$ might significantly exceed the actual realized losses. Could a tighter regret bound potentially be derived, possibly depending on the trajectories observed or norms of the subset of the payoff matrix A realized during play? Might gradient-adaptive methods provide tighter bounds here, even though they typically incorporate regularisation?\n* Regularised methods often exhibit greater robustness against perturbations in the payoff function. Given this, could the authors comment on how robust OFP is to noise or inaccuracies in the feedback or computation of the payoff matrix A?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the performance of unregularized online learning (the fictitious play algorithm aka FTL) in two-player zero sum games.  More specifically, they analyze FTL with two classic modifications (optimism and alternation) known to improve regret performance of regularized learning algorithms in games, and establish a positive result for the former and a negative result for the latter when regularization is removed, specifically in two-action games such as matching pennies.\n\nSTRENGTHS AND WEAKNESSES:\nThe positive result for optimistic FTL, though constrained to two-action games, I believe to be meaningful progress in the study of unregularized learning in games, which still has major open problems such as the weak fictitious play conjecture.  It is a genre of algorithms that is worth understanding due to their per-iterate time efficiency in solving large two-player zero-sum games in practice where the cost of computing a regularized best response is potentially high.  Also, the separation demonstrated from the performance of Alternating FTL gives grounds to the fact that optimism is perhaps the more natural modification to vanilla online learning in games.\n\nThis result is also interesting as it is somewhat unexpected to me.  The O(1) regret in two-player zero-sum games due to Syrgkanis et. al. comes from the RVU regret bound.  In the worst case, this regret bound makes no improvement over the adversarial $O(\\sqrt{T})$ rate, as to be expected due to the existence of simple Bernoulli adversaries forcing $\\Omega(\\sqrt{T})$.  However, with the added structure that the loss sequence experienced by the learner is coming from the learning of opponents in a game, we have that this loss sequence is *smooth*.  This smoothness, when plugged into the RVU bound, gives the improved $o(\\sqrt{T})$, even $O(1)$ sum of regret bound, which is sufficient for convergence to equilibrium in zero-sum games, unlike general sum games.  Thus, what astonishes me is that you can also get the matching $O(1)$ regret bound in this setting where non-regularized algorithms are inherently not smooth, having effectively infinite step size and always selecting actions at the boundary.  Accordingly, the proof technique of this result must be built in a fundamentally different way than RVU.  I find the potential argument the authors utilize instead to be quite satisfying.\n\nThe main drawback of this work is in its limitation to two-player zero-sum games, and more so *two-action* two-player zero-sum games.  As with my earlier skepticism, I find it unlikely that you will have a positive ($O(1)$ regret) result for unregularized learning in multi-player general-sum games as these learners lack smoothness.  Also, the potential argument as presented here does not seem to have hope to extend to many action two-player zero-sum games.  That said, the class of two-action two-player zero-sum games is sufficiently rich to establish learning-in-games lower bounds for even regularized algorithms without optimism (i.e. MWU having $\\Omega(\\sqrt{T})$ regret in matching pennies a la Chen and Peng).  So, the fact that you get a positive result for even matching pennies, as long as you are using optimism, even without regularization, I find to be a meaningful result standing on its own when juxtaposed to the Chen Peng result.  Thus, I recommend the acceptance of this result despite the aforementioned limitations, and perhaps the publicization of this result will encourage others to attempt to tackle the unregularized, optimistic, many action, two player zero sum setting.  A positive result there could be a potential boon for practical equilibrium computation.\n\nQUESTIONS:\nCould the authors clarify where the potential approach fails for many-action games?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper studies the regret of optimistic fictitious play in $2 \times 2$ zero-sum games. The authors show that the algorithm achieves constant regret without requiring any regularization. The main technical contribution lies in proving that an energy function (or potential function) remains bounded by an absolute constant. Numerical simulations are provided to support the theoretical findings.

STRENGTHS AND WEAKNESSES:
Strengths:

- The paper is very well-written. The contributions are clearly stated, the results and simulations are nicely presented, and the technical approach is easy to follow.

- Zero-sum games are a benchmark setting in game theory, and fictitious play is one of the most classical algorithms. However, understanding the convergence rate of the average iterate (or regret) has remained a long-standing open problem (e.g., Karlin’s weak conjecture). Showing constant regret—which implies a $\mathcal{O}(1/T)$ convergence rate for the average iterate—using a variant of fictitious play is a significant and meaningful contribution.

Weaknesses:

(1) The main result is limited to $2 \times 2$ games, as acknowledged by the authors. While this is a natural first step, it limits the generality of the findings.

(2) Since similar energy functions have already been constructed in prior work, the technical novelties of this paper are somewhat unclear.

QUESTIONS:
Can the authors more clearly state the fundamental challenges in extending their result beyond the $2 \times 2$ setting, and discuss any potential directions or ideas for addressing these challenges?

LIMITATIONS:
The main limitation is that the results work only for $2 \times 2$ games.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02