Title: From Average-Iterate to Last-Iterate Convergence in Games: A Reduction and Its Applications
Venue: NeurIPS 2025
Accepted: yes
Keywords: Last-Iterate Convergence

Learning in Games

Zero-Sum Games

Bandit Feedback

Online Learning
Abstract: The convergence of online learning algorithms in games under self-play is a fundamental question in game theory and machine learning. Among various notions of convergence, last-iterate convergence is particularly desirable, as it reflects the actual decisions made by the learners and captures the day-to-day behavior of the learning dynamics. While many algorithms are known to converge in the average-iterate, achieving last-iterate convergence typically requires considerably more effort in both the design and the analysis of the algorithm. Somewhat surprisingly, we show in this paper that for a large family of games, there exists a simple black-box reduction that transforms the average iterates of an uncoupled learning dynamics into the last iterates of a new uncoupled learning dynamics, thus also providing a reduction from last-iterate convergence to average-iterate convergence. Our reduction applies to games where each player’s utility is linear in both their own strategy and the joint strategy of all opponents. This family includes two-player bimatrix games and generalizations such as multi-player polymatrix games. By applying our reduction to the Optimistic Multiplicative Weights Update algorithm, we obtain new state-of-the-art last-iterate convergence rates for uncoupled learning dynamics in multi-player zero-sum polymatrix games: (1) an $O(\frac{\log d}{T})$ last-iterate convergence rate under gradient feedback, representing an exponential improvement in the dependence on the dimension $d$ (i.e., the maximum number of actions available to either player); and (2) an $\tilde{O}(d^{\frac{1}{5}}T^{-\frac{1}{5}})$ last-iterate convergence rate under bandit feedback, improving upon the previous best rates of $\tilde{O}(\sqrt{d}T^{-\frac{1}{8}})$ and $\tilde{O}(\sqrt{d}T^{-\frac{1}{6}})$.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies last-iterate-convergence methods for solving normal-form games. Instead of proposing a new method from scratch, it introduces a black-box reduction (A2L) that converts any online learning algorithm with average-iterate convergence into one that achieves last-iterate convergence, under the assumption of linear utilities. The authors apply the reduction to OMWU, and derive state-of-the-art convergence guarantees under gradient feedback and bandit feedback.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe paper is in general clear and well organized. Its significance and novelty over prior work are effectively communicated, with both the high-level intuition and technical contributions presented at appropriate points throughout the paper.\n\nThe reduction method is simple and intuitive. Though highly dependent on the linear utility assumption, the method still covers a number of important game classes. Clear theoretical benefits --  faster convergence rates, uncoupled learning dynamics, last-iterate convergence -- are shown in solving zero-sum games under gradient and bandit feedback, without requiring major algorithmic changes .\n\nWeakness\n\nThe authors explained the merit of last-iterate convergence a few times in the paper, but I am not fully convinced. From the perspectives of practical implementation, tracking the average iterates is simple to carry out and does not introduce any additional time or memory complexity. I wonder if there are more concrete practical benefits from last-iterate-convergent algorithms that can help provide a better motivation.\n\nAssumption 1 is another limitation, and I cannot see how any of the arguments in the paper can be applied/extended when the utility is nonlinear. However, this does not mean the work is unimportant. The normal-form game framework with linear utilities captures several fundamental and widely studied game classes, such as two-player bimatrix games and multi-player polymatrix games, including their zero-sum variants. Just being able to solve these games alone is a contribution.\n\nQUESTIONS:\n1) I would be interested to see how A2L-OMWU compares with other EG/OG algorithms known to have last-iterate convergence that the authors referenced in Related Works (line 102 and below). The purpose is to see whether A2L-OMWU truly converges faster both in theory and in practice, or whether the other algorithms also perform well in practice, with their current inferior convergence rates being an artifact of analysis.\n\n2) Can any of the argument be generalized to Markov games (with linear reward function)? \n\n3) A few highly relevant references should be included and discussed that design last-iterate-convergent algorithms for zero-sum games based on perturbation/regularization, including Abe et al., 2022, Abe et al., 2023, Wu et al., 2023, Zeng et al., 2023.\n\nReferences\n\nAbe, K., Sakamoto, M., & Iwasaki, A. (2022, August). Mutation-driven follow the regularized leader for last-iterate convergence in zero-sum games. In Uncertainty in Artificial Intelligence (pp. 1-10). PMLR.\n\nAbe, K., Ariu, K., Sakamoto, M., Toyoshima, K., & Iwasaki, A. (2023). Last-Iterate Convergence with Full and Noisy Feedback in Two-Player Zero-Sum Games. In AISTATS 2023, International Conference on Artificial Intelligence and Statistics, 25-27 April 2023, Palau de Congressos, Valencia, Spain (Vol. 206, pp. 7999-8028). MLResearchPress.\n\nWu, J., Barakat, A., Fatkhullin, I., & He, N. (2023). Learning Zero-Sum Linear Quadratic Games with Improved Sample Complexity and Last-Iterate Convergence. arXiv preprint arXiv:2309.04272.\n\nZeng, S., Doan, T., & Romberg, J. (2022). Regularized gradient descent ascent for two-player zero-sum Markov games. Advances in Neural Information Processing Systems, 35, 34546-34558.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the convergence of online learning algorithms in games, focusing on the theoretical and practically importance setting, the last-iterate convergence property of algorithms used in a large class of games. The technique of reduction from last iterates to average-iterate seems novel and significant. The results and techniques also implies generalizations in multi-player setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\nThe paper is well written and easy to follow. I think the authors have done a great effort trying to make the paper accessable to readers who are not experts in the narrow area. \n\nThe contributions of this paper on the improvement of convergence rates are clear and comparison to related results are presented. \n\nThe importance of reduction is the improvement for uncoupled learning in multi-player zero-sum polymatrix games. This reduction technique enables authors to obtain a last-iterate convergence rate for OMWU that matches its average-iterate rate under the same setting, say the gradient feedback. The bandit feedback is analyzed based on the same reduction technique, and this paper propose a OMWU-bandit algorithm that achieves a SOTA convergence rate compared to existing results.\n\nWeakness\n\nI think a general reader might be interested in the reduction technique used in this paper. If the authors could have provided a pointer to literature or open a section to discuss the background, the current layout of Section 3 mainly focuses on the technical side of this paper, and the extensive study mentioned section 4 refers to Opmtimistic learning algorithms. Since the results of this paper already say for the power of this key technique, a detailed introduction to its idea based on related works will be more helpful to curious readers.\n\nQUESTIONS:\nI am not totally sure whether the reduction technique is an innovation compared to the literature. \nWhat are other related papers that might be considered inspiring to this approach?\n\nLIMITATIONS:\nYes, the limitation of the results and method is clearly stated in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes A2L, a black-box reduction that transforms any online learning algorithm with average-iterate convergence guarantees into one whose last iterates converge. The reduction applies to games with payoffs linear in players’ strategies, including zero-sum bimatrix games and multi-player polymatrix games. Furthermore, they show that applying A2L to Optimistic Multiplicative Weights Update  improved last-iterate convergence rates and the fastest known convergence under bandit feedback.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1) The black box reduction is simple but able to provide stronger last-iterate convergence rates for zero-sum games, i.e., they get a last-iterate rate of $O(\\log d /T)$, when they apply their black-box reduction to OMWU.\n\n2) They also get improved rates for the gradient feedback and the bandit case for zero-sum games, where they show that the ``reduced'' versions of Bandit-OMWU achieve rates of $O(T^{-1/5})$, compared to prior approaches that achieve O($T^{-1/8}$).\n\nWeakness:\n\n1) They already state that their techniques are limited to games that satisfy the Linear utilities assumption (Assumption 1), such as bimatrix games and polymatrix games.\n\n2) Although it is mainly a theory paper, it would be nice to see how the A2L-OMWU performs in practice against OMWU and OGDA for instance.\n\nSee the questions section as I would like to seek further clarification.\n\nQUESTIONS:\n1) The reduction reminds me of the paper by Cutkosky (2019), where they do anytime online to batch conversion. Can the authors comment on any similarities/differences with their approach. In general, I think it would be useful to cite it. \n\n2) Following from the previous question Cutkosky (2019) have a weighted average version in their reduction and I was wondering, if your reduction can also have such weights embedded and whether that could be useful?\n\n3) It is known that Cai et al. (2024) have this game defined as $A_{\\delta}$ in their paper that shows the lower bound for OMWU. If I understand your results correctly, the A2L-OMWU, in its rates does not have a game dependent constant? If so this must be highlighted.\n\n4) The reduction was shown through OMWU, but can it be applied to standard (non-optimistic) versions to get last-iterate convergence? Is it also applicable to general FTRL? \n\n5) It would be nice to see some experiments on large-scale games to see the performance of A2L-OMWU, OMWU and OGDA in practice, with respect to number of steps and also the wall-clock time.\n\n6) Could the authors also clarify the dependence on the other constants when compared to Cai et al. (2023)? in Theorem 4?\n\nReferences:\n\nCutkosky, Ashok. \"Anytime online-to-batch, optimism and acceleration.\" International conference on machine learning. PMLR, 2019.\n\nYang Cai, Haipeng Luo, Chen-Yu Wei, and Weiqiang Zheng. “Uncoupled and convergent learning in two-player zero-sum markov games with bandit feedback”. In: Advances in Neural Information Processing Systems 36 (2023), pp. 36364–36406. \n\nYang Cai, Gabriele Farina, Julien Grand-Clément, Christian Kroer, Chung-Wei Lee, Haipeng Luo, and Weiqiang Zheng. “Fast Last-Iterate Convergence of Learning in Games Requires Forgetful Algorithms”. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
average iterate convergence rates. There has been increasing interest
in deriving last-iterate convergence guarantees for equilibrium
learning algorithms. Such guarantees are more practically useful, but
also harder to obtain.

The main contribution of the paper is an algorithm that takes an
algorithm with an average-iterate convergence guarantee, and turns it
into an algorithm with a last iterate convergence guarantee. The
construction assumes that the utility function is linear in their own
strategy. They consider both settings with gradient feedback and
bandit feedback.

The authors argue that in some cases, this leads to state of the art
results. By applying the approach to the OMWA algorithm for zero-sum
polymatrix games, this leads to state of the art last-iterate
convergence guarantees.

STRENGTHS AND WEAKNESSES:
This is an interesting result worthy of publishing. In particular, it
may provide an easier route to last-iterate guarantees. It is an open
question how practically useful the approach it.

The paper is well presented, and I only have some minor suggested
improvements.

The proofs for the gradient-based setting are relatively short and
straightforward, while the bandit based settings are more involved. I
have read the proofs and only have a few remarks.

QUESTIONS:
Minor remarks

Second inequality after 585 (proof of Theorem 3): Comparing with Lemma
2, there seems to be a (n-1) factor missing. Do you need to assume
smaller eta?

Line 598: Please give some more explanation for the expression for
delta^t.

It would have been nice to see the theoretical results complemented
with numerical experiments, e.g., comparing OMWU with A2L-OMWU on some
interesting games.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03