SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\lambda_{*} = \max \{|\lambda |\mid \lambda$  is an eigenvalue of  $\left(\prod_{t = 1}^{\mathcal{T}}\mathcal{A}_t\right),\lambda \neq 1\}$  , and  $\lambda_{*} <   1$
Note that in a periodic game, the iterative matrices of learning dynamics are also periodic, which means that the learning difference systems of these learning dynamics are periodic systems. According to Floquet theorem, see proposition 2.4, the study of dynamical behaviors of a periodic system can be reduced to the autonomous system whose asymptotic behavior is determined by  $\prod_{t=1}^{T} \mathcal{A}_t$ .
The key point on the proof of Theorem 3.1 is an observation that the iterative matrix of extra-gradient is a normal matrix, which makes it possible to calculate the Jordan normal form of  $\prod_{t=1}^{\mathcal{T}} \mathcal{A}_t$  for arbitrary large  $\mathcal{T}$ . The details of proof are left to Appendix B.
In the following theorem, we provide an example demonstrating that when two players use the optimistic gradient descent ascent or negative momentum method, their strategy will diverge at an exponential rate, regardless of how they choose their step-sizes and momentum parameters.
Theorem 3.2. Consider a periodic game with period  $\mathcal{T} = 2$ , and described by the following payoff matrix
$$
A _ {t} = \left\{ \begin{array}{l l} [ 1, - 1 ], & t \text {i s o d d} \\ [ - 1, 1 ], & t \text {i s e v e n} \end{array} \right. \tag {6}
$$
with  $x_{t} \in \mathbb{R}$ ,  $y_{t} \in \mathbb{R}^{2}$ . If two players use optimistic gradient descent ascent or negative momentum method, then regardless of how they choose step sizes and momentum parameters, we have
$$
\sup  _ {s \in [ t ]} \Delta_ {i, s} \in \Omega (\lambda^ {t}), \text {w h e r e} \lambda > 1, i \in \{1, 2 \}.
$$
Here  $\lambda$  is determined by the largest modulus of the eigenvalues of the iterative matrix of optimistic gradient descent ascent or negative momentum method.
To prove theorem 3.2, we directly calculate the characteristic polynomials of the iterative matrices products in one period for optimistic gradient descent ascent and negative momentum method under the game defined by (6). To show that these systems have an exponential divergence rate, it is sufficient to demonstrate that their characteristic polynomials have a root with modulus larger than 1. We achieve this by using the Schur stable theorem, which is also employed to demonstrate the last iterate convergence of several learning dynamics in time-independent game (Zhang and Yu [2020]). The proof is deferred to Appendix C.
In Figure (1), we present the function curves of  $\Delta_{1,t}$  for these three types of game dynamics under the periodic game defined by (6). From the experimental results, extra-gradient converges, while both optimistic gradient descent ascent and negative momentum method diverge.
Figure 1: Function curves of  $\Delta_{1,t}$  of the game presented in Theorem 3.2. Extra-gradient converges, while the other two methods diverge.
3.3 Convergent perturbed game
Recall that the payoff matrix of a convergent perturbed game has form  $A_{t} = A + B_{t}$ , where  $\lim_{t\to \infty}B_t = 0$ , and we refer to the zero-sum game defined by payoff matrix  $A$  as the stable game. We denote
$$
\Delta_ {t} = \left\| A ^ {\top} x _ {t} \right\| _ {2} + \left\| A y _ {t} \right\| _ {2}, \tag {7}
$$
thus  $\Delta_t$  measures the distance between the strategy  $(x_t, y_t)$  in the  $t$ -th round and the Nash equilibrium of the stable game defined by  $A$ . Moreover,  $\Delta_t = 0$  if and only if  $(x_t, y_t)$  is an equilibrium of the stable game.
In the literature on linear difference systems, a common assumption that needs to be added for convergence guarantee is the following bounded accumulated perturbations (BAP) assumption (Benzaid and Lutz [1987], Elaydi et al. [1999], Elaydi and Györi [1995]):
$$
\sum_ {t = 0} ^ {\infty} \| B _ {t} \| _ {2} \text {i s}
$$
(BAP assumption)
In the following theorem, we prove that under BAP assumption, all three learning dynamics considered in this paper will make  $\Delta_t$  converge to 0, with a rate dependent on the vanishing rate of  $B_t$ .
Theorem 3.3. Assume that the (BAP assumption) holds, i.e.,  $\sum_{t=0}^{\infty}\|B_t\|_2$  is bounded, and let  $\sigma$  be the maximum modulus of the singular value of payoff matrix  $A$ , then with parameters choice:
for extra-gradient with step size  $\alpha = \eta < \frac{1}{2\sigma}$ ,
- for optimistic gradient descent ascent with step size  $\eta < \frac{1}{2\sigma}$ ,
- for negative momentum method with step size  $\eta < \frac{1}{\sigma}$  and momentum parameters  $\beta_{1} = -\frac{1}{2}$  and  $\beta_{2} = 0$
we have  $\Delta_t$  converge to 0 with rate  $\mathcal{O}(f(t))$ . Here
$$
f (t) = \max  \left\{\lambda^ {t}, \sum_ {i = t / 2} ^ {\infty} \| B _ {i} \| _ {2} \right\},
$$
and  $\lambda \in (0,1)$  is determined by the eigenvalues of the iterative matrix of corresponding learning dynamics and the payoff matrix  $A$  of the stable game.
There are two main ingredients in the proof of Theorem 3.3: firstly, we show that the iterative matrices associated with these learning dynamics are diagonalizable; secondly, these matrices do not have eigenvalues with modulus larger than 1. Moreover, we prove a general result which states any linear difference system satisfying these two conditions will converge. The details of proof are left to Appendix E.
Remark 3.2. The (BAP assumption) can be converted into a constraint on the vanishing rate of  $B_{t}$ : if  $B_{t}$  has a vanishing rate like  $\mathcal{O}\left(\frac{1}{t^{1 + \epsilon}}\right)$ , for some arbitrary small  $\epsilon > 0$ , then  $\sum_{t=0}^{\infty}\|B_t\|_2$  is bounded. We also note that the condition for  $B_{t}$  to vanish at a rate  $\mathcal{O}\left(\frac{1}{t^{1 + \epsilon}}\right)$ ,  $\forall \epsilon > 0$  is necessary to ensure convergence in general linear difference system. For example, consider the 1-dimension system  $x_{t} = (1 + \frac{1}{t})x_{t-1}$  where a  $\mathcal{O}\left(\frac{1}{t}\right)$  convergence rate of perturbations leads to  $x_{t}$  diverging with a  $\Omega(t)$  rate.
Surprisingly, in the next theorem, we prove that Extra-gradient makes  $\Delta_t$  asymptotically converge to 0, without making any further assumptions about the converge rate of  $B_t$ .
Theorem 3.4. In a convergent perturbed game, if two players use Extra-gradient, there holds  $\lim_{t\to \infty}\Delta_t = 0$  with step size  $\alpha = \eta < \frac{1}{2\sigma}$  where  $\sigma$  is the maximum modulus of the singular value of payoff matrix  $A$ .
To prove Theorem 3.4 we first observe that  $\| x_{t}\|_{2} + \| y_{t}\|_{2}$  a non-increasing function of  $t$  since the iterative matrix of extra-gradient is a normal matrix. Next, we prove that if Theorem 3.4 doesn't hold, then  $\| x_{t}\|_{2} + \| y_{t}\|_{2}$  will decrease by a constant for infinite number of times, thus leading to a contradiction with the non-increasing property and the non-negativity of  $\| x_{t}\|_{2} + \| y_{t}\|_{2}$ . The proof is deferred to Appendix F.
In this section we present numerical results for our theoretical results in Section 3.
Experiments on Theorem 3.1 We verify Theorem 3.1 through a period-3 game, the payoff matrices are chosen to be
$$
A _ {1} = \left[ \begin{array}{c c} 1 & 2 \\ 2 & 4 \end{array} \right], A _ {2} = \left[ \begin{array}{c c} 3 & 7 \\ 7 & 1 \end{array} \right], A _ {3} = \left[ \begin{array}{c c} 4 & 2 \\ 4 & 2 \end{array} \right].
$$
We run extra-gradient and optimistic gradient descent ascent on this example, both with step size  $= 0.01$ , the experimental results are presented in Figure (2). We can see extra-gradient (left) makes  $\Delta_{i,t}$  converge, while optimistic gradient descent ascent (right) makes  $\Delta_{i,t}$  diverge. This result supports Theorem 3.1 and provides a numerical example of the separation of extra-gradient and optimistic gradient descent ascent in periodic games.
Figure 2: Function curves of  $\Delta_t$  for extra-gradient (left), and optimistic gradient descent ascent (right).
Experiments on Theorem 3.3 We verify Theorem 3.3 by examples:
$$
A = \left[ \begin{array}{c c} 2 & 3 \\ 4 & 6 \end{array} \right], B _ {1, t} = B \cdot t ^ {- 1. 1}, B _ {2, t} = B \cdot t ^ {- 4}, \text {a n d}, B _ {3, t} = B \cdot t ^ {- 8}
$$
where  $B = [[-15, 70], [-90, 90]]$ . The step size is chosen to be 0.005. The initial points are chosen to be  $x_0 = [15, 13]$ ,  $x_{-1} = [11, 3]$  and  $y_0 = [35, 1]$ ,  $y_{-1} = [35, 1]$ . The experimental results are presented in Figure 3 all of the three dynamics make  $\Delta_t$  converge to 0, and slower convergence rate of perturbations can decelerate the convergence rate of learning dynamics, thus support the convergence result in Theorem 3.3. We also observe that the convergence rate is faster than the upper bound provided in the theorem. Therefore, we conjecture the bound of convergence rate can be improved.
Figure 3: Values of  $\Delta_{t}$  for extra-gradient (left), optimistic gradient descent ascent (middle), negative momentum method (right).
Experiments on Theorem 3.4 We verify Theorem 3.4 by two groups of examples. The perturbations are :
$$
B _ {1, t} = B \cdot t ^ {- 0. 4}, B _ {2, t} = B \cdot t ^ {- 0. 3}, B _ {3, t} = B \cdot t ^ {- 0. 2},
$$
$$
B _ {4, t} = B \cdot \log (t) ^ {- 1. 8}, B _ {5, t} = B \cdot \log (t) ^ {- 1. 5}, B _ {6, t} = B \cdot \log (t) ^ {- 1. 3}.
$$
where  $B = [[-10, 10], [-10, 10]]$ . The payoff matrix of stable game is chosen to be  $A = [[2, 3], [4, 6]]$ . Note that the perturbations do not satisfy (BAP assumption). The experimental results are shown in Figure 4. We can see all these curves converge to 0, thus support the result in Theorem 3.4. Furthermore, we can observe that large perturbations may lead to more oscillations during the convergence processes, which in turn slows down the rate of convergence. We present more experiments in Appendix G.
Figure 4: Values of  $\Delta_{t}$  for extra-gradient with  $A + B_{i,t}, i = 1,2,3$  (left), and  $A_{2} + B_{i,t}, i = 4,5,6$  (right).
In this paper, we study the last-iterate behavior of extra-gradient, optimistic gradient descent ascent, and negative momentum method in two types of time-varying games: periodic game and convergent perturbed game. In the case of periodic game, we prove that extra-gradient will converge to a Nash equilibrium while other two methods diverge. To the best of our knowledge, this is the first result that provides a clear separation between the behavior of extra-gradient methods and optimistic gradient descent ascent. In the case of convergent perturbed game, we prove all three learning dynamics converge to Nash equilibrium under the BAP assumption, which is commonly used in the literature on dynamical systems.
Our results leave many interesting open questions. Firstly, is the BAP assumption necessary for ensuring convergence in optimistic gradient descent ascent and negative momentum method? Secondly, the bound of convergence rate in Theorem 3.3 may rather slight as we shown in experiments section. Obtaining a tighter bound on the convergence rate is an important future research problem. Thirdly, can the results here be generalized to other settings, such as constrained zero-sum game?
Yi Feng is supported by the Fundamental Research Funds for the Central Universities. Ioannis Panageas wants to thank a startup grant. Xiao Wang acknowledges Grant 202110458 from SUFE and support from the Shanghai Research Center for Data Science and Decision Technology.
Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On the convergence of no-regret learning dynamics in time-varying games. arXiv preprint arXiv:2301.11241, 2023.
Z Benzaid and DA Lutz. Asymptotic representation of solutions of perturbed systems of linear difference equations. Studies in Applied Mathematics, 77(3):195-221, 1987.
David Blackwell. An analog of the minimax theorem for vector payoffs. In Pacific J. Math., pages 1-8, 1956.
George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation, 13(1): 374, 1951.
Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Finite-time last-iterate convergence for learning in multi-player games. In NeurIPS, 2022.
Adrian Rivera Cardoso, Jacob D. Abernethy, He Wang, and Huan Xu. Competing against nash equilibria in adversarially changing zero-sum games. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pages 921-930. PMLR, 2019.
Fritz Colonius and Wolfgang Kliemann. Dynamical systems and linear algebra, volume 158. American Mathematical Society, 2014.
Yuri Nesterov. Smooth minimization of non-smooth functions. Math. Program., 103(1):127-152, 2005.
Mihály Pituk. More on poincaré's and perron's theorems for difference equations. The Journal of Difference Equations and Applications, 8(3):201-216, 2002.
Leonid Denisovich Popov. A modification of the arrow-hurwicz method for search of saddle points. Mathematical notes of the Academy of Sciences of the USSR, 28:845-848, 1980.
Abhishek Roy, Yifang Chen, Krishnakumar Balasubramanian, and Prasant Mohapatra. Online and bandit algorithms for nonstationary stochastic saddle-point optimization. CoRR, abs/1912.01698, 2019.
Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E. Schapire. Fast convergence of regularized learning in games. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2989-2997, 2015.
Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Guodong Zhang and Yuanhao Wang. On the suboptimality of negative momentum for minimax optimization. In International Conference on Artificial Intelligence and Statistics, pages 2098-2106. PMLR, 2021.
Guojun Zhang and Yaoliang Yu. Convergence of gradient methods on bilinear zero-sum games. In International Conference on Learning Representations, 2020.
Mengxiao Zhang, Peng Zhao, Haipeng Luo, and Zhi-Hua Zhou. No-regret learning in time-varying zero-sum games. In International Conference on Machine Learning, ICML 2022, volume 162 of Proceedings of Machine Learning Research, pages 26772-26808. PMLR, 2022.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Tom Fawcett and Nina Mishra, editors, Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), August 21-24, 2003, Washington, DC, USA, pages 928-936. AAAI Press, 2003.