Title: On the Last-iterate Convergence in Time-varying Zero-sum Games: Extra Gradient Succeeds where Optimism Fails
Venue: NeurIPS 2023
Accepted: yes
Keywords: zero sum game

time-varying game

optimistic gradient

extra gradient

momentum method
Abstract: Last-iterate convergence has received extensive study in two player zero-sum games starting from bilinear, convex-concave up to settings that satisfy the MVI condition. Typical methods that exhibit last-iterate convergence for the aforementioned games include extra-gradient (EG) and optimistic gradient descent ascent (OGDA). However, all the established last-iterate convergence results hold for the restrictive setting where the underlying repeated game does not change over time.
Recently, a line of research has focused on regret analysis of OGDA  in time-varying games, i.e., games where payoffs evolve with time; the last-iterate behavior of OGDA and EG in time-varying environments remains unclear though. In this paper, we study the last-iterate behavior of various algorithms in two types of unconstrained, time-varying, bilinear zero-sum games: periodic and convergent perturbed games. These models expand upon the usual repeated game formulation and incorporate external environmental factors, such as the seasonal effects on species competition and vanishing external noise. In periodic games, we prove that EG will converge while OGDA and momentum method will diverge. This is quite surprising, as to the best of our knowledge, it is the first result that indicates EG and OGDA have qualitatively different last-iterate behaviors and do not exhibit similar behavior. In convergent perturbed games, we prove all these algorithms converge as long as the game itself stabilizes with a faster rate than $1/t$.

On the Last-iterate Convergence in Time-varying Zero-sum Games: Extra Gradient Succeeds where Optimism Fails

Abstract

Last-iterate convergence has received extensive study in two player zero-sum games starting from bilinear, convex-concave up to settings that satisfy the MVI condition. Typical methods that exhibit last-iterate convergence for the aforementioned games include extra-gradient (EG) and optimistic gradient descent ascent (OGDA). However, all the established last-iterate convergence results hold for the restrictive setting where the underlying repeated game does not change over time. Recently, a line of research has focused on regret analysis of OGDA in time-varying games, i.e., games where payoffs evolve with time; the last-iterate behavior of OGDA and EG in time-varying environments remains unclear though. In this paper, we study the last-iterate behavior of various algorithms in two types of unconstrained, time-varying, bilinear zero-sum games: periodic and convergent perturbed games. These models expand upon the usual repeated game formulation and incorporate external environmental factors, such as the seasonal effects on species competition and vanishing external noise. In periodic games, we prove that EG will converge while OGDA and momentum method will diverge. This is quite surprising, as to the best of our knowledge, it is the first result that indicates EG and OGDA have qualitatively different last-iterate behaviors and do not exhibit similar behavior. In convergent perturbed games, we prove all these algorithms converge as long as the game itself stabilizes with a faster rate than 1 /t .

1 Introduction

A central problem in game theory and min-max optimization is to come up with a pair of vectors ( x, y ) that solves

where X ⊂ R n and Y ⊂ R m are convex sets, and A is a n × m payoff matrix. The above captures two-player zero-sum games in which x /latticetop Ay is interpreted as the payment of the 'min player' x to the 'max player' y . If X = R n and Y = R m the setting is called unconstrained , otherwise it is constrained . Soon after the minimax theorem of Von Neumann was established (for compact X , Y ), learning dynamics such as fictitious play (Brown [1951]) were proposed for solving min-max optimization problems. Blackwell's approachability theorem ( Blackwell [1956]) further propelled the field of online learning, which lead to the discovery of several learning algorithms; such learning methods include multiplicative-weights-update method, online gradient descent/ascent and their optimistic variants and extra-gradient methods.

Last Iterate Convergence. There have been a vast literature on whether or not the aforementioned dynamics converge in an average sense or exhibit last-iterate convergence when applied to zero-sum games. Dating back to Nesterov (Nesterov [2005]), there have been quite a few results showing that online learning algorithms have last-iterate convergence to Nash equilibria in zero-sum games. Examples include optimistic multiplicative weights update (Daskalakis and Panageas [2019], Wei et al. [2021]), optimistic gradient descent ascent (OGDA) (Daskalakis et al. [2018], Liang and Stokes [2019a]) (applied even to GANs) for unconstrained zero-sum games, OGDA for constrained zero-sum games (Wei et al. [2021], Cai et al. [2022], Gorbunov et al. [2022b]) and extra-gradient methods (Mertikopoulos et al. [2019], Cai et al. [2022], Gorbunov et al. [2022a]) using various techniques, including sum of squares.

Nevertheless, all aforementioned results assume that the underlying repeated zero-sum game remains invariant throughout the learning process. In many learning environments that assumption is unrealistic, see (Duvocelle et al. [2022], Mai et al. [2018], Cardoso et al. [2019]) and references therein. One more realistic learning setting is where the underlying game is actually changing; this game is called time-varying. There have been quite a few works that deal with time-varying games, where they aim at analyzing the duality gap or dynamic regret (Zhang et al. [2022]) and references therein for OGDA and variants. However, in all these prior works, last-iterate convergence has not been investigated; the main purpose of this paper is to fill in this gap. We aim at addressing the following question:

Will learning algorithms such as optimistic gradient descent ascent or extra-gradient exhibit last-iterate converge in time-varying zero-sum games?

Our contributions. We consider unconstrained two-player zero-sum games with a time-varying payoff matrix (that is the payoff matrix A t depends on time t )

in which the payoff matrix A t varies with time in the following two ways:

Periodic games: A t is a periodic function with period T , i.e., A t + T = A t .

Convergent perturbed games: A t = A + B t , lim t →∞ B t = 0 .

In a repeated time-varying zero-sum game, players choose their learning algorithms and repeatedly play the zero-sum game. In the t -th round, when the players use strategy ( x t , y t ) , they receive their payoff -A /latticetop t x t and A t y t .

In this paper we show the following results:

For periodic games: We prove that when two players use extra-gradient, their strategies will converge to the common Nash equilibrium of the games within a period with an exponential rate, see Theorem 3.1 for details. Additionally, we provide an example where optimistic gradient descent ascent and negative momentum method diverge from the equilibrium with an exponential rate, see Theorem 3.2.

To the best of our knowledge, this is the first result that provides a clear separation between the behavior of extra-gradient methods and optimistic gradient descent ascent.

For convergent perturbed games: Assuming ∑ ∞ t =1 ‖ B t ‖ 2 is bounded, we prove that the extra-gradient, optimistic gradient descent ascent, and negative momentum method all converge to the Nash equilibrium of the game defined by payoff matrix A with a rate determined by { B t } t and singular values of A , see Theorem 3.3. Furthermore, we prove that extra-gradient will asymptotically converge to equilibrium without any additional assumptions on perturbations besides lim t →∞ B t = 0 , see Theorem 3.4.

Related work on time-varying games. The closest work to ours that argues about stabilization of mirror descent type dynamics on convergent strongly monotone games is (Duvocelle et al. [2022]). Most of the literature has been focused on proving either recurrence/oscillating behavior of learning dynamics in time-varying periodic games (Mai et al. [2018], Fiez et al. [2021] ) and references therein or performing regret analysis (Roy et al. [2019], Cardoso et al. [2019], Zhang et al. [2022]). In particular, the latter work extends results on RVU (Syrgkanis et al. [2015]) bounds to argue about dynamic regret (Zinkevich [2003]).

Technical Comparison. We investigate the last iterate behaviors of learning dynamics through their formulation of linear difference systems. This approach has also been used to establish last iterate convergence results for learning dynamics in time-independent games (Zhang and Yu [2020], Liang and Stokes [2019b], Gidel et al. [2019]). One common key point of these works is to prove that a certain matrix has no eigenvalues with modulus larger than 1 , then the last iterate convergence can be guaranteed by the general results of autonomous difference systems. However, this method cannot be generalized to the time-varying games where the corresponding difference systems are non-autonomous. In particular, the dynamical behavior of a non-autonomous system is not determined by the eigenvalues of a single matrix. In fact, it is difficult to establish convergence/divergence results even for non-autonomous system with special structures, such as periodic or perturbed systems. In this paper, to get such results, we employ both general results in linear difference systems, such as Floquet theorem and Gronwall inequality, and special structures of the difference systems associated to learning dynamics.

Organization. In Section 2, we present the necessary background for this work. The main results are stated in Section 3. In Section 4, we present numerical experiments and in Section 5, we conclude with a discussion and propose some future research problems.

2 Preliminaries

2.1 Definitions

Zero-sum blinear game. An unconstrained two players zero-sum game consists of two agents N = { 1 , 2 } , and losses of both players are determined via payoff matrix A ∈ R n × m . Given that player 1 selects strategy x ∈ R n and player 2 selects strategy y ∈ R m , player 1 receives loss u 1 ( x, y ) = 〈 x, Ay 〉 , and player 2 receives loss u 2 ( x, y ) = -〈 y, A /latticetop x 〉 . Naturally, players want to minimize their loss resulting the following min-max problem:

Note that the set { ( x ∗ , y ∗ ) | A /latticetop x ∗ = 0 , Ay ∗ = 0 } represents the set of equilibrium of the game.

Time-varying zero-sum bilinear game. In this paper, we study games in which the payoff matrices vary over time and we define two kinds of such time-varying games.

Definition 2.1 (Periodic games) . A periodic game with period T is an infinite sequence of zero-sum bilinear games { A t } ∞ t =0 ⊂ R n × m , and A t + T = A t for all t ≥ 0 .

Note that the periodic game defined here is the same as Definition 1 in (Fiez et al. [2021]) except for the fact that we are considering a discrete time setting. Therefore, we do not make the assumption that payoff entries are smoothly dependent on t .

Definition 2.2 (Convergent perturbed games) . A convergent perturbed game is an infinite sequence of zero-sum bilinear games { A t } ∞ t =0 ⊂ R n × m , and lim t →∞ A t = A for some A ∈ R n × m . Equivalently, write A t = A + B t , then lim t →∞ B t = 0 . We will refer to the zero-sum bilinear game defined by A as stable game .

Learning dynamics in games. In this paper, we consider three kinds of learning dynamics : optimistic gradient descent ascent (OGDA), extra-gradient (EG) , and negative momentum method. All these methods possess the last-iterate convergence property in repeated game with a time-independent payoff matrix, as demonstrated in previous literature. However, here we state their forms within a time-varying context.

Optimistic gradient descent-ascent. We study the optimistic descent ascent method (OGDA) defined as follows:

Optimistic gradient descent ascent method was proposed in (Popov [1980]), and here we choose the same parameters as (Daskalakis et al. [2018]). The last iterate convergence property of OGDA in unconstrained bilinear game with a time-independent payoff was proved in (Daskalakis et al. [2018]). Recently, there are also works analyzing the regret behaviors of OGDA under a time varying setting (Zhang et al. [2022], Anagnostides et al. [2023]).

Extra gradient. We study the extra gradient descent ascent method (EG) defined as follows:

Note that the extra-gradient method first calculates an intermediate state before proceeding to the next state. Extra-gradient was firstly proposed in (Korpelevich [1976]) with the restriction that α = γ . Here we choose the parameters same as in (Liang and Stokes [2019a]), where the linear convergence rate of extra-gradient in the bilinear zero-sum game with time-independent was also proven. Convergence of extra-gradient on convex-concave game was analyzed in (Nemirovski [2004], Monteiro and Svaiter [2010]), and convergence guarantees for special non-convex-non-concave game was provided in (Mertikopoulos et al. [2019]).

Negative momentum method. We study the alternating negative momentum method (NM), defined as follows:

where β 1 , β 2 ≤ 0 are the momentum parameters.

Applications of negative momentum method in game optimization was firstly proposed in (Gidel et al. [2019]). Note that the algorithm has an alternating implementation: the update rule of y t +1 uses the payoff A /latticetop t +1 x t +1 , thus in each round, the second player chooses his strategy after the first player has chosen his. It was shown in (Gidel et al. [2019]) that both the negative momentum parameters and alternating implementations are crucial for convergence in bilinear zero-sum games with time-independent payoff matrices. Analysis of the convergence rate of negative momentum method in strongly-convex strongly-concave games was provided in (Zhang and Wang [2021]).

2.2 Difference systems

The analysis of the last-iterate behavior of learning algorithms can be reduced to analyzing the dynamical behavior of the associated linear difference systems (Zhang and Yu [2020], Daskalakis and Panageas [2018]). When the payoff matrix is time-independent, the associated difference systems is autonomous. However, as we are studying games with time-varying payoff matrices, we have to deal with non-autonomous difference systems. In general, the convergence behavior of non-autonomous difference systems is much more complicated than that of autonomous ones (Colonius and Kliemann [2014]).

Linear difference system. Given a sequence of iterate matrices {A t } ∞ t =1 ⊂ R n × n and initial condition X 0 ∈ R n , a linear difference system has form

(Linear difference system)

If A t ≡ A is a matrix independent of time, the system is called an autonomous system , otherwise, it is called a non-autonomous system . We care about the asymptotic behavior of X t , that is, what can we say about X t as t →∞ .

Definition 2.3. A point X is called asymptotically stable under the linear difference system if

Moreover, X is called exponentially asymptotically stable if the above limit has an exponentially convergence rate, i.e., ∃ α ∈ (0 , 1) , such that

It is well known that for autonomous linear systems, being asymptotically stable is equivalent to being exponentially asymptotically stable, as shown in Thm 1.5.11 in (Colonius and Kliemann [2014]). However, this equivalence does not hold for non-autonomous systems. †

Linear difference system has a formal solution X T +1 = ∏ T t =0 A t X 0 . However, such a representation does not yield much information about the asymptotic behavior of solution as t →∞ , except in the case of autonomous system. In this paper, we mainly consider two classes of non-autonomous linear difference systems: if the iterate matrix A t is a periodic function of t , the system is called a periodic system; and if A t has a limit as t tends to infinity, the system is called a perturbed system.

Periodic linear system. If the iterate matrix A t in a linear difference system satisfies A t = A t + T , t , then the system is called a periodic linear system. Denote = T j . For a T -

∀ ∈ Z ˜ A ∏ j =1 A T -periodic equation, the eigenvalues α ∈ C of ˜ A is called the Floquet multipliers, and the Floquet exponents are defined by λ j = 1 T ln( | α j | ) . The following Floquet theorem characterizes the stability of a periodic linear difference equation.

Proposition 2.4 (Floquet theorem, (Colonius and Kliemann [2014])) . The zero solution of a periodic linear difference equation is asymptotically stable if and only if all Floquet exponents are negative.

Although Floquet theorem provides a method to determine whether a periodic system converges in general, considerable further work may be necessary in order to obtain explicit convergence criteria for specific equation. That is because even if we know the modulus of the largest eigenvalue of each A t , it is usually difficult to compute the the modulus of the largest eigenvalue of each ∏ T t =0 A t due to the complex behavior of eigenvalues under matrix multiplication.

Perturbed linear system. If the iterative matrix A t in a linear difference system satisfies A t = A + B t and lim t →∞ B t = 0 , then the system is called a perturbed linear system. The convergence behavior of a perturbed linear system is not clear in the literature. A general result in this direction is the following Perron's theorem:

Theorem 2.5 (Perron's theorem, (Pituk [2002])) . If X n is a solution of a perturbed linear system, then either X n = 0 for all sufficient large n or ρ = lim n →∞ n √ ‖ X n ‖ 2 exists and is equal to the modulus of one of the eigenvalues of matrix A .

This result can only guarantee the convergence of X n when all eigenvalues of A have modulus smaller than 1. In this case, lim n →∞ X n = 0 . However, for analyzing the non-autonomous linear systems associated to learning dynamics, it is not sufficient as we will show that the stablized matrix of these systems generally has eigenvalues equal to 1 .

3 Main results

In this section, we present our main results. We present the relationship between learning dynamics and linear difference equation in Section 3.1, investigate the last-iterate behaviors of learning dynamics in a periodic game in Section 3.2, and investigate the last-iterate behaviors of learning dynamics in a convergent perturbed game in Section 3.3. Proofs are deferred to the appendix.

3.1 Learning dynamics as linear difference systems

Formalizing learning dynamics as linear difference systems is useful for studying their dynamical behaviors. In the following, we present the formulation of optimistic gradient descent ascent, extragradient, and negative momentum method as linear difference systems.

Proposition 3.1. Optimistic gradient descent ascent can be written as the following linear difference system:

Extra-gradient can be written as the following linear difference system :

Negative momentum method can be written as the following linear difference system:

It is easy to verify that these linear difference systems are equivalent to their corresponding learning dynamics by directly writing down the matrix-vector product.

We will also refer to the iterative matrix of these linear difference systems as the iterative matrix of their corresponding learning dynamics . In the following, we will study the convergence/divergence behaviors of these linear difference systems under the condition that { A t } t is a periodic game or convergent perturbed game. Note that although an intermediate step ( x t + 1 2 , y t + 1 2 ) is required in extra-gradient, it is eliminated in (3).

3.2 Periodic games

Recall that in a periodic game with period T , the payoff matrices { A s } ∞ s =1 satisfy A s + T = A s for any s > 0 . Define

for i ∈ [ T ] . As in (Daskalakis and Panageas [2018]), we use ∆ i,t as a measurement of the distance between the current strategy ( x t , y t ) and a Nash equilibrium of the zero-sum bilinear game defined by the kernel space of payoff matrix A i . Note that if ( x ∗ , y ∗ ) is a Nash equilibrium of the game defined by A i , then ( A /latticetop i x ∗ , A i y ∗ ) = (0 , 0) , and

thus when strategy is close to equilibrium, ∆ i,t will be small. Moreover, ∆ i,t = 0 if and only if ( x t , y t ) is an equilibrium. In this section, we will consider the convergence/growth rate of ∆ i,t at a function of t .

We firstly consider Extra-gradient method. Denote the iterative matrix in the linear difference form of Extra-gradient (3) as A t . In the following theorem, we prove that if two players use the Extra-gradient method, their strategies will converge to the common Nash equilibrium of games in the period with an exponential rate.

Theorem 3.1. When two players use extra-gradient in a periodic games with period T , with step size α = γ < 1 σ where σ = max { σ ′ | σ ′ is a singular value of A i for some i ∈ [ T ] } . Then

/negationslash where λ ∗ = max { | λ | | λ is an eigenvalue of ( ∏ T t =1 A t ) , λ = 1 } , and λ ∗ < 1 .

Note that in a periodic game, the iterative matrices of learning dynamics are also periodic, which means that the learning difference systems of these learning dynamics are periodic systems. According to Floquet theorem, see proposition 2.4, the study of dynamical behaviors of a periodic system can be reduced to the autonomous system whose asymptotic behavior is determined by ∏ T t =1 A t .

The key point on the proof of Theorem 3.1 is an observation that the iterative matrix of extra-gradient is a normal matrix, which makes it possible to calculate the Jordan normal form of ∏ T t =1 A t for arbitrary large T . The details of proof are left to Appendix B.

In the following theorem, we provide an example demonstrating that when two players use the optimistic gradient descent ascent or negative momentum method, their strategy will diverge at an exponential rate, regardless of how they choose their step-sizes and momentum parameters.

Theorem 3.2. Consider a periodic game with period T = 2 , and described by the following payoff matrix

with x t ∈ R , y t ∈ R 2 . If two players use optimistic gradient descent ascent or negative momentum method, then regardless of how they choose step sizes and momentum parameters, we have

Here λ is determined by the largest modulus of the eigenvalues of the iterative matrix of optimistic gradient descent ascent or negative momentum method.

To prove theorem 3.2, we directly calculate the characteristic polynomials of the iterative matrices products in one period for optimistic gradient descent ascent and negative momentum method under the game defined by (6). To show that these systems have an exponential divergence rate, it is sufficient to demonstrate that their characteristic polynomials have a root with modulus larger than 1 . We achieve this by using the Schur stable theorem, which is also employed to demonstrate the last iterate convergence of several learning dynamics in time-independent game (Zhang and Yu [2020]). The proof is deferred to Appendix C.

In Figure (1), we present the function curves of ∆ 1 ,t for these three types of game dynamics under the periodic game defined by (6). From the experimental results, extra-gradient converges, while both optimistic gradient descent ascent and negative momentum method diverge.

Extra gradient method

Optimistic gradient method

Momentum method

2.00

1.75

step size = 0.015

step size = 0.01

step size = 0.008

800

step size = 0.011

step size = 0.01

step size = 0.012

5000

4000

beta_1 = -0.7, beta_2 = -0.7, step size = 0.014

beta_1 = -0.5, beta_2 = 0, step size = 0.015

beta_1 = -0.6, beta_2 = -0.4, step size = 0.014

1.50

600

1.25

3000

1.00

400

0.75

2000

0.50

200

1000

0.25

000

0

0

0

0009

10000

15000

20000

25000

30000

0

1000

2000

3000

4000

5000

6000

7000

0

1000

2000

3000

4000

5000

0009

7000

3.3 Convergent perturbed game

Recall that the payoff matrix of a convergent perturbed game has form A t = A + B t , where lim t →∞ B t = 0 , and we refer to the zero-sum game defined by payoff matrix A as the stable game. We denote

thus ∆ t measures the distance between the strategy ( x t , y t ) in the t -th round and the Nash equilibrium of the stable game defined by A . Moreover, ∆ t = 0 if and only if ( x t , y t ) is an equilibrium of the stable game.

In the literature on linear difference systems, a common assumption that needs to be added for convergence guarantee is the following bounded accumulated perturbations (BAP) assumption (Benzaid and Lutz [1987], Elaydi et al. [1999], Elaydi and Györi [1995]):

In the following theorem, we prove that under BAP assumption, all three learning dynamics considered in this paper will make ∆ t converge to 0 , with a rate dependent on the vanishing rate of B t .

Theorem 3.3. Assume that the (BAP assumption) holds, i.e., ∑ ∞ t =0 ‖ B t ‖ 2 is bounded, and let σ be the maximum modulus of the singular value of payoff matrix A , then with parameters choice:

for extra-gradient with step size α = η < 1 2 σ ,

for optimistic gradient descent ascent with step size η < 1 2 σ ,

for negative momentum method with step size η < 1 σ and momentum parameters β 1 = -1 2 and β 2 = 0 ,

we have ∆ t converge to 0 with rate O ( f ( t )) . Here

and λ ∈ (0 , 1) is determined by the eigenvalues of the iterative matrix of corresponding learning dynamics and the payoff matrix A of the stable game.

There are two main ingredients in the proof of Theorem 3.3: firstly, we show that the iterative matrices associated with these learning dynamics are diagonalizable; secondly, these matrices do not have eigenvalues with modulus larger than 1 . Moreover, we prove a general results which states any linear difference system satisfying these two conditions will converge. The details of proof are left to Appendix E.

Remark 3.2. The (BAP assumption) can be converted into a constraint on the vanishing rate of B t : if B t has a vanishing rate like O ( 1 t 1+ /epsilon1 ) , for some arbitrary small /epsilon1 > 0 , then ∑ ∞ t =0 ‖ B t ‖ 2 is bounded. We also note that the condition for B t to vanish at a rate O ( 1 t 1+ /epsilon1 ) , ∀ /epsilon1 > 0 is necessary to ensure convergence in general linear difference system. For example, consider the 1-dimension system x t = (1 + 1 t ) x t -1 where a O ( 1 t ) convergence rate of perturbations leads to x t diverging with a Ω ( t ) rate.

Surprisingly, in the next theorem, we prove that Extra-gradient makes ∆ t asymptotically converge to 0 , without making any further assumptions about the converge rate of B t .

Theorem 3.4. In a convergent perturbed game, if two players use Extra-gradient, there holds lim t →∞ ∆ t = 0 with step size α = η < 1 2 σ where σ is the maximum modulus of the singular value of payoff matrix A .

To prove Theorem 3.4, we first observe that ‖ x t ‖ 2 + ‖ y t ‖ 2 a non-increasing function of t since the iterative matrix of extra-gradient is a normal matrix. Next, we prove that if Theorem 3.4 doesn't hold, then ‖ x t ‖ 2 + ‖ y t ‖ 2 will decrease by a constant for infinite number of times, thus leading to a contradiction with the non-increasing property and the non-negativity of ‖ x t ‖ 2 + ‖ y t ‖ 2 . The proof is deferred to Appendix F.

4 Experiments

In this section we present numerical results for our theoretical results in Section 3.

Experiments on Theorem 3.1 We verify Theorem 3.1 through a period3 game, the payoff matrices are chosen to be

We run extra-gradient and optimistic gradient descent ascent on this example, both with step size = 0.01, the experimental results are presented in Figure (2). We can see extra-gradient (left) makes ∆ i,t converge, while optimistic gradient descent ascent (right) makes ∆ i,t diverge. This result supports Theorem 3.1 and provides a numerical example of the separation of extra-gradient and optimistic gradient descent ascent in periodic games.

Delta_i,t of periodic game for EG

Delta_i,t of periodic game for OGDA

800

Delta_1,t

400000

Delta_1,t

700

Delta_2,t

Delta_2,t

Delta_3,t

350000

Delta_3,t

600

300000

500

250000

400

200000

300

150000

200

100000

100

50000

0

0

0

500

1000

1500

2000

2500

3000

3500

0

1000

2000

3000

4000

5000

6000

7000

Experiments on Theorem 3.3 We verify Theorem 3.3 by examples:

where B = [[ -15 , 70] , [ -90 , 90]] . The step size is chosen to be 0 . 005 . The initial points are chosen to be x 0 = [15 , 13] , x -1 = [11 , 3] and y 0 = [35 , 1] , y -1 = [35 , 1] . The experimental results are presented in Figure 3, all of the three dynamics make ∆ t converge to 0 , and slower convergence rate of perturbations can decelerate the convergence rate of learning dynamics, thus support the convergence result in Theorem 3.3. We also observe that the convergence rate is faster than the upper bound provided in the theorem. Therefore, we conjecture the bound of convergence rate can be improved.

Delta_t for extra gradient

Delta_t for optimistc gradient descent ascent

Delta_t for negative momentum method

O(1/t(1.1)

80000

O(1/(1.1))

100000

O(1/t(1.1)

80000

O(1/(4)

70000

O(1/(4)

O(1/(4))

O(1/(8)

((8)/)

((8)/L)

60000

80000

60000

50000

60000

40000

40000

30000

40000

20000

20000

20000

10000

0

0

0

0

500

1000

1500

2000

2500

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

Experiments on Theorem 3.4 We verify Theorem 3.4 by two group of examples. The perturbations are :

and

where B = [[ -10 , 10] , [ -10 , 10]] . The payoff matrix of stable game is chosen to be A = [[2 , 3] , [4 , 6]] . Note that the perturbations do not satisfy (BAP assumption). The experimental results are shown in Figure (4). We can see all these curves converge to 0 , thus support the result in Theorem 3.4. Furthermore, we can observe that large perturbations may lead to more oscillations during the convergence processes, which in turn slows down the rate of convergence. We present more experiments in Appendix G.

convergent curve of Delta_t

convergent curve of Delta_t

200

O(1/**0.4)

250

O(1/log(t)**1.8)

175

O(1/t**0.3)

O(1/log(t)**1.5)

O(1/t**0.2)

O(1/log(t)**1.3)

150

200

125

150

100

75

100

50

50

25

0

0

500

1000

1500

2000

2500

3000

0

200

400

600

800

5 Discussion

In this paper, we study the last-iterate behavior of extra-gradient, optimistic gradient descent ascent, and negative momentum method in two types of time-varying games : periodic game and convergent perturbed game. In the case of periodic game, we prove that extra-gradient will converge to a Nash equilibrium while other two methods diverge. To the best of our knowledge, this is the first result that provides a clear separation between the behavior of extra-gradient methods and optimistic gradient descent ascent. In the case of convergent perturbed game, we prove all three learning dynamics converge to Nash equilibrium under the BAP assumption, which is commonly used in the literature on dynamical systems.

Our results leave many interesting open questions. Firstly, is the BAP assumption necessary for ensuring convergence in optimistic gradient descent ascent and negative momentum method? Secondly, the bound of convergence rate in Theorem 3.3 may rather slight as we shown in experiments section. Obtaining a tighter bound on the convergence rate is an important future research problem. Thirdly, can the results here be generalized to other settings, such as constrained zero-sum game?

References

Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On the convergence of no-regret learning dynamics in time-varying games. arXiv preprint arXiv:2301.11241 , 2023.

Z Benzaid and DA Lutz. Asymptotic representation of solutions of perturbed systems of linear difference equations. Studies in Applied Mathematics , 77(3):195-221, 1987.

David Blackwell. An analog of the minimax theorem for vector payoffs. In Pacific J. Math. , pages 1-8, 1956.

George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation , 13(1): 374, 1951.

Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Finite-time last-iterate convergence for learning in multi-player games. In NeurIPS , 2022.

Adrian Rivera Cardoso, Jacob D. Abernethy, He Wang, and Huan Xu. Competing against nash equilibria in adversarially changing zero-sum games. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019 , volume 97 of Proceedings of Machine Learning Research , pages 921-930. PMLR, 2019.

Fritz Colonius and Wolfgang Kliemann. Dynamical systems and linear algebra , volume 158. American Mathematical Society, 2014.

Constantinos Daskalakis and Ioannis Panageas. The limit points of (optimistic) gradient descent in min-max optimization. Advances in neural information processing systems , 31, 2018.

Constantinos Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In Avrim Blum, editor, 10th Innovations in Theoretical Computer Science Conference, ITCS 2019, January 10-12, 2019, San Diego, California, USA , volume 124 of LIPIcs , pages 27:1-27:18. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2019.

Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training GANs with Optimism. In Proceedings of ICLR , 2018.

Benoit Duvocelle, Panayotis Mertikopoulos, Mathias Staudigl, and Dries Vermeulen. Multiagent online learning in time-varying games. Mathematics of Operations Research , 2022.

S Elaydi and I Györi. Asymptotic theory for delay differene equations. Journal of Difference Equations and Applications , 1(2):99-116, 1995.

Saber Elaydi, Satoru Murakami, and Etsuyo Kamiyama. Asymptotic equivalence for difference equations with infinite delay. Journal of Difference Equations and Applications , 5(1):1-23, 1999.

Tanner Fiez, Ryann Sim, Stratis Skoulakis, Georgios Piliouras, and Lillian Ratliff. Online learning in periodic zero-sum games. Advances in Neural Information Processing Systems , 34, 2021.

Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Rémi Le Priol, Gabriel Huang, Simon Lacoste-Julien, and Ioannis Mitliagkas. Negative momentum for improved game dynamics. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1802-1811. PMLR, 2019.

Eduard Gorbunov, Nicolas Loizou, and Gauthier Gidel. Extragradient method: O(1/K) last-iterate convergence for monotone variational inequalities and connections with cocoercivity. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event , volume 151 of Proceedings of Machine Learning Research , pages 366-402. PMLR, 2022a.

Eduard Gorbunov, Adrien Taylor, and Gauthier Gidel. Last-iterate convergence of optimistic gradient method for monotone variational inequalities. In NeurIPS , 2022b.

Galina M Korpelevich. The extragradient method for finding saddle points and other problems. Matecon , 12:747-756, 1976.

Tengyuan Liang and James Stokes. Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks. AISTATS , 2019a.

Tengyuan Liang and James Stokes. Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 907-915. PMLR, 2019b.

Tung Mai, Milena Mihail, Ioannis Panageas, Will Ratcliff, Vijay V. Vazirani, and Peter Yunker. Cycles in zero-sum differential games and biological diversity. In Éva Tardos, Edith Elkind, and Rakesh Vohra, editors, Proceedings of the 2018 ACM Conference on Economics and Computation, Ithaca, NY, USA, June 18-22, 2018 , pages 339-350. ACM, 2018.

Panayotis Mertikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasekhar, and Georgios Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Renato DC Monteiro and Benar Fux Svaiter. On the complexity of the hybrid proximal extragradient method for the iterates and the ergodic mean. SIAM Journal on Optimization , 20(6):2755-2787, 2010.

Arkadi Nemirovski. Prox-method with rate of convergence o (1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1):229-251, 2004.

Yurii Nesterov. Smooth minimization of non-smooth functions. Math. Program. , 103(1):127-152, 2005.

Mihály Pituk. More on poincaré's and perron's theorems for difference equations. The Journal of Difference Equations and Applications , 8(3):201-216, 2002.

Leonid Denisovich Popov. A modification of the arrow-hurwicz method for search of saddle points. Mathematical notes of the Academy of Sciences of the USSR , 28:845-848, 1980.

Abhishek Roy, Yifang Chen, Krishnakumar Balasubramanian, and Prasant Mohapatra. Online and bandit algorithms for nonstationary stochastic saddle-point optimization. CoRR , abs/1912.01698, 2019.

Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E. Schapire. Fast convergence of regularized learning in games. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 2989-2997, 2015.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Guodong Zhang and Yuanhao Wang. On the suboptimality of negative momentum for minimax optimization. In International Conference on Artificial Intelligence and Statistics , pages 2098-2106. PMLR, 2021.

Guojun Zhang and Yaoliang Yu. Convergence of gradient methods on bilinear zero-sum games. In International Conference on Learning Representations , 2020.

Mengxiao Zhang, Peng Zhao, Haipeng Luo, and Zhi-Hua Zhou. No-regret learning in time-varying zero-sum games. In International Conference on Machine Learning, ICML 2022 , volume 162 of Proceedings of Machine Learning Research , pages 26772-26808. PMLR, 2022.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Tom Fawcett and Nina Mishra, editors, Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), August 21-24, 2003, Washington, DC, USA , pages 928-936. AAAI Press, 2003.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of unconstrained min-max optimization with bilinear structure. Specifically, this paper considers the setting where the payoff matrix $A_t$ changes over time. The two changing dynamics of $A_t$ considered in this paper is the periodic game and the converging perturbed game. Three strategy dynamics are cosidered in this paper, which is OGDA, extra gradient (EG) and Negative momentum method. In this paper, the authors show that in the period game case, under certain initialization, OGDA provably diverges no matter what learning rate is chosen while EG with a certain choice of learning rate converges to the common NE with exponential rate. In the convergent perturbed game case, all the three types of algorithms converge with a proper choice of learning rate. The analysis is basd on analyzing the eigenvalues of the linear operator in the recursive form of the strategies. Empirical results further verifies their theoretical results.\n\nSTRENGTHS:\n- The min-max optimization with time-varying functions is an important question and has wide real-world applications.\n- This paper first considers the last-iterate convergence performance for the unconstrained bilinear min-max problem and show that under two different changing dynamics of the bilinear matrix, OGDA performs provably differently and EG always converges. Specifically, the diverging results for OGDA looks intereting to me.\n- I did not check all the proof details but the general idea of the proof makes sense to me.\n\nWEAKNESSES:\n- One issue is whether the optimization of unconstrained min-max optimization with bilinear structure is interesting. Specifically, even when the game is changing over time, one trivial Nash Equilibrium is $(0,0)$. Therefore, all the three algorithms initialized with $(0,0)$ will converge, though the dynamic of these three algorithms may still be interesting with different initializations. Extending the current framework to a slightly general case $x^\\top A_ty+b_t^\\top x+c_t^\\top y$ or an even more general convex-concave function $f(x,y)$ will strengthen the results a lot.\n- As the authors mentioned, the constrained version of the min-max optimization problem may be more interesting and the dynamic of the strategy can not be expressed in a clean linear recurrsive form as shown in the unconstrained bilinear case.\n\nQUESTIONS:\n- As mentioned in weakness, whether the current result can be extended to a generalized linear case with $f_t(x,y)=x^\\top A_ty+b_t^\\top x+c_t^\\top y$?\n- Is there a specific reason considering the periodic games and the convergent perturbed games? Can the results generalize to some other type of time-varying games with common NE?\n\nLIMITATIONS:\nSee weakness and questions for details.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-18"}

comments: SUMMARY:
This paper studies the last-iterate behavior of three different algorithms on two types of time-varying zero-sum games: periodic and convergent perturbed games. For periodic games, the authors prove that EG will converge while OGDA and the negative momentum method could diverge. For convergent perturbed games, all these algorithms converge as long as the error term decays sufficiently fast.

STRENGTHS:
1. The paper is well-organized and easy to follow.
2. The different behavior between EG and OGDA seems novel in the literature.

WEAKNESSES:
1. Some results lack an intuitive explanation.
2. Some figures are not well-plotted. Maybe a log-log graph is more proper.

QUESTIONS:
1. Could the authors provide some examples of periodic and convergent perturbed games?

2. In line 225, the authors say that the key point on the convergence of EG for periodic games is that the iterative matrix is a normal matrix. Could the authors give a more intuitive explanation? Moreover, the update rule of EG only depends on the current point $(x_t, y_t)$, while OGDA and negative momentum method also need to employ the gradient information at the last point $(x_{t-1}, y_{t-1})$. Is this a reason for the different behavior of these methods?

3. For the experiments on Theorem 3.4, the authors only plot the convergent curve of EG. I wonder about the behabior of the other two methods in this case.

LIMITATIONS:
See Questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper studies the problem of learning Nash equilibria in two-player zero-sum bilinear games where the payoff matrix varies with time.
When the payoff matrix is a periodic function, it is proven that the extra gradient algorithm converges to a Nash equilibrium, whereas the optimistic gradient descent ascent and negative momentum method could diverge from the equilibrium.
Furthermore, it is shown that all the above three algorithms converge to the equilibrium for the convergent perturbed game.

STRENGTHS:
* The paper is well-written and studies an interesting problem. 
* The author provided an instance of periodic zero-sum games where extra gradient algorithms and optimistic gradient descent ascent algorithms behave distinctly differently.
* The analysis looks to be sound, and the experimental results seem to bare out the theory.

WEAKNESSES:
* Perhaps a short intuitive explanation of why extra gradient is better than optimistic gradient descent ascent would help readers understand the main results of this paper.
* The titles and legends in Figures 1,2,3 and 4 should be enlarged.

QUESTIONS:
* (Section 2.1) What is the definition of $\mathrm{ker}(\cdot)$?
* (Section 3.2) What are the key differences between the extra gradient algorithm and the other two algorithms?
* (Section 3.2) For the periodic games with $\lambda_{\ast}=0$, would similar convergence/divergence results hold to Theorem 3.1 and 3.2?
* (Section 4) Do the optimistic gradient descent ascent algorithm and negative momentum method empirically converge in the same instance in the experiments for Figure 4?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOver the last few years an extensive literature has studied the last iterate convergence of learning dynamics in zero-sum games, particularly those using optimism and extra-gradient approaches.  This paper extends that approach to two classes of unconstrained non-stationary game (periodic and decaying noise) and shows a difference in behavior between the two classes.\n\nSTRENGTHS:\nThe problem studied is natural and clearly explained with strong results, including the interesting split in behavior (Theorems 3.1 and 3.2)\n\nThe approach uses several technical innovations derived from the literature on non-autonomous linear difference systems.\n\nWEAKNESSES:\nThe particular classes of games studied could be better motivated.  Is there a reason to focus on these other than technical convenience?  Do they have important applications?  What more general but harder class is this a step towards?  The conclusion mentions constrained games as future work, nothing about other interesting directions fof future work.  Are there other theorems from non-autonomous linear difference systems that seem promising?\n\nThere is a gap left by Theorems 3.3 and 3.4 about whether the other two dynamics converge under the weaker conditions of Theorem 3.4.  Figure 4 gives an experiment on this setting, so it seems like a missed opportunity to not at least show what the other dynamics do on this example.  \n\nMinor comments:\n- There is some inconsistency in the way “equilibrium” is used.  It is defined on line 98 as what I would instead have called the set of equilibria.  However, in other places, like line 252 it seems to be used to refer to a single element of this set.  Depending on how this is resolved, some care may be needed in places about whether something is “the” or “a” equilibrium (e.g. on line 216).\n- Line 178 and 181 the formulas use n but the limits are taken over t\n- Line 216 the “common Nash equilibrium” is not defined, nor is there an explanation of why it mush exist (presumably because 0 is always an equilibrium?)\n- Line 305 group[s]\n\nQUESTIONS:\nPlease respond regarding the two main weaknesses.\n\nLIMITATIONS:\nThe conditions to which the results apply are clearly explained.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates the problem of last-iterate convergence in time-varying zero-sum games. Specifically, the authors study the last-iterate convergence of three kinds of algorithms (OGDA, EG, and negative momentum method) considering two kinds of time-varying games with specific structures, i.e., periodic games and convergent perturbed games. The authors obtained a convergence rate of EG in periodic games while providing a counterexample showing that OGDA and negative momentum method will diverge in this kind of game. In convergent perturbed games, the authors showed that all three kinds of algorithm will converge with a rate dependent on the perturbation $B_t$. Finally, experiments validate the statements proposed by the authors.\n\nSTRENGTHS:\nIn general, I think there is no last-iterate convergence rate in time-varying games due to the changing nature of the games. However, with specific problem structures, I believe that last-iterate convergence results can be established, and this work provides a piece of clear evidence. Overall, although the solutions are relatively simple by modeling the learning algorithms as dynamical systems and leveraging some existing results in the control theory, I am satisfied with the motivation and the final results of this work.\n\nWEAKNESSES:\nThere are no significant weaknesses that affect my rating of the whole paper. In the following, I only list some minor issues.\n\n1. The notion of $T$ in periodic games. The notion of $T$ is commonly used to refer to the total number of rounds in online learning. In this work, the authors use $T$ to represent the number of rounds inside each period, which may lead to unnecessary understandings. I suggest the authors could choose a different notion to denote the period length.\n2. The notion of $\\text{ker}(\\cdot)$. This notion, which appears in Line 98 for the first time, seems not to be defined or given a clear description before.\n3. In Line 120, the authors mention that 'Recently, there are also works analyzing the regret behaviors of OGDA under a time-varying setting [1]'. The reference to [1] is not accurate enough. The work of [33], which is published at ICML2022, firstly gives a comprehensive study of the optimistic methods in time-varying zero-sum games by optimizing multiple performance measures simultaneously. I suggest the authors could refer to [33] in the aforementioned statement to make the credits more accurate.\n4. The negative momentum method requires the $x$-player to evolve to the next round first to give its gradient $A_{t+1} x_{t+1}$ to the $y$-player. In some cases where the learning procedure is strictly in an online style, this algorithm is NOT applicable since both players must act first to make the game evolve. I suggest the authors could give a discussion on this point in the revised version.\n\nTypos:\n1. Line 89: 'Section 4' not 'section 4'.\n2. Line 188: 'Section 3.3' not 'Section3.3'.\n3. The reference to theorems are not unified, e.g., in Line 237, 'theorem 3.2', and in Line 255, 'Theorem (3.1)'.\n\nQUESTIONS:\nI only have a major question on Theorem 3.1. What is the range of $t$? If $t$ is only chosen in $[1,T]$, choosing $t=T$ yields a convergence rate of $O(\\lambda_* \\text{poly}(T))$, which can be seen as a constant in terms of $T$. As mentioned in the 'Weaknesses' part, a bound linear in $T$ is pretty bad in the standard online learning convention. Theorem 3.1 exhibits an exponential rate only when $t$ can be significantly larger than the period length. Can the authors give some further explanations on this point?\n\nIn the end, I am very curious about whether the method in this paper, which models the learning algorithms as dynamical systems, can be applied to the constrained case, where the projection operations may bring some unique challenges to the modeling issue. Can the authors give some further explanations on this point?\n\nLIMITATIONS:\nPlease mainly refer to the 'Weaknesses' part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-22"}