Convergence of Gradient Methods on Bilinear Zero-Sum Games

1 INTRODUCTION . Min-max optimization has received significant attention recently due to the popularity of generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) , adversarial training ( Madry et al. , 2018 ) and reinforcement learning ( Du et al. , 2017 ; Dai et al. , 2018 ) , just to name some examples . Formally , given a bivariate function f ( x , y ) , we aim to find a saddle point ( x∗ , y∗ ) such that f ( x∗ , y ) ≤ f ( x∗ , y∗ ) ≤ f ( x , y∗ ) , ∀x ∈ Rn , ∀y ∈ Rn . ( 1.1 ) Since the beginning of game theory , various algorithms have been proposed for finding saddle points ( Arrow et al. , 1958 ; Dem ’ yanov & Pevnyi , 1972 ; Gol ’ shtein , 1972 ; Korpelevich , 1976 ; Rockafellar , 1976 ; Bruck , 1977 ; Lions , 1978 ; Nemirovski & Yudin , 1983 ; Freund & Schapire , 1999 ) . Due to its recent resurgence in ML , new algorithms specifically designed for training GANs were proposed ( Daskalakis et al. , 2018 ; Kingma & Ba , 2015 ; Gidel et al. , 2019b ; Mescheder et al. , 2017 ) . However , due to the inherent non-convexity in deep learning formulations , our current understanding of the convergence behaviour of new and classic gradient algorithms is still quite limited , and existing analysis mostly focused on bilinear games or strongly-convex-strongly-concave games ( Tseng , 1995 ; Daskalakis et al. , 2018 ; Gidel et al. , 2019b ; Liang & Stokes , 2019 ; Mokhtari et al. , 2019b ) . Nonzero-sum bilinear games , on the other hand , are known to be PPAD-complete ( Chen et al. , 2009 ) ( for finding approximate Nash equilibria , see e.g . Deligkas et al . ( 2017 ) ) . In this work , we study bilinear zero-sum games as a first step towards understanding general min-max optimization , although our results apply to some simple GAN settings ( Gidel et al. , 2019a ) . It is well-known that certain gradient algorithms converge linearly on bilinear zero-sum games ( Liang & Stokes , 2019 ; Mokhtari et al. , 2019b ; Rockafellar , 1976 ; Korpelevich , 1976 ) . These iterative algorithms usually come with two versions : Jacobi style updates or Gauss–Seidel ( GS ) style . In a Jacobi style , we update the two sets of parameters ( i.e. , x and y ) simultaneously whereas in a GS style we update them alternatingly ( i.e. , one after the other ) . Thus , Jacobi style updates are naturally amenable to parallelization while GS style updates have to be sequential , although the latter is usually found to converge faster ( and more stable ) . In numerical linear algebra , the celebrated Stein–Rosenberg theorem ( Stein & Rosenberg , 1948 ) formally proves that in solving certain linear systems , GS updates converge strictly faster than their Jacobi counterparts , and often with a larger set of convergent instances . However , this result does not readily apply to bilinear zero-sum games . Our main goal here is to answer the following questions about solving bilinear zero-sum games : • When exactly does a gradient-type algorithm converge ? Contributions We summarize our main results from §3 and §4 in Table 1 and 2 respectively , with supporting experiments given in §5 . We use σ1 and σn to denote the largest and the smallest singular values of matrixE ( see equation 2.1 ) , and κ : = σ1/σn denotes the condition number . The algorithms will be introduced in §2 . Note that we generalize gradient-type algorithms but retain the same names . Table 1 shows that in most cases that we study , whenever Jacobi updates converge , the corresponding GS updates converge as well ( usually with a faster rate ) , but the converse is not true ( §3 ) . This extends the well-known Stein–Rosenberg theorem to bilinear games . Furthermore , Table 2 tells us that by generalizing existing gradient algorithms , we can obtain faster convergence rates . 2 PRELIMINARIES . In the study of GAN training , bilinear games are often regarded as an important simple example for theoretically analyzing and understanding new algorithms and techniques ( e.g . Daskalakis et al. , 2018 ; Gidel et al. , 2019a ; b ; Liang & Stokes , 2019 ) . It captures the difficulty in GAN training and can represent some simple GAN formulations ( Arjovsky et al. , 2017 ; Daskalakis et al. , 2018 ; Gidel et al. , 2019a ; Mescheder et al. , 2018 ) . Mathematically , bilinear zero-sum games can be formulated as the following min-max problem : minx∈Rn maxy∈Rn x > Ey + b > x+ c > y . ( 2.1 ) The set of all saddle points ( see definition in eq . ( 1.1 ) ) is : { ( x , y ) |Ey + b = 0 , E > x+ c = 0 } . ( 2.2 ) Throughout , for simplicity we assume E to be invertible , whereas the seemingly general case with non-invertible E is treated in Appendix G. The linear terms are not essential in our analysis and we take b = c = 0 throughout the paper1 . In this case , the only saddle point is ( 0,0 ) . For bilinear games , it is well-known that simultaneous gradient descent ascent does not converge ( Nemirovski & Yudin , 1983 ) and other gradient-based algorithms tailored for min-max optimization have been proposed ( Korpelevich , 1976 ; Daskalakis et al. , 2018 ; Gidel et al. , 2019a ; Mescheder et al. , 2017 ) . These iterative algorithms all belong to the class of general linear dynamical systems ( LDS , a.k.a . 1If they are not zero , one can translate x and y to cancel the linear terms , see e.g . Gidel et al . ( 2019b ) . matrix iterative processes ) . Using state augmentation z ( t ) : = ( x ( t ) , y ( t ) ) we define a general k-step LDS as follows : z ( t ) = ∑k i=1Aiz ( t−i ) + d , ( 2.3 ) where the matrices Ai and vector d depend on the gradient algorithm ( examples can be found in Appendix C.1 ) . Define the characteristic polynomial , withA0 = −I : p ( λ ) : = det ( ∑k i=0Aiλ k−i ) . ( 2.4 ) The following well-known result decides when such a k-step LDS converges for any initialization : Theorem 2.1 ( e.g . Gohberg et al . ( 1982 ) ) . The LDS in eq . ( 2.3 ) converges for any initialization ( z ( 0 ) , . . . , z ( k−1 ) ) iff the spectral radius r : = max { |λ| : p ( λ ) = 0 } < 1 , in which case { z ( t ) } converges linearly with an ( asymptotic ) exponent r. Therefore , understanding the bilinear game dynamics reduces to spectral analysis . The ( sufficient and necessary ) convergence condition reduces to that all roots of p ( λ ) lie in the ( open ) unit disk , which can be conveniently analyzed through the celebrated Schur ’ s theorem ( Schur , 1917 ) : Theorem 2.2 ( Schur ( 1917 ) ) . The roots of a real polynomial p ( λ ) = a0λn + a1λn−1 + · · ·+ an are within the ( open ) unit disk of the complex plane iff ∀k ∈ { 1 , 2 , . . . , n } , det ( PkP > k −Q > kQk ) > 0 , where Pk , Qk are k × k matrices defined as : [ Pk ] i , j = ai−j1i≥j , [ Qk ] i , j = an−i+j1i≤j . In the theorem above , we denoted 1S as the indicator function of the event S , i.e . 1S = 1 if S holds and 1S = 0 otherwise . For a nice summary of related stability tests , see Mansour ( 2011 ) . We therefore define Schur stable polynomials to be those polynomials whose roots all lie within the ( open ) unit disk of the complex plane . Schur ’ s theorem has the following corollary ( proof included in Appendix B.2 for the sake of completeness ) : Corollary 2.1 ( e.g . Mansour ( 2011 ) ) . A real quadratic polynomial λ2 + aλ + b is Schur stable iff b < 1 , |a| < 1 + b ; A real cubic polynomial λ3 + aλ2 + bλ + c is Schur stable iff |c| < 1 , |a+ c| < 1 + b , b− ac < 1− c2 ; A real quartic polynomial λ4 + aλ3 + bλ2 + cλ+ d is Schur stable iff |c− ad| < 1− d2 , |a+ c| < b+ d+ 1 , and b < ( 1 + d ) + ( c− ad ) ( a− c ) / ( d− 1 ) 2 . Let us formally define Jacobi and GS updates : Jacobi updates take the form x ( t ) = T1 ( x ( t−1 ) , y ( t−1 ) , . . . , x ( t−k ) , y ( t−k ) ) , y ( t ) = T2 ( x ( t−1 ) , y ( t−1 ) , . . . , x ( t−k ) , y ( t−k ) ) , while Gauss–Seidel updates replace x ( t−i ) with the more recent x ( t−i+1 ) in operator T2 , where T1 , T2 : Rnk × Rnk → Rn can be any update functions . For LDS updates in eq . ( 2.3 ) we find a nice relation between the characteristic polynomials of Jacobi and GS updates in Theorem 2.3 ( proof in Appendix B.1 ) , which turns out to greatly simplify our subsequent analyses : Theorem 2.3 ( Jacobi vs. Gauss–Seidel ) . Let p ( λ , γ ) = det ( ∑k i=0 ( γLi +Ui ) λ k−i ) , whereAi = Li + Ui and Li is strictly lower block triangular . Then , the characteristic polynomial of Jacobi updates is p ( λ , 1 ) while that of Gauss–Seidel updates is p ( λ , λ ) . Compared to the Jacobi update , in some sense the Gauss–Seidel update amounts to shifting the strictly lower block triangular matrices Li one step to the left , as p ( λ , λ ) can be rewritten as det ( ∑k i=0 ( Li+1 +Ui ) λ k−i ) , with Lk+1 : = 0 . This observation will significantly simplify our comparison between Jacobi and Gauss–Seidel updates . Next , we define some popular gradient algorithms for finding saddle points in the min-max problem min x max y f ( x , y ) . ( 2.5 ) We present the algorithms for a general ( bivariate ) function f although our main results will specialize f to the bilinear case in eq . ( 2.1 ) . Note that we introduced more “ step sizes ” for our refined analysis , as we find that the enlarged parameter space often contains choices for faster linear convergence ( see §4 ) . We only define the Jacobi updates , while the GS counterparts can be easily inferred . We always use α1 and α2 to define step sizes ( or learning rates ) which are positive . Gradient descent ( GD ) The generalized GD update has the following form : x ( t+1 ) = x ( t ) − α1∇xf ( x ( t ) , y ( t ) ) , y ( t+1 ) = y ( t ) + α2∇yf ( x ( t ) , y ( t ) ) . ( 2.6 ) When α1 = α2 , the convergence of averaged iterates ( a.k.a . Cesari convergence ) for convex-concave games is analyzed in ( Bruck , 1977 ; Nemirovski & Yudin , 1978 ; Nedić & Ozdaglar , 2009 ) . Recent progress on interpreting GD with dynamical systems can be seen in , e.g. , Mertikopoulos et al . ( 2018 ) ; Bailey et al . ( 2019 ) ; Bailey & Piliouras ( 2018 ) . Extra-gradient ( EG ) We study a generalized version of EG , defined as follows : x ( t+1/2 ) = x ( t ) − γ1∇xf ( x ( t ) , y ( t ) ) , y ( t+1/2 ) = y ( t ) + γ2∇yf ( x ( t ) , y ( t ) ) ; ( 2.7 ) x ( t+1 ) = x ( t ) − α1∇xf ( x ( t+1/2 ) , y ( t+1/2 ) ) , y ( t+1 ) = y ( t ) + α2∇yf ( x ( t+1/2 ) , y ( t+1/2 ) ) . ( 2.8 ) EG was first proposed in Korpelevich ( 1976 ) with the restriction α1 = α2 = γ1 = γ2 , under which linear convergence was proved for bilinear games . Convergence of EG on convex-concave games was analyzed in Nemirovski ( 2004 ) ; Monteiro & Svaiter ( 2010 ) , and Mertikopoulos et al . ( 2019 ) provides convergence guarantees for specific non-convex-non-concave problems . For bilinear games , a slightly more generalized version was proposed in Liang & Stokes ( 2019 ) where α1 = α2 , γ1 = γ2 , with linear convergence proved . For later convenience we define β1 = α2γ1 and β2 = α1γ2 . Optimistic gradient descent ( OGD ) We study a generalized version of OGD , defined as follows : x ( t+1 ) = x ( t ) − α1∇xf ( x ( t ) , y ( t ) ) + β1∇xf ( x ( t−1 ) , y ( t−1 ) ) , ( 2.9 ) y ( t+1 ) = y ( t ) + α2∇yf ( x ( t ) , y ( t ) ) − β2∇yf ( x ( t−1 ) , y ( t−1 ) ) . ( 2.10 ) The original version of OGD was given in Popov ( 1980 ) with α1 = α2 = 2β1 = 2β2 and rediscovered in the GAN literature ( Daskalakis et al. , 2018 ) . Its linear convergence for bilinear games was proved in Liang & Stokes ( 2019 ) . A slightly more generalized version with α1 = α2 and β1 = β2 was analyzed in Peng et al . ( 2019 ) ; Mokhtari et al . ( 2019b ) , again with linear convergence proved . The stochastic case was analyzed in Hsieh et al . ( 2019 ) . Momentum method Generalized heavy ball method was analyzed in Gidel et al . ( 2019b ) : x ( t+1 ) = x ( t ) − α1∇xf ( x ( t ) , y ( t ) ) + β1 ( x ( t ) − x ( t−1 ) ) , ( 2.11 ) y ( t+1 ) = y ( t ) + α2∇yf ( x ( t ) , y ( t ) ) + β2 ( y ( t ) − y ( t−1 ) ) . ( 2.12 ) This is a modification of Polyak ’ s heavy ball ( HB ) ( Polyak , 1964 ) , which also motivated Nesterov ’ s accelerated gradient algorithm ( NAG ) ( Nesterov , 1983 ) . Note that for both x-update and the y-update , we add a scale multiple of the successive difference ( e.g . proxy of the momentum ) . For this algorithm our result below improves those obtained in Gidel et al . ( 2019b ) , as will be discussed in §3 . EG and OGD as approximations of proximal point algorithm It has been observed recently in Mokhtari et al . ( 2019b ) that for convex-concave games , EG ( α1 = α2 = γ1 = γ2 = η ) and OGD ( α1/2 = α2/2 = β1 = β2 = η ) can be treated as approximations of the proximal point algorithm ( Martinet , 1970 ; Rockafellar , 1976 ) when η is small . With this result , one can show that EG and OGD converge to saddle points sublinearly for smooth convex-concave games ( Mokhtari et al. , 2019a ) . We give a brief introduction of the proximal point algorithm in Appendix A ( including a linear convergence result for the slightly generalized version ) . The above algorithms , when specialized to a bilinear function f ( see eq . ( 2.1 ) ) , can be rewritten as a 1-step or 2-step LDS ( see . eq . ( 2.3 ) ) . See Appendix C.1 for details .

Summary: The paper presents exact conditions for the convergence of several gradient based methods for solving bilinear games. In particular, the methods under study are Gradient Descent(GD), Extragradient (EG), Optimizatic Gradient descent (OGD) and Momentum methods. For these methods, the authors provide convergence rates (with optimal parameter setup) for both alternating (Gauss-Seidel) and simultaneous (Jacobi) updates.