Convergence of Gradient Methods on Bilinear Zero-Sum Games

1 INTRODUCTION . Min-max optimization has received significant attention recently due to the popularity of generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) , adversarial training ( Madry et al. , 2018 ) and reinforcement learning ( Du et al. , 2017 ; Dai et al. , 2018 ) , just to name some examples . Formally , given a bivariate function f ( x , y ) , we aim to find a saddle point ( x∗ , y∗ ) such that f ( x∗ , y ) ≤ f ( x∗ , y∗ ) ≤ f ( x , y∗ ) , ∀x ∈ Rn , ∀y ∈ Rn . ( 1.1 ) Since the beginning of game theory , various algorithms have been proposed for finding saddle points ( Arrow et al. , 1958 ; Dem ’ yanov & Pevnyi , 1972 ; Gol ’ shtein , 1972 ; Korpelevich , 1976 ; Rockafellar , 1976 ; Bruck , 1977 ; Lions , 1978 ; Nemirovski & Yudin , 1983 ; Freund & Schapire , 1999 ) . Due to its recent resurgence in ML , new algorithms specifically designed for training GANs were proposed ( Daskalakis et al. , 2018 ; Kingma & Ba , 2015 ; Gidel et al. , 2019b ; Mescheder et al. , 2017 ) . However , due to the inherent non-convexity in deep learning formulations , our current understanding of the convergence behaviour of new and classic gradient algorithms is still quite limited , and existing analysis mostly focused on bilinear games or strongly-convex-strongly-concave games ( Tseng , 1995 ; Daskalakis et al. , 2018 ; Gidel et al. , 2019b ; Liang & Stokes , 2019 ; Mokhtari et al. , 2019b ) . Nonzero-sum bilinear games , on the other hand , are known to be PPAD-complete ( Chen et al. , 2009 ) ( for finding approximate Nash equilibria , see e.g . Deligkas et al . ( 2017 ) ) . In this work , we study bilinear zero-sum games as a first step towards understanding general min-max optimization , although our results apply to some simple GAN settings ( Gidel et al. , 2019a ) . It is well-known that certain gradient algorithms converge linearly on bilinear zero-sum games ( Liang & Stokes , 2019 ; Mokhtari et al. , 2019b ; Rockafellar , 1976 ; Korpelevich , 1976 ) . These iterative algorithms usually come with two versions : Jacobi style updates or Gauss–Seidel ( GS ) style . In a Jacobi style , we update the two sets of parameters ( i.e. , x and y ) simultaneously whereas in a GS style we update them alternatingly ( i.e. , one after the other ) . Thus , Jacobi style updates are naturally amenable to parallelization while GS style updates have to be sequential , although the latter is usually found to converge faster ( and more stable ) . In numerical linear algebra , the celebrated Stein–Rosenberg theorem ( Stein & Rosenberg , 1948 ) formally proves that in solving certain linear systems , GS updates converge strictly faster than their Jacobi counterparts , and often with a larger set of convergent instances . However , this result does not readily apply to bilinear zero-sum games . Our main goal here is to answer the following questions about solving bilinear zero-sum games : • When exactly does a gradient-type algorithm converge ? Contributions We summarize our main results from §3 and §4 in Table 1 and 2 respectively , with supporting experiments given in §5 . We use σ1 and σn to denote the largest and the smallest singular values of matrixE ( see equation 2.1 ) , and κ : = σ1/σn denotes the condition number . The algorithms will be introduced in §2 . Note that we generalize gradient-type algorithms but retain the same names . Table 1 shows that in most cases that we study , whenever Jacobi updates converge , the corresponding GS updates converge as well ( usually with a faster rate ) , but the converse is not true ( §3 ) . This extends the well-known Stein–Rosenberg theorem to bilinear games . Furthermore , Table 2 tells us that by generalizing existing gradient algorithms , we can obtain faster convergence rates . 2 PRELIMINARIES . In the study of GAN training , bilinear games are often regarded as an important simple example for theoretically analyzing and understanding new algorithms and techniques ( e.g . Daskalakis et al. , 2018 ; Gidel et al. , 2019a ; b ; Liang & Stokes , 2019 ) . It captures the difficulty in GAN training and can represent some simple GAN formulations ( Arjovsky et al. , 2017 ; Daskalakis et al. , 2018 ; Gidel et al. , 2019a ; Mescheder et al. , 2018 ) . Mathematically , bilinear zero-sum games can be formulated as the following min-max problem : minx∈Rn maxy∈Rn x > Ey + b > x+ c > y . ( 2.1 ) The set of all saddle points ( see definition in eq . ( 1.1 ) ) is : { ( x , y ) |Ey + b = 0 , E > x+ c = 0 } . ( 2.2 ) Throughout , for simplicity we assume E to be invertible , whereas the seemingly general case with non-invertible E is treated in Appendix G. The linear terms are not essential in our analysis and we take b = c = 0 throughout the paper1 . In this case , the only saddle point is ( 0,0 ) . For bilinear games , it is well-known that simultaneous gradient descent ascent does not converge ( Nemirovski & Yudin , 1983 ) and other gradient-based algorithms tailored for min-max optimization have been proposed ( Korpelevich , 1976 ; Daskalakis et al. , 2018 ; Gidel et al. , 2019a ; Mescheder et al. , 2017 ) . These iterative algorithms all belong to the class of general linear dynamical systems ( LDS , a.k.a . 1If they are not zero , one can translate x and y to cancel the linear terms , see e.g . Gidel et al . ( 2019b ) . matrix iterative processes ) . Using state augmentation z ( t ) : = ( x ( t ) , y ( t ) ) we define a general k-step LDS as follows : z ( t ) = ∑k i=1Aiz ( t−i ) + d , ( 2.3 ) where the matrices Ai and vector d depend on the gradient algorithm ( examples can be found in Appendix C.1 ) . Define the characteristic polynomial , withA0 = −I : p ( λ ) : = det ( ∑k i=0Aiλ k−i ) . ( 2.4 ) The following well-known result decides when such a k-step LDS converges for any initialization : Theorem 2.1 ( e.g . Gohberg et al . ( 1982 ) ) . The LDS in eq . ( 2.3 ) converges for any initialization ( z ( 0 ) , . . . , z ( k−1 ) ) iff the spectral radius r : = max { |λ| : p ( λ ) = 0 } < 1 , in which case { z ( t ) } converges linearly with an ( asymptotic ) exponent r. Therefore , understanding the bilinear game dynamics reduces to spectral analysis . The ( sufficient and necessary ) convergence condition reduces to that all roots of p ( λ ) lie in the ( open ) unit disk , which can be conveniently analyzed through the celebrated Schur ’ s theorem ( Schur , 1917 ) : Theorem 2.2 ( Schur ( 1917 ) ) . The roots of a real polynomial p ( λ ) = a0λn + a1λn−1 + · · ·+ an are within the ( open ) unit disk of the complex plane iff ∀k ∈ { 1 , 2 , . . . , n } , det ( PkP > k −Q > kQk ) > 0 , where Pk , Qk are k × k matrices defined as : [ Pk ] i , j = ai−j1i≥j , [ Qk ] i , j = an−i+j1i≤j . In the theorem above , we denoted 1S as the indicator function of the event S , i.e . 1S = 1 if S holds and 1S = 0 otherwise . For a nice summary of related stability tests , see Mansour ( 2011 ) . We therefore define Schur stable polynomials to be those polynomials whose roots all lie within the ( open ) unit disk of the complex plane . Schur ’ s theorem has the following corollary ( proof included in Appendix B.2 for the sake of completeness ) : Corollary 2.1 ( e.g . Mansour ( 2011 ) ) . A real quadratic polynomial λ2 + aλ + b is Schur stable iff b < 1 , |a| < 1 + b ; A real cubic polynomial λ3 + aλ2 + bλ + c is Schur stable iff |c| < 1 , |a+ c| < 1 + b , b− ac < 1− c2 ; A real quartic polynomial λ4 + aλ3 + bλ2 + cλ+ d is Schur stable iff |c− ad| < 1− d2 , |a+ c| < b+ d+ 1 , and b < ( 1 + d ) + ( c− ad ) ( a− c ) / ( d− 1 ) 2 . Let us formally define Jacobi and GS updates : Jacobi updates take the form x ( t ) = T1 ( x ( t−1 ) , y ( t−1 ) , . . . , x ( t−k ) , y ( t−k ) ) , y ( t ) = T2 ( x ( t−1 ) , y ( t−1 ) , . . . , x ( t−k ) , y ( t−k ) ) , while Gauss–Seidel updates replace x ( t−i ) with the more recent x ( t−i+1 ) in operator T2 , where T1 , T2 : Rnk × Rnk → Rn can be any update functions . For LDS updates in eq . ( 2.3 ) we find a nice relation between the characteristic polynomials of Jacobi and GS updates in Theorem 2.3 ( proof in Appendix B.1 ) , which turns out to greatly simplify our subsequent analyses : Theorem 2.3 ( Jacobi vs. Gauss–Seidel ) . Let p ( λ , γ ) = det ( ∑k i=0 ( γLi +Ui ) λ k−i ) , whereAi = Li + Ui and Li is strictly lower block triangular . Then , the characteristic polynomial of Jacobi updates is p ( λ , 1 ) while that of Gauss–Seidel updates is p ( λ , λ ) . Compared to the Jacobi update , in some sense the Gauss–Seidel update amounts to shifting the strictly lower block triangular matrices Li one step to the left , as p ( λ , λ ) can be rewritten as det ( ∑k i=0 ( Li+1 +Ui ) λ k−i ) , with Lk+1 : = 0 . This observation will significantly simplify our comparison between Jacobi and Gauss–Seidel updates . Next , we define some popular gradient algorithms for finding saddle points in the min-max problem min x max y f ( x , y ) . ( 2.5 ) We present the algorithms for a general ( bivariate ) function f although our main results will specialize f to the bilinear case in eq . ( 2.1 ) . Note that we introduced more “ step sizes ” for our refined analysis , as we find that the enlarged parameter space often contains choices for faster linear convergence ( see §4 ) . We only define the Jacobi updates , while the GS counterparts can be easily inferred . We always use α1 and α2 to define step sizes ( or learning rates ) which are positive . Gradient descent ( GD ) The generalized GD update has the following form : x ( t+1 ) = x ( t ) − α1∇xf ( x ( t ) , y ( t ) ) , y ( t+1 ) = y ( t ) + α2∇yf ( x ( t ) , y ( t ) ) . ( 2.6 ) When α1 = α2 , the convergence of averaged iterates ( a.k.a . Cesari convergence ) for convex-concave games is analyzed in ( Bruck , 1977 ; Nemirovski & Yudin , 1978 ; Nedić & Ozdaglar , 2009 ) . Recent progress on interpreting GD with dynamical systems can be seen in , e.g. , Mertikopoulos et al . ( 2018 ) ; Bailey et al . ( 2019 ) ; Bailey & Piliouras ( 2018 ) . Extra-gradient ( EG ) We study a generalized version of EG , defined as follows : x ( t+1/2 ) = x ( t ) − γ1∇xf ( x ( t ) , y ( t ) ) , y ( t+1/2 ) = y ( t ) + γ2∇yf ( x ( t ) , y ( t ) ) ; ( 2.7 ) x ( t+1 ) = x ( t ) − α1∇xf ( x ( t+1/2 ) , y ( t+1/2 ) ) , y ( t+1 ) = y ( t ) + α2∇yf ( x ( t+1/2 ) , y ( t+1/2 ) ) . ( 2.8 ) EG was first proposed in Korpelevich ( 1976 ) with the restriction α1 = α2 = γ1 = γ2 , under which linear convergence was proved for bilinear games . Convergence of EG on convex-concave games was analyzed in Nemirovski ( 2004 ) ; Monteiro & Svaiter ( 2010 ) , and Mertikopoulos et al . ( 2019 ) provides convergence guarantees for specific non-convex-non-concave problems . For bilinear games , a slightly more generalized version was proposed in Liang & Stokes ( 2019 ) where α1 = α2 , γ1 = γ2 , with linear convergence proved . For later convenience we define β1 = α2γ1 and β2 = α1γ2 . Optimistic gradient descent ( OGD ) We study a generalized version of OGD , defined as follows : x ( t+1 ) = x ( t ) − α1∇xf ( x ( t ) , y ( t ) ) + β1∇xf ( x ( t−1 ) , y ( t−1 ) ) , ( 2.9 ) y ( t+1 ) = y ( t ) + α2∇yf ( x ( t ) , y ( t ) ) − β2∇yf ( x ( t−1 ) , y ( t−1 ) ) . ( 2.10 ) The original version of OGD was given in Popov ( 1980 ) with α1 = α2 = 2β1 = 2β2 and rediscovered in the GAN literature ( Daskalakis et al. , 2018 ) . Its linear convergence for bilinear games was proved in Liang & Stokes ( 2019 ) . A slightly more generalized version with α1 = α2 and β1 = β2 was analyzed in Peng et al . ( 2019 ) ; Mokhtari et al . ( 2019b ) , again with linear convergence proved . The stochastic case was analyzed in Hsieh et al . ( 2019 ) . Momentum method Generalized heavy ball method was analyzed in Gidel et al . ( 2019b ) : x ( t+1 ) = x ( t ) − α1∇xf ( x ( t ) , y ( t ) ) + β1 ( x ( t ) − x ( t−1 ) ) , ( 2.11 ) y ( t+1 ) = y ( t ) + α2∇yf ( x ( t ) , y ( t ) ) + β2 ( y ( t ) − y ( t−1 ) ) . ( 2.12 ) This is a modification of Polyak ’ s heavy ball ( HB ) ( Polyak , 1964 ) , which also motivated Nesterov ’ s accelerated gradient algorithm ( NAG ) ( Nesterov , 1983 ) . Note that for both x-update and the y-update , we add a scale multiple of the successive difference ( e.g . proxy of the momentum ) . For this algorithm our result below improves those obtained in Gidel et al . ( 2019b ) , as will be discussed in §3 . EG and OGD as approximations of proximal point algorithm It has been observed recently in Mokhtari et al . ( 2019b ) that for convex-concave games , EG ( α1 = α2 = γ1 = γ2 = η ) and OGD ( α1/2 = α2/2 = β1 = β2 = η ) can be treated as approximations of the proximal point algorithm ( Martinet , 1970 ; Rockafellar , 1976 ) when η is small . With this result , one can show that EG and OGD converge to saddle points sublinearly for smooth convex-concave games ( Mokhtari et al. , 2019a ) . We give a brief introduction of the proximal point algorithm in Appendix A ( including a linear convergence result for the slightly generalized version ) . The above algorithms , when specialized to a bilinear function f ( see eq . ( 2.1 ) ) , can be rewritten as a 1-step or 2-step LDS ( see . eq . ( 2.3 ) ) . See Appendix C.1 for details .

Summary: This paper studies the convergence of multiple methods (Gradient, extragradient, optimistic and momentum) on a bilinear minmax game. More precisely, this paper uses spectral condition to study the difference between simultaneous (Jacobi) and alternating (Gau\ss-Seidel) updates. The analysis is based on Schur theorem and give necessary and sufficient condition for convergence.