Local Convergence Analysis of Gradient Descent Ascent with Finite Timescale Separation

1 INTRODUCTION . In this paper we study learning in zero-sum games of the form min x1∈X1 max x2∈X2 f ( x1 , x2 ) where the objective function of the game f is assumed to be sufficiently smooth and potentially nonconvex and non-concave in the strategy spaces X1 and X2 respectively with each Xi a precompact subset of Rni . This general problem formulation has long been fundamental in game theory ( Başar & Olsder , 1998 ) and recently it has become central to machine learning with applications in generative adversarial networks ( Goodfellow et al. , 2014 ) , robust supervised learning ( Madry et al. , 2018 ; Sinha et al. , 2018 ) , reinforcement and multi-agent reinforcement learning ( Rajeswaran et al. , 2020 ; Zhang et al. , 2019 ) , imitation learning ( Ho & Ermon , 2016 ) , constrained optimization ( Cherukuri et al. , 2017 ) , and hyperparameter optimization ( Lorraine et al. , 2020 ; MacKay et al. , 2019 ) . The gradient descent-ascent learning dynamics are widely studied as a potential method for efficiently computing equilibria in game formulations . However , in zero-sum games , a number of past works highlight problems with this learning dynamic including both non-convergence to meaningful critical points as well as convergence to critical points devoid of game theoretic meaning , where common notions of ‘ meaningful ’ equilibria include the local Nash and local minmax ( Stackelberg ) concepts . For instance , in bilinear games , gradient descent-ascent avoids local Nash and Stackelberg equilibria due to the inherent instability of the update rule for this class . Fortunately , in this class of games , regularization or gradient-based learning dynamics that employ different numerical discretization schemes ( as compared to forward Euler for gradient descent-ascent ) are known to alleviate this issue ( Daskalakis et al. , 2018 ; Mertikopoulos et al. , 2019 ; Zhang & Yu , 2020 ) . For the more general nonlinear nonconvex-nonconcave class of games , it has been shown gradient descentascent with a shared learning rate is prone to reaching critical points that are neither local Nash equilibria nor local Stackelberg equilibria ( Daskalakis & Panageas , 2018 ; Jin et al. , 2020 ; Mazumdar et al. , 2020 ) . While an important negative result , it does not rule out the prospect that gradient descent-ascent may be able to guarantee equilibrium convergence as it fails to account for a key structural parameter of the dynamics , namely the ratio of learning rates between the players . Motivated by the observation that the order of play between players is fundamental to the definition of the game , the role of timescale separation in gradient descent-ascent has recently been explored theoretically ( Chasnov et al. , 2019 ; Heusel et al. , 2017 ; Jin et al. , 2020 ) . On the empirical side , it has been widely demonstrated that timescale separation in gradient descent-ascent is crucial to improving the solution quality when training generative adversarial networks ( Arjovsky et al. , 2017 ; Goodfellow et al. , 2014 ; Heusel et al. , 2017 ) . Denoting γ1 as the learning rate of the player 1 , the learning rate of player 2 can be redefined as γ2 = τγ1 where τ = γ2/γ1 > 0 is the learning rate ratio . Toward understanding the effect of timescale separation , Jin et al . ( 2020 ) show the locally stable critical points of gradient descent-ascent coincide with the set of strict local minmax/Stackelberg equilibrium across the spectrum of sufficiently smooth zero-sum games as τ →∞ . In other words , all ‘ bad critical points ’ ( critical points lacking game-theoretic meaning ) become unstable and all ‘ good critical points ’ ( game-theoretically meaningful equilibria ) remain or become locally exponentially stable ( cf . Definition 3 ) as τ → ∞ . While a promising theoretical development , gradient descent-ascent with a timescale separation approaching infinity does not lead to a practical learning rule and the analysis of it does not necessarily provide insights into the common usage of a reasonable finite timescale separation . An important observation is that choosing τ arbitrarily large with the goal of ensuring local equilibrium convergence can lead to numerically ill-conditioned problems . This highlights the significance of understanding the exact range of learning rate ratios that guarantee local stability . Moreover , our experiments in Section 5 ( Dirac-GAN ) and in Appendix K show that modest values of τ are typically sufficient to guarantee stability of only equilibria which allows for larger choices of γ1 and results in faster convergence to an equilibrium . Contributions . We show gradient descent-ascent locally converges to a critical point for a range of finite learning rate ratios if and only if the critical point is a strict local Stackelberg equilibria ( Theorem 1 ) .1 This result is constructive in the sense that we explicitly characterize the exact range of learning rate ratios for which the guarantee holds . Furthermore , we show all other critical points are unstable for a range of finite learning rate ratios that we explicitly construct ( Theorem 2 ) . To our knowledge , the aforementioned guarantees are the first of their kind in nonconvex-nonconcave zero-sum games for an implementable first-order method . Moreover , the technical results in this work rely on tools that have not appeared in the machine learning and optimization communities analyzing games . Finally , we extend these results to gradient penalty regularization methods in generative adversarial networks ( Theorem 3 ) , thereby providing theoretical guarantees for a common combination of heuristics used in practice , and empirically demonstrate the benefits and trade-offs of regularization and timescale separation on the Dirac-GAN along with image datasets . 2 PRELIMINARIES . A two–player zero-sum continuous game is defined by a collection of costs ( f1 , f2 ) where f1 ≡ f and f2 ≡ −f with f ∈ Cr ( X , R ) for some r ≥ 2 and where X = X1 × X2 with each Xi a precompact subset of Rni for i ∈ { 1 , 2 } and n = n1 + n2 . Each player i ∈ I seeks to minimize their cost fi ( xi , x−i ) with respect to their choice variable xi where x−i is the vector of all other actions xj with j 6= i . We denote Difi as the derivative of fi with respect to xi , Dijfi as the partial derivative of Difi with respect to xj , and D2i fi as the partial derivative of Difi with respect to xi . Mathematical Notation . Given a matrixA ∈ Rn1×n2 , let vec ( A ) ∈ Rn1n2 be its vectorization such that vec ( A ) takes rows ai of A , transposes them and stacks them vertically in order of their index . Let ⊗ and ⊕ denote the Kronecker product and sum respectively , where A⊕B = A⊗ I + I ⊗B . Moreover , is an operator that generates an 12n ( n+1 ) × 1 2n ( n+1 ) matrix from a matrixA ∈ R n×n such thatA A = H+n ( A⊕A ) Hn whereH+n = ( H > n Hn ) −1H > n is the ( left ) pseudo-inverse ofHn , a full column rank duplication matrix . Let λ+max ( · ) be the largest positive real root of its argument if it exists and zero otherwise . See Lancaster & Tismenetsky ( 1985 ) and Appendix B for more detail . Equilibrium . There are natural equilibrium concepts depending on the order of play : the ( local ) Nash equilibrium concept in the case of simultaneous play and the ( local ) Stackelberg ( equivalently minmax in zero-sum games ) equilibrium concept in the case of hierarchical play ( Başar & Olsder , 1Following Fiez et al . ( 2020 ) , we refer to strict local Stackelberg as differential Stackelberg throughout . 1998 ) . Formal local equilibrium definitions are provided in Appendix B , while here we characterize the different equilibrium notions in terms of sufficient conditions on player costs as is typical in the machine learning and optimization literature ( see , e.g. , Berard et al . 2020 ; Daskalakis & Panageas 2018 ; Fiez et al . 2020 ; Goodfellow 2016 ; Jin et al . 2020 ; Mazumdar et al . 2020 ; Wang et al . 2020 ) . The following definition is characterized by sufficient conditions for a local Nash equilibrium . Definition 1 ( Differential Nash Equilibrium , Ratliff et al . 2013 ) . The joint strategy x ∈ X is a differential Nash equilibrium if D1f ( x ) = 0 , −D2f ( x ) = 0 , D21f ( x ) > 0 , and D22f ( x ) < 0 . The Jacobian of the vector of individual gradients g ( x ) = ( D1f ( x ) , −D2f ( x ) ) is defined by J ( x ) = [ D21f ( x ) D12f ( x ) −D > 12f ( x ) −D22f ( x ) ] . ( 1 ) Let S1 ( · ) denote the Schur complement of ( · ) with respect to the n2×n2 block in ( · ) . The following definition is characterized by sufficient conditions for a local Stackelberg equilibrium . Definition 2 ( Differential Stackelberg Equilibrium , Fiez et al . 2020 ) . The joint strategy x ∈ X is a differential Stackelberg equilibrium if D1f ( x ) = 0 , −D2f ( x ) = 0 , S1 ( J ( x ) ) > 0 , D22f ( x ) < 0 . Learning Dynamics . We study agents seeking equilibria of the game via a learning algorithm and consider arguably the most natural learning rule in zero-sum continuous games : gradient descentascent ( GDA ) . Moreover , we investigate this learning rule with timescale separation between the players . Let τ = γ2/γ1 be the learning rate ratio and define Λτ = blockdiag ( In1 , τIn2 ) where Ini is a ni × ni identity matrix . The τ -GDA dynamics with g ( x ) = ( D1f ( x ) , −D2f ( x ) ) are given by xk+1 = xk − γ1Λτg ( xk ) . ( 2 ) 3 STABILITY OF CONTINUOUS TIME GDA WITH TIMESCALE SEPARATION . To characterize the convergence of τ -GDA , we begin by studying its continuous time limiting system ẋ = −Λτg ( x ) . ( 3 ) The Jacobian of the system from ( 3 ) is given by Jτ ( x ) = ΛτJ ( x ) where J ( x ) is defined in ( 1 ) . Observe that critical points ( x∗ such that g ( x∗ ) = 0 ) are shared between τ -GDA and ( 3 ) . Thus , by analyzing the stability of the continuous time system around critical points as a function of the timescale separation τ using the Jacobian Jτ ( x∗ ) , we can draw conclusions about the stability and convergence of the discrete time system τ -GDA . It well known that a critical point is locally ( exponentially ) stable when the spectrum of −Jτ ( x∗ ) is in the open left-half complex plane C◦− ( cf . Theorem B.1 , Appendix B ) . Throughout , we use the broader term “ stable ” to mean the following . Definition 3 . A critical point x∗ is locally exponentially stable for ẋ = −Λτg ( x ) if an only if spec ( −Jτ ( x∗ ) ) ⊂ C◦− ( or , equivalently , spec ( Jτ ( x∗ ) ) ⊂ C◦+ ) where C◦− and C◦+ denote the open left-half and right-half complex plane , respectively . We now show differential Stackelberg equilibria are the only critical points that are stable for a range of finite learning rate ratios,2 whereas the remainder of critical points are unstable for a range of finite learning rate ratios . Importantly , we characterize the learning rate ratios for which the results hold .

Summary: Motivated by the many applications of min-max optimization problems in Machine Learning, the authors examine the effect of using different learning rates for each player in Gradient Descent Ascent (GDA) for non-convex non-concave optimization problems. Prior work has already established that making the learning rate of one player infinitely larger than other player's learning rate alleviates the cycling problems of GDA and makes game-theoretically meaningful equilibria the only asymptotically stable fixed points. The main contribution of this work is that it proves that we can get the same stability guarantees while keeping the learning rates of both players finite. This is crucial for practical applications where using unbounded learning rates in not an option. The authors employ this result to prove a variety of local convergence results in both deterministic ans stochastic settings.