On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach

1 INTRODUCTION We consider differentiable sequential games with two players : a leader who can commit to an action , and a follower who responds after observing the leader ’ s action . Particularly , we focus on the zero-sum case of this problem which is also known as minimax optimization , i.e. , min x∈Rn max y∈Rm f ( x , y ) . Unlike simultaneous games , many practical machine learning algorithms , including generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ; Arjovsky et al. , 2017 ) , adversarial training ( Madry et al. , 2018 ) and primal-dual reinforcement learning ( Du et al. , 2017 ; Dai et al. , 2018 ) , explicitly specify the order of moves between players and the order of which player acts first is crucial for the problem . Therefore , the classical notion of local Nash equilibrium from simultaneous games may not be a proper definition of local optima for sequential games since mini- max is in general not equal to maximin . Instead , we consider the notion of local minimax ( Jin et al. , 2019 ) which takes into account the sequential structure of minimax optimization . The vanilla algorithm for solving sequential minimax optimization is gradient descent-ascent ( GDA ) , where both players take a gradient update simultaneously . However , GDA is known to suffer from two drawbacks . First , it has undesirable convergence properties : it fails to converge to some local minimax and can converge to fixed points that are not local minimax ( Jin et al. , 2019 ; Daskalakis and Panageas , 2018 ) . Second , GDA exhibits strong rotation around fixed points , which requires using very small learning rates ( Mescheder et al. , 2017 ; Balduzzi et al. , 2018 ) to converge . In this paper , we propose Follow-the-Ridge ( FR ) , an algorithm for minimax optimization that addresses both issues . Specifically , we elucidate the cause of undesirable convergence of GDA – the ∗These two authors contributed equally . 1Our code is made public at : https : //github.com/gd-zhang/Follow-the-Ridge leader whose gradient step takes the system away from the ridge . By adding a correction term to the follower , we explicitly cancel out negative effects of the leader ’ s update . Intuitively , the combination of the leader ’ s update and the correction term is parallel to the ridge in the landscape ( see Fig . 1 ) , hence the name Follow-the-Ridge . Overall , our contributions are the following : • We propose a novel algorithm for minimax optimization which has exact local convergence to local minimax points . Previously , this property was only known to be satisfied when the leader moves infinitely slower than the follower in gradient descent-ascent ( Jin et al. , 2019 ) . • We show theoretically and empirically that FR addresses the notorious rotational behaviour of gradient dynamics around fixed points ( Balduzzi et al. , 2018 ) and thus allows a much larger learning rate compared to GDA . • We prove that our algorithm is compatible with standard acceleration techniques such as preconditioning and positive momentum , which can speed up convergence significantly . • We further show that our algorithm also applies to general-sum Stackelberg games ( Fiez et al. , 2019 ; Zeuthen , 1935 ) with similar theoretical guarantees . • Finally , we demonstrate empirically our algorithm improves the convergence performance in both toy minimax problems and GAN training compared to existing methods . 2 PRELIMINARIES . 2.1 MINIMAX OPTIMIZATION . We consider sequential games with two players where one player is deemed the leader and the other the follower . We denote leader ’ s action by x ∈ Rn , and the follower ’ s action by y ∈ Rm . The leader aims at minimizing the cost function f ( x , y ) while the follower aims at maximizing f ( x , y ) . The only assumption we make on the cost function is the following . Assumption 1. f is twice differentiable everywhere , and thrice differentiable at critical points . ∇2yyf is invertible ( i.e. , non-singular ) . The global solution to the sequential game minx maxy f ( x , y ) is an action pair ( x∗ , y∗ ) , such that y∗ is the global optimal response to x∗ for the follower , and that x∗ is the global optimal action for the leader assuming the follower always play the global optimal response . We call this global solution the global minimax . However , finding this global minimax is often intractable ; therefore , we follow Jin et al . ( 2019 ) and take local minimax as the local surrogate . Definition 1 ( local minimax ) . ( x∗ , y∗ ) is a local minimax for f ( x , y ) if ( 1 ) y∗ is a local maximum of f ( x∗ , · ) ; ( 2 ) x∗ is a local minimum of φ ( x ) : = f ( x , r ( x ) ) , where r ( x ) is the implicit function defined by ∇yf ( x , y ) = 0 in a neighborhood of x∗ with r ( x∗ ) = y∗ . In the definition above , the implicit function r ( · ) : Rn → Rm is a local best response for the follower , and is a ridge in the landscape of f ( x , y ) . Local minimaxity captures an equilibrium in a two-player sequential game if both players are only allowed to change their strategies locally . For notational convenience , we define ∇f ( x , y ) = [ ∇xf , ∇yf ] > , ∇2f ( x , y ) = [ Hxx Hxy Hyx Hyy ] . In principle , local minimax can be characterized in terms of the following first-order and second-order conditions , which were established in Jin et al . ( 2019 ) . Proposition 1 ( First-order Condition ) . Any local minimax ( x∗ , y∗ ) satisfies∇f ( x∗ , y∗ ) = 0 . Proposition 2 ( Second-order Necessary Condition ) . Any local minimax ( x∗ , y∗ ) satisfies Hyy 4 0 and Hxx −HxyH−1yyHyx < 0 . Proposition 3 ( Second-order Sufficient Condition ) . Any stationary point ( x∗ , y∗ ) satisfying Hyy ≺ 0 and Hxx −HxyH−1yyHyx 0 is a local minimax . The concept of global/local minimax is different from Nash equilibrium and local Nash , which are the equilibrium concepts typically studied for simultaneous games ( see Nash et al . ( 1950 ) ; Ratliff et al . ( 2016 ) for more details ) . In particular , we note that the concept of Nash equilibrium or local Nash does not reflect the order between the min-player and the max-player and may not exist even for simple functions ( Jin et al. , 2019 ) . In general , the set of local minimax is a superset of local Nash . Under some mild assumptions , local minimax points are guaranteed to exist ( Jin et al. , 2019 ) . However , the set of stable fixed points of GDA , roughly speaking the set of points that GDA locally converges to , is a different superset of local Nash ( Jin et al. , 2019 ) . The relation between the three sets of points is illustrated in Fig . 2 . 2.2 STABILITY OF DISCRETE DYNAMICAL SYSTEMS . Gradient-based methods can reliably find local stable fixed points – local minima in single-objective optimization . Here , we generalize the concept of stability to games by taking game dynamics as a discrete dynamical system . An iteration of the form zt+1 = w ( zt ) can be viewed as a discrete dynamical system , where in our case w : Rn+m → Rn+m . If w ( z ) = z , then z is called a fixed point . We study the stability of fixed points as a proxy to local convergence of game dynamics . Definition 2 . Let J denote the Jacobian of w at a fixed point z . If it has spectral radius ρ ( J ) ≤ 1 , then we call z a stable fixed point . If ρ ( J ) < 1 , then we call z a strictly stable fixed point . It is known that strict stability implies local convergence ( e.g. , see Galor ( 2007 ) ) . In other words , if z is a strictly stable fixed point , there exists a neighborhood U of z such that when initialized in U , the iteration steps always converge to z . 3 UNDESIRABLE BEHAVIOURS OF GDA . In this section , we discuss the undesirable behaviours of GDA in more detail . Recall that the update rule of GDA is given by xt+1 ← xt − η∇xf , yt+1 ← yt + η∇yf , ( 1 ) where we assume the same learning rate for both the leader and the follower for simplicity2 . As illustrated in Fig . 2 , the set of stable fixed points of GDA can include points that are not local minimax and , perhaps even worse , some local minimax are not necessarily stable fixed points of GDA . Here , we first give an example that a stable fixed point of GDA is not a local minimax . Consider minx maxy f ( x , y ) = 3x 2 + y2 + 4xy ; the only stationary point of this problem is ( 0 , 0 ) and the Jacobian of GDA at this point is J = I− η [ 6 4 −4 −2 ] . It is easy to see that the eigenvalues of J are e1 = e2 = 1− 2η . Therefore , by Definition 2 , ( 0 , 0 ) is a strictly stable fixed point of GDA . However , one can show that Hyy = 2 > 0 which doesn ’ t satisfy the second-order necessary condition of local minimax . Similarly , one can easily find examples in which a local minimax is not in the set of stable fixed points of GDA , e.g. , minx∈R maxy∈R f ( x , y ) = −3x2 − y2 + 4xy ( see Fig . 1 ) . In this example , the two Jacobian eigenvalues are both greater than 1 no matter how small the learning rate is . In other words , GDA fails to converge to ( 0 , 0 ) for almost all initializations ( Daskalakis and Panageas , 2018 ) . As we will discuss in the next section , the main culprit of the undesirable behaviours of GDA is the leader whose gradient update −η∇xf pushes the whole system away from the ridge or attracts the system to non-local-minimax points . By contrast , the follower ’ s step η∇yf can pull the system closer to the ridge ( see Fig . 1 ) or push it away from bad fixed points . To guarantee convergence to local minimax ( or avoid bad fixed points ) , we have to use a very small learning rate for the leader ( Jin et al. , 2019 ; Fiez et al. , 2019 ) so that the η∇yf term dominates . In the next section , we offer an alternative approach which explicitly cancels out undesirable effects of −η∇xf , thereby allowing us to use larger learning rates for the leader . 2In general , the learning rates of two players can be different . Since our arguments apply to general setting as long as the ratio ηx/ηy is a positive constant , so we assume the same learning rate for convenience .

Summary: The present work proposes a new algorithm, "Follow the Ridge" (FR) that uses second order gradient information to iteratively find local minimax points, or Stackelberg equilibria in two player continuous games. The authors show rigorously that the only stable fixed points of their algorithm are local minimax points and that their algorithm therefore converges locally exactly to those points. They show that the resulting optimizer is compatible with heuristics like RMSProp and Momentum. They further evaluate their algorithm on polynomial toy problems and simple GANs.