On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach

1 INTRODUCTION We consider differentiable sequential games with two players : a leader who can commit to an action , and a follower who responds after observing the leader ’ s action . Particularly , we focus on the zero-sum case of this problem which is also known as minimax optimization , i.e. , min x∈Rn max y∈Rm f ( x , y ) . Unlike simultaneous games , many practical machine learning algorithms , including generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ; Arjovsky et al. , 2017 ) , adversarial training ( Madry et al. , 2018 ) and primal-dual reinforcement learning ( Du et al. , 2017 ; Dai et al. , 2018 ) , explicitly specify the order of moves between players and the order of which player acts first is crucial for the problem . Therefore , the classical notion of local Nash equilibrium from simultaneous games may not be a proper definition of local optima for sequential games since mini- max is in general not equal to maximin . Instead , we consider the notion of local minimax ( Jin et al. , 2019 ) which takes into account the sequential structure of minimax optimization . The vanilla algorithm for solving sequential minimax optimization is gradient descent-ascent ( GDA ) , where both players take a gradient update simultaneously . However , GDA is known to suffer from two drawbacks . First , it has undesirable convergence properties : it fails to converge to some local minimax and can converge to fixed points that are not local minimax ( Jin et al. , 2019 ; Daskalakis and Panageas , 2018 ) . Second , GDA exhibits strong rotation around fixed points , which requires using very small learning rates ( Mescheder et al. , 2017 ; Balduzzi et al. , 2018 ) to converge . In this paper , we propose Follow-the-Ridge ( FR ) , an algorithm for minimax optimization that addresses both issues . Specifically , we elucidate the cause of undesirable convergence of GDA – the ∗These two authors contributed equally . 1Our code is made public at : https : //github.com/gd-zhang/Follow-the-Ridge leader whose gradient step takes the system away from the ridge . By adding a correction term to the follower , we explicitly cancel out negative effects of the leader ’ s update . Intuitively , the combination of the leader ’ s update and the correction term is parallel to the ridge in the landscape ( see Fig . 1 ) , hence the name Follow-the-Ridge . Overall , our contributions are the following : • We propose a novel algorithm for minimax optimization which has exact local convergence to local minimax points . Previously , this property was only known to be satisfied when the leader moves infinitely slower than the follower in gradient descent-ascent ( Jin et al. , 2019 ) . • We show theoretically and empirically that FR addresses the notorious rotational behaviour of gradient dynamics around fixed points ( Balduzzi et al. , 2018 ) and thus allows a much larger learning rate compared to GDA . • We prove that our algorithm is compatible with standard acceleration techniques such as preconditioning and positive momentum , which can speed up convergence significantly . • We further show that our algorithm also applies to general-sum Stackelberg games ( Fiez et al. , 2019 ; Zeuthen , 1935 ) with similar theoretical guarantees . • Finally , we demonstrate empirically our algorithm improves the convergence performance in both toy minimax problems and GAN training compared to existing methods . 2 PRELIMINARIES . 2.1 MINIMAX OPTIMIZATION . We consider sequential games with two players where one player is deemed the leader and the other the follower . We denote leader ’ s action by x ∈ Rn , and the follower ’ s action by y ∈ Rm . The leader aims at minimizing the cost function f ( x , y ) while the follower aims at maximizing f ( x , y ) . The only assumption we make on the cost function is the following . Assumption 1. f is twice differentiable everywhere , and thrice differentiable at critical points . ∇2yyf is invertible ( i.e. , non-singular ) . The global solution to the sequential game minx maxy f ( x , y ) is an action pair ( x∗ , y∗ ) , such that y∗ is the global optimal response to x∗ for the follower , and that x∗ is the global optimal action for the leader assuming the follower always play the global optimal response . We call this global solution the global minimax . However , finding this global minimax is often intractable ; therefore , we follow Jin et al . ( 2019 ) and take local minimax as the local surrogate . Definition 1 ( local minimax ) . ( x∗ , y∗ ) is a local minimax for f ( x , y ) if ( 1 ) y∗ is a local maximum of f ( x∗ , · ) ; ( 2 ) x∗ is a local minimum of φ ( x ) : = f ( x , r ( x ) ) , where r ( x ) is the implicit function defined by ∇yf ( x , y ) = 0 in a neighborhood of x∗ with r ( x∗ ) = y∗ . In the definition above , the implicit function r ( · ) : Rn → Rm is a local best response for the follower , and is a ridge in the landscape of f ( x , y ) . Local minimaxity captures an equilibrium in a two-player sequential game if both players are only allowed to change their strategies locally . For notational convenience , we define ∇f ( x , y ) = [ ∇xf , ∇yf ] > , ∇2f ( x , y ) = [ Hxx Hxy Hyx Hyy ] . In principle , local minimax can be characterized in terms of the following first-order and second-order conditions , which were established in Jin et al . ( 2019 ) . Proposition 1 ( First-order Condition ) . Any local minimax ( x∗ , y∗ ) satisfies∇f ( x∗ , y∗ ) = 0 . Proposition 2 ( Second-order Necessary Condition ) . Any local minimax ( x∗ , y∗ ) satisfies Hyy 4 0 and Hxx −HxyH−1yyHyx < 0 . Proposition 3 ( Second-order Sufficient Condition ) . Any stationary point ( x∗ , y∗ ) satisfying Hyy ≺ 0 and Hxx −HxyH−1yyHyx 0 is a local minimax . The concept of global/local minimax is different from Nash equilibrium and local Nash , which are the equilibrium concepts typically studied for simultaneous games ( see Nash et al . ( 1950 ) ; Ratliff et al . ( 2016 ) for more details ) . In particular , we note that the concept of Nash equilibrium or local Nash does not reflect the order between the min-player and the max-player and may not exist even for simple functions ( Jin et al. , 2019 ) . In general , the set of local minimax is a superset of local Nash . Under some mild assumptions , local minimax points are guaranteed to exist ( Jin et al. , 2019 ) . However , the set of stable fixed points of GDA , roughly speaking the set of points that GDA locally converges to , is a different superset of local Nash ( Jin et al. , 2019 ) . The relation between the three sets of points is illustrated in Fig . 2 . 2.2 STABILITY OF DISCRETE DYNAMICAL SYSTEMS . Gradient-based methods can reliably find local stable fixed points – local minima in single-objective optimization . Here , we generalize the concept of stability to games by taking game dynamics as a discrete dynamical system . An iteration of the form zt+1 = w ( zt ) can be viewed as a discrete dynamical system , where in our case w : Rn+m → Rn+m . If w ( z ) = z , then z is called a fixed point . We study the stability of fixed points as a proxy to local convergence of game dynamics . Definition 2 . Let J denote the Jacobian of w at a fixed point z . If it has spectral radius ρ ( J ) ≤ 1 , then we call z a stable fixed point . If ρ ( J ) < 1 , then we call z a strictly stable fixed point . It is known that strict stability implies local convergence ( e.g. , see Galor ( 2007 ) ) . In other words , if z is a strictly stable fixed point , there exists a neighborhood U of z such that when initialized in U , the iteration steps always converge to z . 3 UNDESIRABLE BEHAVIOURS OF GDA . In this section , we discuss the undesirable behaviours of GDA in more detail . Recall that the update rule of GDA is given by xt+1 ← xt − η∇xf , yt+1 ← yt + η∇yf , ( 1 ) where we assume the same learning rate for both the leader and the follower for simplicity2 . As illustrated in Fig . 2 , the set of stable fixed points of GDA can include points that are not local minimax and , perhaps even worse , some local minimax are not necessarily stable fixed points of GDA . Here , we first give an example that a stable fixed point of GDA is not a local minimax . Consider minx maxy f ( x , y ) = 3x 2 + y2 + 4xy ; the only stationary point of this problem is ( 0 , 0 ) and the Jacobian of GDA at this point is J = I− η [ 6 4 −4 −2 ] . It is easy to see that the eigenvalues of J are e1 = e2 = 1− 2η . Therefore , by Definition 2 , ( 0 , 0 ) is a strictly stable fixed point of GDA . However , one can show that Hyy = 2 > 0 which doesn ’ t satisfy the second-order necessary condition of local minimax . Similarly , one can easily find examples in which a local minimax is not in the set of stable fixed points of GDA , e.g. , minx∈R maxy∈R f ( x , y ) = −3x2 − y2 + 4xy ( see Fig . 1 ) . In this example , the two Jacobian eigenvalues are both greater than 1 no matter how small the learning rate is . In other words , GDA fails to converge to ( 0 , 0 ) for almost all initializations ( Daskalakis and Panageas , 2018 ) . As we will discuss in the next section , the main culprit of the undesirable behaviours of GDA is the leader whose gradient update −η∇xf pushes the whole system away from the ridge or attracts the system to non-local-minimax points . By contrast , the follower ’ s step η∇yf can pull the system closer to the ridge ( see Fig . 1 ) or push it away from bad fixed points . To guarantee convergence to local minimax ( or avoid bad fixed points ) , we have to use a very small learning rate for the leader ( Jin et al. , 2019 ; Fiez et al. , 2019 ) so that the η∇yf term dominates . In the next section , we offer an alternative approach which explicitly cancels out undesirable effects of −η∇xf , thereby allowing us to use larger learning rates for the leader . 2In general , the learning rates of two players can be different . Since our arguments apply to general setting as long as the ratio ηx/ηy is a positive constant , so we assume the same learning rate for convenience .

Summary: This paper designs a set of dynamics for learning in games called follow-the-ridge with the goal of finding local stackelberg equilibria. The main theoretical results show that the only stable attractors of the dynamics are stackelberg equilibria. Moreover, the authors give a deterministic convergence rate for the vanilla algorithm and a convergence rate using momentum. Empirical results show the learning dynamics cancel out rotational components and drive the vector field to zero rapidly, while reaching good performance on simple GAN examples.