On the Impossibility of Global Convergence in Multi-Loss Optimization

1 INTRODUCTION . Problem Setting . As multi-agent architectures proliferate in machine learning , it is becoming increasingly important to understand the dynamics of gradient-based methods when optimizing multiple interacting goals , otherwise known as differentiable games . This framework encompasses GANs ( Goodfellow et al. , 2014 ) , intrinsic curiosity ( Pathak et al. , 2017 ) , imaginative agents ( Racanière et al. , 2017 ) , synthetic gradients ( Jaderberg et al. , 2017 ) , hierarchical reinforcement learning ( Wayne & Abbott , 2014 ; Vezhnevets et al. , 2017 ) and multi-agent RL in general ( Busoniu et al. , 2008 ) . The interactions between learning agents make for vastly more complex mechanics : naively applying gradient descent on each loss simultaneously is known to diverge even in simple bilinear games . Related Work . A large number of methods have recently been proposed to alleviate the failings of simultaneous gradient descent : adaptations of single-loss algorithms such as Extragradient ( EG ) ( Azizian et al. , 2019 ) and Optimistic Mirror Descent ( OMD ) ( Daskalakis et al. , 2018 ) , Alternating Gradient Descent ( AGD ) for finite regret ( Bailey et al. , 2019 ) , Consensus Optimization ( CO ) for GAN training ( Mescheder et al. , 2017 ) , Competitive Gradient Descent ( CGD ) based on solving a bilinear approximation of the loss functions ( Schaefer & Anandkumar , 2019 ) , Symplectic Gradient Adjustment ( SGA ) based on a novel decomposition of game mechanics ( Balduzzi et al. , 2018 ; Letcher et al. , 2019a ) , and opponent-shaping algorithms including Learning with OpponentLearning Awareness ( LOLA ) ( Foerster et al. , 2018 ) and its convergent counterpart , Stable Opponent Shaping ( SOS ) ( Letcher et al. , 2019b ) . Let A be this set of algorithms . Each has shown promising theoretical implications and empirical results , but none offers insight into global convergence in the non-convex setting , which includes the vast majority of machine learning applications . One of the main roadblocks compared with single-loss optimization has been noted by Schaefer & Anandkumar ( 2019 ) : “ a convergence proof in the nonconvex case analogue to Lee et al . ( 2016 ) is still out of reach in the competitive setting . A major obstacle to this end is the identification of a suitable measure of progress ( which is given by the function value in the single agent setting ) , since norms of gradients can not be expected to decay monotonously for competitive dynamics in non-convex-concave games. ” It has been established that Hamiltonian Gradient Descent converges in two-player zero-sum games under a “ sufficiently bilinear ” condition by Abernethy et al . ( 2019 ) , but this algorithm is unsuitable for optimization as it can not distinguish between minimization and maximization ( Hsieh et al. , 2020 , Appendix C.4 ) . Global convergence has also been established for some algorithms in a few special cases : potential and Hamiltonian games ( Balduzzi et al. , 2018 ) , zero-sum games satisfying the twosided Polyak-Łojasiewicz condition ( Yang et al. , 2020 ) , zero-sum linear quadratic games ( Zhang et al. , 2019 ) and zero-sum games whose loss and first three derivatives are bounded ( Mangoubi & Vishnoi , 2020 ) . These are significant contributions with several applications of interest , but do not include any of the architectures mentioned above . Finally , Balduzzi et al . ( 2020 ) show that GD dynamics are bounded under a ‘ negative sentiment ’ assumption in smooth markets , which do include GANs – but this does not imply convergence , as we will show . On the other hand , failure of global convergence has been shown for the Multiplicative Weights Update method by Palaiopanos et al . ( 2017 ) , for policy-gradient algorithms by Mazumdar et al . ( 2020 ) , and for simultaneous and alternating gradient descent ( simGD and AGD ) by Vlatakis-Gkaragkounis et al . ( 2019 ) ; Bailey et al . ( 2019 ) , with interesting connections to Poincaré recurrence . Nonetheless , nothing is claimed about other optimization methods . Farnia & Ozdaglar ( 2020 ) show that GANs may have no Nash equilibria , but it does not follow that algorithms fail to converge since there may be locally-attracting but non-Nash critical points ( Mazumdar et al. , 2019 , Example 2 ) . Finally , Hsieh et al . ( 2020 ) uploaded a preprint just after the completion of this work with a similar focus to ours . They prove that generalized Robbins-Monro schemes may converge with arbitrarily high probability to spurious attractors . This includes simGD , AGD , stochastic EG , optimistic gradient and Kiefer-Wolfowitz . However , Hsieh et al . ( 2020 ) focus on the possible occurrence of undesirable convergence phenomena for stochastic algorithms . We instead prove that desirable convergence properties can not simultaneously hold for all algorithms ( including deterministic ) . Moreover , their results apply only to decreasing step-sizes whereas ours include constant step-sizes . These distinctions are further highlighted by Hsieh et al . ( 2020 ) in the further related work section . Taken together , our works give a fuller picture of the failure of global convergence in multi-loss optimization . Contribution . We prove that global convergence in multi-loss optimization is fundamentally incompatible with the ‘ reasonable ’ requirement that algorithms avoid strict maxima and converge only to critical points . We construct a two-player game with zero-sum interactions whose losses are coercive and analytic , but whose only critical point is a strict maximum ( Theorem 1 ) . Reasonable algorithms must either diverge to infinite losses or cycle ( bounded non-convergent iterates ) . One might hope that global convergence could at least be guaranteed in games with strict minima and no other critical points . On the contrary we show that strict minima can have arbitrarily small regions of attraction , in the sense that reasonable algorithms will fail to converge there with arbitrarily high probability for fixed initial parameter distribution ( Theorem 2 ) . Finally , restricting the game class even further , we construct a zero-sum game in which all algorithms in A ( as defined in Appendix A ) are proven to cycle ( Theorem 3 ) . It may be that cycles do not arise in high-dimensional games of interest including GANs . Proving or disproving this is an important avenue for further research , but requires that we recognise the impossibility of global guarantees in the first place . 2 BACKGROUND . 2.1 SINGLE LOSSES : GLOBAL CONVERGENCE OF GRADIENT DESCENT . Given a continuously differentiable function f : Rd → R , let θk+1 = θk − α∇f ( θk ) be the iterates of gradient descent with learning rate α , initialised at θ0 . Under standard regularity conditions , gradient descent converges globally to critical points : Proposition 1 . Assume f ∈ C2 has compact sublevel sets and is either analytic or has isolated critical points . For any θ0 ∈ Rd , define U0 = { f ( θ ) ≤ f ( θ0 ) } and let L < ∞ be a Lipschitz constant for ∇f in U0 . Then for any 0 < α < 2/L we have limk θk = θ̄ for some critical point θ̄ . The requirements for convergence are relatively mild : 1. f has compact sublevel sets iff f is coercive , lim‖θ‖→∞ f ( θ ) =∞ , which mostly holds in machine learning since f is a loss function . 2. f has isolated critical points if it is a Morse function ( nondegenerate Hessian at critical points ) , which holds for almost all C2 functions . More precisely , Morse functions form an open , dense subset of all functions f ∈ C2 ( Rd , R ) in the Whitney C2-topology . 3 . Global Lipschitz continuity is not assumed , which would fail even for cubic polynomials . The goal of this paper is to prove that similar ( even weaker ) guarantees can not be obtained in the multi-loss setting – not only for GD , but for any reasonable algorithm . This has to do with the more complex nature of gradient vector fields arising from multiple losses . 2.2 DIFFERENTIABLE GAMES . Following Balduzzi et al . ( 2018 ) , we frame the problem of multi-loss optimization as a differentiable game among cooperating and competing agents/players . These may simply be different internal components of a single system , like the generator and discriminator in GANs . Definition 1 . A differentiable game is a set of n agents with parameters θ = ( θ1 , . . . , θn ) ∈ Rd and twice continuously differentiable losses Li : Rd → R , where θi ∈ Rdi for each i and ∑ i di = d. Losses are not assumed to be convex/concave in any of the parameters . In practice , losses need only be differentiable almost-everywhere : think of neural nets with rectified linear units . If n = 1 , the ‘ game ’ is simply to minimise a given loss function . We write ∇iLk = ∇θiLk and ∇ijLk = ∇θj∇θiLk for any i , j , k , and define the simultaneous gradient of the game ξ = ( ∇1L1 , . . . , ∇nLn ) T ∈ Rd as the concatenation of each player ’ s gradient . If each agent independently minimises their loss using GD with learning rate α , the parameter update for all agents is given by θ ← θ − αξ ( θ ) . We call this simultaneous gradient descent ( simGD ) , or GD for short . We call θ̄ a critical point if ξ ( θ̄ ) = 0 . Now introduce the ‘ Hessian ’ ( or Jacobian ) of the game as the block matrix H = ∇ξ = ∇11L 1 · · · ∇1nL1 ... . . . ... ∇n1Ln · · · ∇nnLn ∈ Rd×d . Importantly note that H is not symmetric in general unless n = 1 , in which case we recover the usual Hessian H = ∇2L . However H can be decomposed into symmetric and anti-symmetric components as H = S + A ( Balduzzi et al. , 2018 ) . A second useful decomposition has appeared recently in ( Letcher et al. , 2019b ) and ( Schaefer & Anandkumar , 2019 ) : H = Hd + Ho where Hd and Ho are the matrices of diagonal and off-diagonal blocks ; formally , Hd = ⊕ i∇iiLi . One solution concept for differentiable games , analogous to the single-loss case , is defined as follows . Definition 2 . A critical point θ̄ is a ( strict , local ) minimum if H ( θ̄ ) 0.1 These were named ( strict ) stable fixed points by Balduzzi et al . ( 2018 ) , but the term is usually reserved in dynamical systems to the larger class defined by Hessian eigenvalues with positive real parts , which is implied but not equivalent to H 0 for non-symmetric matrices . In particular , strict minima are ( differential ) Nash equilibria as defined by Mazumdar et al . ( 2019 ) , since diagonal blocks must also be positive definite : ∇iiLi ( θ̄ ) 0 . The converse does not hold . Algorithm class . This paper is concerned with any algorithm whose iterates are obtained by initialising θ0 and applying a function F to the previous iterates , namely θk+1 = F ( θk , . . . , θ0 ) . This holds for all gradient-based methods ( deterministic or stochastic ) ; most of them are only functions 1For non-symmetric matrices , positive definiteness is defined as H 0 iff uTHu > 0 for all non-zero u ∈ Rd . This is equivalent to the symmetric part S of H being positive definite . of the current iterate θk , so that θk = F k ( θ0 ) . All probabilistic statements in this paper assume that θ0 is initialised following any bounded and continuous measure ν on Rd . Continuity is a weak requirement and widely holds across machine learning , while boundedness mostly holds in practice since the bounded region can be made large enough to accommodate required initial points . For single-player games , the goal of such algorithms is for θk to converge to a local ( perhaps global ) minimum as k → ∞ . The goal is less clear for differentiable games , but is generally to reach a minimum or a Nash equilibrium . In the case of GANs the goal might be to reach parameters that produce realistic images , which is more challenging to define formally . Throughout the text we use the term ( limit ) cycle to mean bounded but non-convergent iterates . This terminology is used because bounded iterates are non-convergent if and only if they have at least two accumulation points , between which they must ‘ cycle ’ infinitely often . This is not to be taken literally : the set of accumulation points may not even be connected . Hsieh et al . ( 2020 ) provide a more complete characterisation of these cycles . Game class . Expecting global guarantees in all differentiable games is excessive , since every continuous dynamical system arises as simultaneous GD on the loss functions of a differentiable game ( Balduzzi et al. , 2020 , Lemma 1 ) . For this reason , the aforementioned authors have introduced a vastly more tractable class of games called markets . Definition 3 . A ( smooth ) market is a differentiable game where interactions between players are pairwise zero-sum , namely , Li ( θ ) = Li ( θi ) + ∑ j 6=i gij ( θ i , θj ) with gij ( θi , θj ) + gji ( θj , θi ) = 0 for all i , j . This generalises zero-sum games while remaining amenable to optimization and aggregation , meaning that “ we can draw conclusions about the gradient-based dynamics of the collective by summing over properties of its members ” ( Balduzzi et al. , 2020 ) . Moreover , this class captures a large number of applications including GANs and related architectures , intrinsic curiosity modules , adversarial training , task-suites and population self-play . One would modestly hope for some reasonable algorithm to converge globally in markets . We will prove that even this is too much to ask .

Summary: 1. For Theorem 1, as the reviewer understands it, for an optimization problem whose only critical point is a strict maxima, it only has four outcomes, which are listed in the theorem. The result seems quite intuitive and provides very limited understanding for the problem. Please list other possible outcomes for the general problem and state in such way that the paper finds some impossible outcomes which can be excluded for consideration.