Chaos of Learning Beyond Zero-sum and Coordination via Game Decompositions

1 INTRODUCTION . In Machine Learning ( ML ) , it is of primary interest to understand how agents learn in competitive environments . This is more strongly propelled recently due to the success of Generative Adversarial Networks ( GANs ) , which can be viewed as two neural-networks playing a zero-sum game . As such , Evolutionary Game Theory ( EGT ) ( Hofbauer & Sigmund ( 1998 ) ; Sandholm ( 2010 ) ) , a decades-old area devoted to the study of adaptive ( learning ) behaviors of agents in competitive environments arising from Economics , Biology , and Physics , has drawn attention from the ML community . In contrast with the typical optimization ( or no-regret ) approach in ML , EGT provides a dynamicalsystemic perspective to understand ML processes , which has already provided new insights into a number of ML-related problems . This perspective is particularly helpful in studying “ learning in games ” , where irregular behaviors are commonly observed , but the ML community currently lacks of a rigorous method to analyze such systems . In this paper , we study Lyapunov chaos , a central notion that captures instability and unpredictability in dynamical systems . We characterize general normal games where popular learning algorithms exhibit chaotic behaviors . Lyapunov chaos captures the butterfly effect : when the starting point of a dynamical system is slightly perturbed , the resulting trajectories and final outcomes diverge quickly ; see Definition 1 for a formal definition . The perturbations correspond to round-off errors of numerical algorithms in ML ( and Computer Science in general ) .1 While significant efforts have been spent in analyzing and 1In games , such perturbations can also occur due to errors in measuring payoffs . minimizing round-off effects of floating-point computations ( Demmel ( 1997 ) ) , they are unavoidable in general2 . As round-offs are inevitable , and the round-off schemes can vary from machine to machine due to various hardware and software factors , we surely want to avoid chaotic learning that does not fulfill our primary goals in building predictable and reproducible learning systems . Such issues are exemplified by a quote from Ali Rahimi ’ s NIPS ’ 2017 test-of-time award speech : “ Someone on another team changed the default rounding mode of some Tensorflow internals from ‘ truncate toward zero ’ to ‘ round to even ’ . Our training broke , our error rate went from less than 25 % error to ∼ 99.97 % error. ” To avoid chaotic learning , we first need to understand how it can arise . This is the main motivation of our work . Recently , in the context of “ learning in games ” , Cheung & Piliouras ( 2019 ; 2020 ) presented interesting theoretical analyses to show that in two-person zero-sum and graphical constant-sum games , if the agents employ Multiplicative Weights Update ( MWU ) or Follow-theRegularized-Leader ( FTRL ) algorithms , then Lyapunov chaos occurs everywhere in the cumulative payoff space ; the same result holds for the optimistic variant of MWU ( OMWU ) in coordination games.3 While zero-sum and coordination games are interesting in their own rights , they are rather small subspaces within the family of general normal games . In this paper , we present techniques and tools for characterizing general games where MWU , FTRL , or OMWU are Lyapunov chaotic almost everywhere . Next , we give an overview of our contributions and a discussion of related work . Our Contributions . To show the results about chaos mentioned above , Cheung & Piliouras ( 2019 ; 2020 ) used a classical technique in the study of dynamical systems called volume analysis . Volume analysis considers a set of starting points of positive Lebesgue measure , e.g . a ball centred at a point ; volume is an alternative name for Lebesgue measure in this context . When this set of starting points evolves according to the rule of dynamical system , it becomes a new set with a different volume . Intuitively , volume is a measure of the range of possible outcomes , so the larger it is , the more unpredictable the system is . If the set ’ s volume increases exponentially with time , then its diameter increases exponentially too , which implies Lyapunov chaos . This indicates that when players repeatedly play the games by employing the respective learning algorithms , a slight perturbation on the initiating condition can lead to a wide range of possible cumulative payoffs in the long run . This can be shown to imply instability in the mixed strategy space . The technical starting point of Cheung & Piliouras is to show that when all agents in a bimatrix game G use MWU with step-size , and when a set S in the cumulative payoff space evolves for one time step , its volume change can be expressed as 2 ∫ S CG ( s ) ds +O ( 4 ) , where CG is a function that depends on the game G , which we will define in Section 2 . Clearly , the sign of CG dictates the volume change behaviors for all small enough . Cheung & Piliouras showed that CG is a positive function when G is a two-person zero-sum game . For a large region in the cumulative payoff space , this implies the volume change per time step is Ω ( 2 ) ·volume ( S ) , i.e . volume expands exponentially . They also showed that CG is a negative function if G is a two-person coordination game . To extend their volume expansion results to general bimatrix games , we first discover that CG admits a clean decoupling w.r.t . the canonical decomposition of such games into zero-sum and coordination components ( Basar & Ho ( 1974 ) ; Kalai & Kalai ) . Precisely , given any two-person general game ( A , B ) , it can be written uniquely as a direct sum of a zero-sum game ( Z , −Z ) and a coordination game ( C , C ) , where Z = ( A − B ) /2 and C = ( A + B ) /2 . Interestingly , we find that CG ( · ) = C ( Z , −Z ) ( · ) + C ( C , C ) ( · ) in Lemma 6 . Recall from the last paragraph that C ( Z , −Z ) ( · ) is always positive , while C ( C , C ) ( · ) is always negative . Thus , to see whether volume expansion occurs , it boils down to comparing the strengths of C ( Z , −Z ) ( · ) and −C ( C , C ) ( · ) . We also discover that the function CG is invariant upon additions of trivial matrices to the game G ; see Definition 7 and Lemma 8 . An immediate application of trivial matrices is for bimatrix potential games ( Monderer & Shapley ( 1996 ) ) , for which we show it can be transformed to a coordination game via additions of trivial matrices . With the result in Cheung & Piliouras ( 2020 ) , this immediately implies that OMWU in any bimatrix potential game is Lyapunov chaotic everywhere in the cumulative payoff space ( Observation 10 ) . Based on the above discoveries , we identify two characterizations of bimatrix games where MWU and FTRL are Lyapunov chaotic almost everywhere ( Theorems 15 and 17 ) . As said before , 2For instance , in learning algorithms we often evaluate the function ex , while e is an irrational number . 3They also showed some other chaos results for various combinations of learning algorithms and games . the key is to compare the strengths of the C-functions of the zero-sum and the coordination components . The comparison is done via our new notion of matrix domination ( see Definition 11 ) , and also a linear program ( Eqn . ( 7 ) ) which is designed to prune out the trivial-matrix projection and keep the remaining part minimal . Such family of games has positive Lebesgue measure in the bimatrix game space , so it is not confined to any proper game subspace4 . This justifies the claim that the occurrences of chaos are not only circumstantial , but a rather substantial issue in learning in games . Analogous result holds for OMWU . For games with any number of players ( multi-player games ) , we use an observation in Cheung & Piliouras ( 2019 ) , coupled with our new findings about bimatrix games discussed above , to present a new family of graphical games where MWU is Lyapunov chaotic almost everywhere ( Theorem 18 ) ; the new family strictly includes all graphical constant-sum games . To facilitate volume analyses of learning in multi-player games , we establish their local equivalence of volume change with graphical games . Briefly , we show that CG ( p ) for a general game G is the same as CH ( p ) for some graphical game H ; H will depend on the point p , that ’ s why we say the equivalence is local ( Theorem 19 ) . This provides an intuitive procedure for understanding volume changes , as the volume change of learning in a graphical game is easier to compute . This is used to show that the volume-changing behaviors of MWU and OMWU are opposite to each other . We use these to analyze MWU and OMWU in multi-player potential games ; in particular , we show that CG ( p ) of a multi-player potential game G is identical to CC ( p ) of a corresponding multi-player graphical coordination game , while CC ( p ) ≤ 0 for any p ( Proposition 21 ) .5 Related Work . MWU and its variant , such as FTRL and Optimistic MWU , play important roles in online learning . We recommend the texts of Cesa-Bianchi & Lugosi ( 2006 ) and Hart & Mas-Colell ( 2013 ) for a modern overview of online learning from Machine Learning or Economics perspectives . Recently , there is a stream of works that examine how learning algorithms behave in games or min-max optimization from a dynamical-systemic perspective . This provides new insights into learning systems which could hardly be obtained with classical tools in ML . For instance , some learning systems are shown to be nearly-periodic under the notion of Poincaré recurrence ( Piliouras & Shamma ( 2014 ) ; Mertikopoulos et al . ( 2018 ) ) . Using potential functions first proposed in EGT and other tools from mathematics , surprising behaviors of first-order methods in zero-sum games and min-max optimization were discovered ( Daskalakis & Panageas ( 2018 ; 2019 ) ; Bailey & Piliouras ( 2018 ) ; Cheung ( 2018 ) ) . In Appendix A , we give an account on some further related work . Empirical evidences of Lyapunov chaos of learning in games were reported by Sato et al . ( 2002 ) and Galla & Farmer ( 2013 ) . Li-Yorke chaos , another classical chaos notion , was proved to occur in several learning-in-game systems ( Palaiopanos et al . ( 2017 ) ; Chotibut et al . ( 2020 ) ) . Volume analysis has long been a technique of interest in the study of population and game dynamics . It was discussed in a number of famous texts ; see ( Hofbauer & Sigmund , 1998 , Section 11 ) , ( Fudenberg & Levine , 1998 , Section 3 ) and ( Sandholm , 2010 , Chapter 9 ) . We use game decomposition in this paper , which is a natural and generic approach to extend compelling results for a specific family of games to more general games . Let H denote the specific family of games with some compelling properties . Given a general game , we seek to decompose it into a sum of its projection on H , plus one or more residue components . If the residues are small , then it is plausible that those compelling properties extend ( approximately ) . In seeking of games where learning is stable , game decompositions were used ( Candogan et al . ( 2011 ; 2013a ; b ) ; Letcher et al . ( 2019 ) ) withH being potential games .

Summary: This paper provides tools for classifying the payoff dynamics in general-sum n-player games as Lyapunov chaotic assuming three common algorithms are used: multiplicative weights update (FTRL with entropy regularizer), optimistic MWU, and FTRL with L2 regularizer. Previous work (Cheung & Piliouras) showed that the existence of Lyapunov chaos in the dual space is indicated by the sign of a function C of the game. This work shows that this function can be decomposed into a sum: C of the zero-sum part + C of the coordination part. They also show how to use trivial matrices (which don't affect C) to further reduce parts of the game in a way that eases the analysis. As part of their analysis, they prove that the set of bi-matrix games exhibiting chaos has positive Lebesgue measure and discuss how the relative strength of the zero-sum and coordination parts determines the ultimate sign of C.