Teamwork makes von Neumann work:Min-Max Optimization in Two-Team Zero-Sum Games

Motivated by recent advances in both theoretical and applied aspects of multiplayer games , spanning from e-sports to multi-agent generative adversarial networks , we focus on min-max optimization in team zero-sum games . In this class of games , players are split into two teams with payoffs equal within the same team and of opposite sign across the opponent team . Unlike the textbook twoplayer zero-sum games , finding a Nash equilibrium in our class can be shown to be CLS-hard , i.e. , it is unlikely to have a polynomial-time algorithm for computing Nash equilibria . Moreover , in this generalized framework , we establish that even asymptotic last iterate or time average convergence to a Nash Equilibrium is not possible using Gradient Descent Ascent ( GDA ) , its optimistic variant , and extra gradient . Specifically , we present a family of team games whose induced utility is non-multilinear with non-attractive per-se mixed Nash Equilibria , as strict saddle points of the underlying optimization landscape . Leveraging techniques from control theory , we complement these negative results by designing a modified GDA that converges locally to Nash equilibria . Finally , we discuss connections of our framework with AI architectures with team competition structures like multi-agent generative adversarial networks . 1 INTRODUCTION . Team competition has played a central role in the development of Game Theory ( Marschak , 1955 ; von Stengel & Koller , 1997 ; Bacharach , 1999 ; Gold , 2005 ) , Economics ( Marschak , 1955 ; Gottinger , 1974 ) and Evolutionary Biology ( Nagylaki , 1993 ; Nowak et al. , 2004 ) , however , the behavior of the underlying dynamics within the teams are usually sidelined . Either for reasons of mathematical convenience or bigger picture understanding , “ teams ” in literature are typically modeled as if they were unitary actors , i.e. , single individuals without unveiling the internal decision-making of the team members ( see Kim et al . ( 2019 ) ) . For instance , in the biology setting of weak selection model ( Nagylaki , 1993 ; Chastain et al. , 2014 ; Mehta et al. , 2015 ) species are modeled to compete as teams , while at the crux of the matter the genes of each species are the actual players and their alleles are the actions in the survival game . Similarly , it was the social-media collaboration of the Reddit retail trading crowd as a team that touches off the last year ’ s GameStop frenzy of short squeeze ( Umar et al. , 2021 ; Hasso et al. , 2021 ) transforming the markets into a tug of war game against the team of Wall Street hedge funds . Recently , these intrinsic details behind the competition among teams have attracted renewed interest in the Machine Learning community , motivated by the advent of multi-agent systems that are used for generative tasks or playing complex games like CTF ( Jaderberg et al. , 2019 ) or Starcraft ( Vinyals et al. , 2019 ) . So as to win this kind of games , self-training AI systems have to develop both collaborative attributes ( coordination within each team ) as well as contesting ones ( competition across the teams ) . Moreover , following the complementary thread of multi-agent generative adversarial network research , the creation of a pool by efficient incumbent agents , either in generators ( Arora et al. , 2017 ; Hoang et al. , 2017 ; 2018 ; Zhang et al. , 2018 ; Tang , 2020 ) , or discriminators ( Hardy et al. , 2019 ; Albuquerque et al. , 2019 ) has been tested providing significant statistical and computa- tional benefits . In this direction , researchers strive to harness the efficacy of distributed processing , utilizing shallower networks that can learn all the while more diverse datasets 1 . In order to shed some light on this persistent strain of research , the main premise of the theoretical scaffolding developed in this paper is that The “ unitary two-players ” min-max approach misses the critical component of the collective strategy making within each competing team . Our class of games . In this regard , we turn our attention to Two-Team Zero-Sum games , proposed by Schulman & Vazirani ( 2019b ) , a quite general class of min-max optimization problems that include bilinear games as well as a wide range of non-convex non-concave games . In this class , the players fall in two teams of size k1 , k2 and submit their own probabilistic strategy vector independently , akin to a general normal form multi-player game . Following the econometric common value assumption of Marschak ( 1955 ) , what makes a group of players a team is that in any outcome the players of each team receive an identical payoff . Thus , to build some intuition , it is easy to see that if perfect coordination existed within each team , the interaction between the teams is merely a zero-sum game between two “ virtual ” players . To streamline our presentation here , we defer the more precise description of our model to Section 2 . Challenges behind Two-Team Zero-Sum games . In the archetypical case of two players , i.e. , ( k1 = k2 = 1 ) , min-max strategies are typically thought of as the axiomatically correct predictions thanks to the seminal Von Neumann ’ s minmax theorem ( Von Neumann , 1928 ) . Unfortunately , min-max optimization for case k > 1 is a much more tenuous affair : Schulman & Vazirani ( 2019b ) preclude the existence of unique value by presenting a family of team games where min max 6= max min together with bounds about this duality gap , which quantifies exactly the effect of exchanging the order of strategy commitment either between the teams or the players thereof . If defining the correct figure of merit for Team games is rife with frustration , what is even more demanding is understanding what kind of algorithms/dynamics are able to solve this problem when a game-theoretically meaningful solution exists : Firstly , computing local Nash Equilibria ( NE ) in general non-convex non-concave games is PPAD-complete ( Daskalakis et al. , 2009 ; 2021 ) . Thus , all well-celebrated first-order methods , like gradient descent-ascent ( Lin et al. , 2020 ; Daskalakis & Panageas , 2019 ) , its optimistic ( Popov , 1980 ; Daskalakis & Panageas , 2018 ; Mertikopoulos et al. , 2019 ) and extra gradient variant ( Korpelevich , 1976 ) would require an exponential number of steps in the parameters of the problem to find an appoximate NE under Nemirovsky-Yudin ( Nemirovskij & Yudin , 1983 ) oracle optimization model . Secondly , even if a regret notion could be defined , noregret methodology is guarranteed to attract only to the set of coarse correlated equilibria ( CCE ) ( Fudenberg , 1991 ; Hannan , 2016 ; Flokas et al. , 2020 ; Giannou et al. , 2021 ) , a weaker notion that may be exclusively supported on strictly dominated strategies , even for simple symmetric two-player games ( See also Viossat & Zapechelnyuk ( 2013 ) ) . Whilst the aforementioned intractability failures for the general case of non-convex non-concave min-max problems provides significant insights , they can not a fortiori answer the fundamental question , restricted in the model of Two-Team Zero-Sum Games : Can we compute Nash equilibria in Two-Team Zero-Sum Games and ultimately are there first-order methods that converge to them under tangible guarantees ? Our results . To the best of our knowledge , the following contributions are the first-of-its-kind type of results for the case of Two-Team Zero-Sum games : • For the case of the computational complexity of approximate ( possibly mixed ) NE we establish a sweeping negative result proving that it is CLS-hard ( Theorem 3.1 ) , i.e. , is computationally harder than finding pure NE in a congestion game or finding approximate gradient descent fixed points . • From an optimization perspective , we settle these questions with a resounding “ no ” for all the well-known discrete gradient flow variations . Specifically , we present a simple family of twoteam with two-players zero-sum games where Projected-GDA , Optimistic-GDA , and Extra Gradient fail even to stabilize around a mixed NE , when they are initialized nearby ( Theorem 3.5 ) . 1Indeed , from the training perspective , it ’ s more computationally preferable to back-propagate through two equally sized neural networks with smaller capacity rather than through a giant single one that would be twice as deep . ( Tang , 2020 ) Additionally , for the category GDA in the non-degenerate team games with unique mixed NE , one could acquire an even stronger result for any high-dimensional configuration of actions and players . ( Theorem 3.2 ) • In order to make some substantial headway under the burden of the above instability results , we shift our attention to adaptive control generalizations of the celebrated Washout filters–traditionally used for stabilizing the Dutch-roll motion of an aircraft during a flight ( Hassouneh et al. , 2004 ; Grant & Reid , 1997 ) . Inspired by this framework , we propose the modified KPV-GDA2 which consists of a tandem combination of GDA together with a stabilizing feedback introduces by Bazanella et al . ( 1997 ) . { state ( k+1 ) = state ( k ) + ηGDA ( state ( k+1 ) ) + ηK ( state ( k ) − stress ( k ) ) stress ( k+1 ) = stress ( k ) + ηP ( state ( k ) − stress ( k ) ) ( KPV-GDA ) The main linchpin of KPV-GDA method is the Simon ’ s and Theil ’ s ( Simon , 1956 ; Theil , 1957 ) certainty equivalence principle , a widely used methodology in Control theory in developing applied dynamic rational expectations models . According to this principle , the feedback law is split into two optimization steps whereby K−step attracts quickly the state to the stress while P−step converges slowly to the fixed points of GDA . Compared with the plethora of the proposed dynamics for min-max problems , the crucial advantage of the afore-described technique is that does not introduce any extra fixed points than GDA ’ s ones . In Section 2.2 , we provide some illustrative examples of KPV-GDA technique , while in Theorem 3.7 we prove the existence of such control feedback for our class of games . • Finally , in Section 4 we provide a series of experiments in simple two-team zero-sum games showcasing both the messy behaviors of traditional methods like GDA , OGDA and the power of KPV-GDA method in these optimization environments . Additionally , we show that multi-agent GAN architectures achieve better performance than the single-agent ones , in terms of network capacity , when they are trained in synthetic or real-world datasets like CIFAR10 . 2 PRELIMINARIES . 2.1 DEFINITIONS . Our setting . Formally , a two-team game in normal form is defined as a tuple Γ = Γ ( N , A , u ) consisting of ( i ) a finite set of players N , split into two teams A , B with kA and kB players correspondingly such that : N = NA ∪ NB = { A1 , · · · , AkA , B1 , · · · , BkB } ; ( ii ) a finite set of actions ( or pure strategies ) Ai = { α1 , . . . , αni } per player i ∈ N ; ( iii ) each team ’ s payoff function uA , uB : A → R , where A : = ∏ iAi denotes the ensemble of all possible action profiles α = ( αA1 , . . . , αAkA , αB1 , . . . , αBkB ) while the individual utility of a player is identical to her teammates , i.e. , ui = uA & uj = uB ∀ ( i , j ) ∈ NA×NB . In this general context , players could also adhere mixed strategies , i.e , probability distributions sk ∈ ∆ ( Ak ) over the pure strategies αk ∈ Ak . Correspondingly , we define the product distributions x = sA1 ⊗ · · · ⊗ sAkA , y = sB1 ⊗ · · · ⊗ sBkB as the teams ’ strategies . Collectively , we will write X : = ∏ i∈NA Xi = ∏ i∈NA ∆ ( Ai ) , Y : =∏ i∈NA Yi = ∏ i∈NB ∆ ( Ai ) the space of mixed strategy profiles of teams A , B . Similarly with the bilinear two-player games , the teams ’ utility functions can be expressed via the payoff-tensors A , B ∈ Rτ with τ = ∏ i∈N |Ai| and acquire the form3 : uA = Ayx & uB = B y x ( 2.1 ) 2The name “ KPV-GDA ” is an initialism from ( K , P ) -Vaned Gradient Descent Ascent method ; Just like the tail section of an aircraft where the vanes are flight control surfaces that control the unstable yaw , ( K , P ) control feedback aims to stabilize the unstable arrows of Gradient flow around a mixed NE . 3Figuratively , the latter form denotes what is known as a tensor contraction given : Ayx =∑ i , ... , j , k ... , l xi , ... , jAi , ... , j , k ... , lyk ... , l If x , y have shapes ( i , j ) and ( k , l ) that would be equivalent to u = einsum ( ’ ijkl , ij , kl ’ , A , x , y ) In terms of solutions , we focus on the per player Nash Equilibrium ( NE ) , i.e. , a state strategy profile s∗ = ( x , y ) = ( ( s∗A1 , . . . , s ∗ AkA ) , ( s∗B1 , . . . , s ∗ BkB ) ) such that ui ( s ∗ ) ≥ ui ( si ; s∗−i ) 4 for all si ∈ ∆ ( Ai ) and all i ∈ N ( NE ) The state strategy profile s∗ is called pure if every player of both teams chooses a single action ; otherwise we say that it is mixed . Finally , a two-team game is called two-team zero-sum if uA = −uB or equivalently A + B = O . Remark 2.1 . A quite technical prerequisite for the rest of this work , we will assume that a succinct representation of the utility tensors of the game is available or equivalently that a payoff oracle provides efficiently both the value of the utility function and its derivatives for a specific input , which is consistent with the vast majority of the applications that are described in the literature ( von Stengel & Koller , 1997 ) . A first approach on computing Nash equilibria in Two-Team Zero-Sum games . Given the existence of the duality-gap between the min max and max min , in lieu of the two-player zero-sum game , an equilibrium in our setting can not be computed via linear programming . For the goal of computing Nash equilibria in two-team zero-sum games , we have experimented with a selection of first-order methods that have been utilized with varying success in the setting of the two-person zerosum case . Namely , we analyze the following methods : i ) Gradient Descent-Ascent ii ) Optimistic Gradient Descent-Ascent iii ) Extra Gradient Method iv ) Optimistic Multiplicative Weights Update Method We defer their precise definitions in Appendix B . The below remark will play a key role in the sequel . Remark 2.2 . Any fixed point of the aforementioned discrete-time dynamics on the utility function corresponds necessarily to the Nash equilibria of the game . Hence , an important testbed for the long-run behavior of GDA , OGDA , and EG methods is to examine whether these methods stabilize around their fixed points , which effectively constitute the Nash equilibria of the game . In Section 3.2 , we show that in lack of pure Nash equilibria , all the above methods fail to stabilize on their fixed points even for a simple class of ( 2 , 2 ) -players game , and as a consequence to the mixed Nash equilibria of the game . The presence of these results showcases the need for a different approach that lies outside purely optimization-based ideas . Inspired by the applications of washout filters to stabilize highly susceptible systems and their adaptive control generalizations , we design a new incarnation of GDA vaned by two matrices-control feedback . Surprisingly , in contrast with the aforementioned traditional methods , our proposed technique accomplishes last-iterate stabilization on its fixed point , i.e. , the mixed Nash equilibria of the team game . ( K , P ) -Vaned GDA Method . After concatenating the vectors of the minimizing and the maximizing agents z ( k ) = ( x ( k ) , y ( k ) ) we can write our method , for appropriate matrices K , P : z ( k+1 ) = ΠZ { z ( k ) + η ( −∇xf ( z ( k ) ) ∇yf ( z ( k ) ) ) + ηK ( z ( k ) − θ ( k ) ) } θ ( k+1 ) = ΠZ { θ ( k ) + ηP ( z ( k ) − θ ( k ) ) } ( 2.2 ) Intuitively , the added variable θ ( k ) holds an estimate of the fixed point , and through the feedback ηK ( z ( k ) − θ ( k ) ) the vector z stabilizes around that estimate which slowly moves towards the real fixed point of the plain GDA dynamic . It is crucial to note that no additional fixed points are introduced to the system .

Summary: This work considers two-team zero-sum games where two teams with opposite objectives are facing each other. While the literature is quite extensive on learning in zero-sum games, very little is known for two-team zero sum-games. This paper first shows that this problem is much harder than classical zero-sum games by showing that the computation of a NE is CLS hard. Besides showing the failure of classical optimization methods for zero-sum games, the authors propose an optimisation algorithm better suited to this setting, which converges locally to a NE for carefully tuned hyperparameters.