A Generalized Training Approach for Multiagent Learning

1 INTRODUCTION . Creating agents that learn to interact in large-scale systems is a key challenge in artificial intelligence . Impressive results have been recently achieved in restricted settings ( e.g. , zero-sum , two-player games ) using game-theoretic principles such as iterative best response computation ( Lanctot et al. , 2017 ) , self-play ( Silver et al. , 2018 ) , and evolution-based training ( Jaderberg et al. , 2019 ; Liu et al. , 2019 ) . A key principle underlying these approaches is to iteratively train a growing population of player policies , with population evolution informed by heuristic skill ratings ( e.g. , Elo ( Elo , 1978 ) ) or game-theoretic solution concepts such as Nash equilibria . A general application of this principle is embodied by the Policy-Space Response Oracles ( PSRO ) algorithm and its related extensions ( Lanctot et al. , 2017 ; Balduzzi et al. , 2019 ) . Given a game ( e.g. , poker ) , PSRO constructs a higherlevel meta-game by simulating outcomes for all match-ups of a population of players ’ policies . It then trains new policies for each player ( via an oracle ) against a distribution over the existing meta-game policies ( typically an approximate Nash equilibrium , obtained via a meta-solver1 ) , appends these new policies to the meta-game population , and iterates . In two-player zero sum games , fictitious play ( Brown , 1951 ) , double oracle ( McMahan et al. , 2003 ) , and independent reinforcement learning can all be considered instances of PSRO , demonstrating its representative power ( Lanctot et al. , 2017 ) . Prior applications of PSRO have used Nash equilibria as the policy-selection distribution ( Lanctot et al. , 2017 ; Balduzzi et al. , 2019 ) , which limits the scalability of PSRO to general games : Nash equilibria are intractable to compute in general ( Daskalakis et al. , 2009 ) ; computing approximate Nash equilibria is also intractable , even for some classes of two-player games ( Daskalakis , 2013 ) ; finally , when they can be computed , Nash equilibria suffer from a selection problem ( Harsanyi et al. , 1988 ; Goldberg et al. , 2013 ) . It is , thus , evident that the reliance of PSRO on the Nash equilibrium as the driver of population growth is a key limitation , preventing its application to general games . Recent work has proposed a scalable alternative to the Nash equilibrium , called α-Rank , which applies readily to general games ( Omidshafiei et al. , 2019 ) , making it a promising candidate for population-based training . Given that the formal study of PSRO has only been conducted under the restricted settings determined by the limitations of Nash equilibria , establishing its theoretical and empirical behaviors under alternative meta-solvers remains an important and open research problem . We study several PSRO variants in the context of general-sum , many-player games , providing convergence guarantees in several classes of such games for PSRO instances that use α-Rank as a meta-solver . We also establish connections between Nash and α-Rank in specific classes of games , and identify links between α-Rank and the Projected Replicator Dynamics employed in prior PSRO instances ( Lanctot et al. , 2017 ) . We develop a new notion of best response that guarantees convergence to the α-Rank distribution in several classes of games , verifying this empirically in randomly-generated general-sum games . We conduct empirical evaluations in Kuhn and Leduc Poker , first establishing our approach as a competitive alternative to Nash-based PSRO by focusing on two-player variants of these games that have been investigated in these prior works . We subsequently demonstrate empirical results extending beyond the reach of PSRO with Nash as a meta-solver by evaluating training in 3- to 5-player games . Finally , we conduct preliminary evaluations in MuJoCo soccer ( Liu et al. , 2019 ) , another complex domain wherein we use reinforcement learning agents as oracles in our proposed PSRO variants , illustrating the feasibility of the approach . 2 PRELIMINARIES . Games We consider K-player games , where each player k ∈ [ K ] has a finite set of pure strategies Sk . Let S = ∏ k S k denote the space of pure strategy profiles . Denote by S−k = ∏ l 6=k S l the set of pure strategy profiles excluding those of player k. Let M ( s ) = ( M1 ( s ) , . . . , MK ( s ) ) ∈ RK denote the vector of expected player payoffs for each s ∈ S. A game is said to be zero-sum if∑ kM k ( s ) = 0 for all s ∈ S. A game is said to be symmetric if all players have identical strategy sets Sk , and for any permutation ρ , strategy profile ( s1 , . . . , sK ) ∈ S , and index k ∈ [ K ] , one has Mk ( s1 , . . . , sK ) = Mρ ( k ) ( sρ ( 1 ) , . . . , sρ ( K ) ) . A mixed strategy profile is defined as π ∈ ∆S , a tuple representing the probability distribution over pure strategy profiles s ∈ S. The expected payoff to player k under a mixed strategy profile π is given byMk ( π ) = ∑ s∈S π ( s ) M k ( s ) . Nash Equilibrium ( NE ) Given a mixed profile π , the best response for a player k is defined BRk ( π ) = arg maxν∈∆ Sk [ Mk ( ν , π−k ) ] . A factorized mixed profile π ( s ) = ∏ k π k ( sk ) is a Nash equilibrium ( NE ) if πk ∈ BRk ( π ) for all k ∈ [ K ] . Define NASHCONV ( π ) =∑ kM k ( BRk ( π ) , π−k ) −Mk ( π ) ; roughly speaking , this measures “ distance ” from an NE ( Lanctot et al. , 2017 ) . In prior PSRO instances ( Lanctot et al. , 2017 ) , a variant of the replicator dynamics ( Taylor and Jonker , 1978 ; Maynard Smith and Price , 1973 ) , called the Projected Replicator Dynamics ( PRD ) , has been used as an approximate Nash meta-solver ( see Appendix E for details on PRD ) . α-Rank While NE exist in all finite games ( Nash , 1950 ) , their computation is intractable in general games , and their non-uniqueness leads to an equilibrium-selection problem ( Harsanyi et al. , 1988 ; Goldberg et al. , 2013 ) . This limits their applicability as the underlying driver of training beyond the two-player , zero-sum regime . Recently , an alternate solution concept called α-Rank was proposed by 1A meta-solver is a method that computes , or approximates , the solution concept that is being deployed . Player 1 ’ s policy set ...... Player k ’ s policy set Player K ’ s policy set Profile distribution Meta-solver Oracle ...... Profile distribution Randomly initialize player policy sets ... ... Game simulations ( a ) Complete : compute missing payoff tensorM entries via game simulations . ( b ) Solve : given the updated payoff tensor M , calculate metastrategy π via meta-solver M. ( c ) Expand : append a new policy to each player ’ s policy space using the oracle O . Figure 1 : Overview of PSRO ( M , O ) algorithm phases . Omidshafiei et al . ( 2019 ) , the key associated benefits being its uniqueness and efficient computation in many-player and general-sum games , making it a promising means for directing multiagent training . The α-Rank distribution is computed by constructing the response graph of the game : each strategy profile s ∈ S of the game is a node of this graph ; a directed edge points from any profile s ∈ S to σ ∈ S in the graph if ( 1 ) s and σ differ in only a single player k ’ s strategy and ( 2 ) Mk ( σ ) > Mk ( s ) . α-Rank constructs a random walk along this directed graph , perturbing the process by injecting a small probability of backwards-transitions from σ to s ( dependent on a parameter , α , whose value is prescribed by the algorithm ) ; this ensures irreducibility of the resulting Markov chain and the existence of a unique stationary distribution , π ∈ ∆S , called the α-Rank distribution . The masses of π are supported by the sink strongly-connected components ( SSCCs ) of the response graph ( Omidshafiei et al. , 2019 ) . For more details on α-Rank , see Appendix D and Rowland et al . ( 2019 ) . Oracles We define an oracle O as an abstract computational entity that , given a game , computes policies with precise associated properties . For instance , a best-response oracle Ok ( π ) = BRk ( π ) computes the best-response policy for any player k , given a profile π . One may also consider approximate-best-response oracles that , e.g. , use reinforcement learning to train a player k ’ s policy against a fixed distribution over the other players ’ policies , π−k . Oracles play a key role in populationbased training , as they compute the policies that are incrementally added to players ’ growing policy populations ( McMahan et al. , 2003 ; Lanctot et al. , 2017 ; Balduzzi et al. , 2019 ) . The choice of oracle O also affects the training convergence rate and final equilibrium reached ( e.g. , Nash or α-Rank ) . Empirical Game-theoretic Analysis PSRO relies on principles from empirical game-theoretic analysis ( EGTA ) ( Walsh et al. , 2002 ; Phelps et al. , 2004 ; Wellman , 2006 ) . Given a game ( e.g. , poker ) , EGTA operates via construction of a higher-level ‘ meta-game ’ , where strategies s correspond to policies ( e.g. , ‘ play defensively ’ in poker ) rather than atomic actions ( e.g. , ‘ fold ’ ) . A meta-payoff table M is then constructed by simulating games for all joint policy combinations , with entries corresponding to the players ’ expected utilities under these policies . Game-theoretic analysis can then be conducted on the meta-game in a manner analogous to the lower-level game , albeit in a much more scalable manner . As the theoretical discussion hereafter pertains to the meta-game , we use s , M , and π to respectively refer to policies , payoffs , and distributions at the meta-level , rather than the underlying low-level game . In our analysis , it will be important to distinguish between SSCCs of the underlying game , and of the meta-game constructed by PSRO ; we refer to the latter as meta-SSCCs . 3 POLICY-SPACE RESPONSE ORACLES : NASH AND BEYOND . We first overview Policy-Space Response Oracles ( PSRO ) prior to presenting our findings . Given an underlying game ( e.g. , Poker ) , PSRO first initializes the policy space S using randomly-generated policies , then expands the players ’ policy populations in three iterated phases : complete , solve , and Algorithm 1 PSRO ( M , O ) 1 : Initialize the players ’ policy set S = ∏ k S k via random policies 2 : for iteration ∈ { 1 , 2 , · · · } do 3 : Update payoff tensorM for new policy profiles in S via game simulations . ( Fig . 1a ) 4 : Compute the meta-strategy π using meta-solverM ( M ) . ( Fig . 1b ) 5 : Expand the policy space for each player k ∈ [ K ] via Sk ← Sk ∪ Ok ( π ) . ( Fig . 1c ) Game type M O Converges to α-Rank ? SP α-Rank BR 7 ( Example 1 ) SP α-Rank PBR 3 ( Sub-SSCC , † Proposition 3 ) MP α-Rank BR 7 ( Example 2 ) MP α-Rank PBR 3 ( With novelty-bound oracle , † Proposition 1 ) SP / MP Uniform or Nash BR 7 ( Examples 4 and 5 , Appendix A.2 ) Table 1 : Theory overview . SP and MP , resp. , denote single and multi-population games . BR and PBR , resp. , denote best response and preference-based best response . †Defined in the noted propositions . expand ( see Algorithm 1 and Fig . 1 ) . In the complete phase , a meta-game consisting of all match-ups of these joint policies is synthesized , with missing payoff entries in M completed through game simulations . Next , in the solve phase , a meta-solverM computes a profile π over the player policies ( e.g. , Nash , α-Rank , or uniform distributions ) . Finally , in the expand phase , an oracle O computes at least one new policy s′k for each player k ∈ [ K ] , given profile π . As other players ’ policy spaces S−k and profile π−k are fixed , this phase involves solving a single-player optimization problem . The new policies are appended to the respective players ’ policy sets , and the algorithm iterates . We use PSRO ( M , O ) to refer to the PSRO instance using meta-solverM and oracle O . Notably , PSRO-based training for two-player symmetric games can be conducted using a single population of policies that is shared by all players ( i.e. , Sk is identical for all k ) . Thus , we henceforth refer to twoplayer symmetric games as ‘ single-population games ’ , and more generally refer to games that require player-specific policy populations as ‘ multi-population games ’ . Recent investigations of PSRO have solely focused on Nash-based meta-solvers and best-response-based oracles ( Lanctot et al. , 2017 ; Balduzzi et al. , 2019 ) , with theory focused around the two-player zero-sum case . Unfortunately , these guarantees do not hold in games beyond this regime , making investigation of alternative meta-solvers and oracles critical for further establishing PSRO ’ s generalizability .

Summary: This paper extends the original PSRO paper to use an $\alpha$-Rank based metasolver instead of the projected replicator dynamics and Nash equilibria based metasolvers in the original. To this end, the paper modifies the original idea of Best-Response (BR) oracle since it can ignore some strategies in $\alpha$-Rank defining SSCC to introduce the idea of _preference-based_ Best-Response (PBR) oracle. The need for a different oracle is well justified especially with the visualization in the Appendix. The main contributions that the paper seems to be going for is a theoretical analysis of $\alpha$-Rank based PSRO compared to standard PSRO. From the PBR's description (especially in Sec 4.3) it seems the paper is intereseted in expanding the population with novel agents rather than finding the "best" single agent which is not well defined for complex games with intransitivities. Nevertheless, it seems that BR is mostly compatible with PBR for symmetric zero-sum two-player games.