NeuPL: Neural Population Learning

Learning in strategy games ( e.g . StarCraft , poker ) requires the discovery of diverse policies . This is often achieved by iteratively training new policies against existing ones , growing a policy population that is robust to exploit . This iterative approach suffers from two issues in real-world games : a ) under finite budget , approximate best-response operators at each iteration needs truncating , resulting in under-trained good-responses populating the population ; b ) repeated learning of basic skills at each iteration is wasteful and becomes intractable in the presence of increasingly strong opponents . In this work , we propose Neural Population Learning ( NeuPL ) as a solution to both issues . NeuPL offers convergence guarantees to a population of best-responses under mild assumptions . By representing a population of policies within a single conditional model , NeuPL enables transfer learning across policies . Empirically , we show the generality , improved performance and efficiency of NeuPL across several test domains1 . Most interestingly , we show that novel strategies become more accessible , not less , as the neural population expands . The need for learning not one , but a population of strategies is rooted in classical game theory . Consider the purely cyclical game of rock-paper-scissors , the performance of individual strategies is meaningless as improving against one entails losing to another . By contrast , performance can be meaningfully examined between populations . A population consisting of pure strategies { rock , paper } does well against a singleton population of { scissors } because in the meta-game where both populations are revealed , a player picking strategies from the former can always beat a player choosing from the latter2 . This observation underpins the unifying population learning framework of Policy Space Response Oracle ( PSRO ) where a new policy is trained to best-respond to a mixture over previous policies at each iteration , following a meta-strategy solver ( Lanctot et al. , 2017 ) . Most impressively , Vinyals et al . ( 2019 ) explored the strategy game of StarCraft with a league of policies , using a practical variation of PSRO . The league counted close to a thousand sophisticated deep RL agents as the population collectively became robust to exploits . Unfortunately , such empirical successes often come at considerable costs . Population learning algorithms with theoretical guarantees are traditionally studied in normal-form games ( Brown , 1951 ; McMahan et al. , 2003 ) where best-responses can be solved exactly . This is in stark contrast to real-world Game-of-Skills ( Czarnecki et al. , 2020 ) — such games are often temporal in nature , where best-responses can only be approximated with computationally intensive methods ( e.g . deep RL ) . This has two implications . First , for a given opponent , one can not efficiently tell apart good-responses that temporarily plateaued at local optima from globally optimal best-responses . As a result , approximate best-response operators are often truncated prematurely , according to hand-crafted schedules ( Lanctot et al. , 2017 ; Mcaleer et al. , 2020 ) . Second , real-world games often afford strategy-agnostic transitive ∗Currently at Reality Labs , work carried out while at DeepMind . †Work carried out while at DeepMind . 1See https : //neupl.github.io/demo/ for supplementary illustrations . 2This is formally quantified by Relative Population Performance , see Definition A.1 ( Balduzzi et al. , 2019 ) . skills that are pre-requisite to strategic reasoning . Learning such skills from scratch at each iteration in the presence of evermore skillful opponents quickly becomes intractable beyond a few iterations . This iterative and isolated approach is fundamentally at odds with human learning . For humans , mastering diverse strategies often facilitates incremental strategic innovation and learning about new strategies does not stop us from revisiting and improving upon known ones ( Caruana , 1997 ; Krakauer et al. , 2006 ) . In this work , we make progress towards endowing artificial agents with similar capability by extending population learning to real-world games . Specifically , we propose NeuPL , an efficient and general framework that learns and represents diverse policies in symmetric zero-sum games within a single conditional network , using the computational infrastructure of simple self-play ( Section 1.2 ) . Theoretically , we show that NeuPL converges to a sequence of iterative best-responses under certain conditions ( Section 1.3 ) . Empirically , we illustrate the generality of NeuPL by replicating known results of population learning algorithms on the classical domain of rockpaper-scissors as well as its partially-observed , spatiotemporal counterpart running-with-scissors ( Vezhnevets et al. , 2020 ) ( Section 2.1 ) . Most interestingly , we show that NeuPL enables transfer learning across policies , discovering exploiters to strong opponents that would have been inaccessible to comparable baselines ( Section 2.2 ) . Finally , we show the appeal of NeuPL in the challenge domain of MuJoCo Football ( Liu et al. , 2019 ) where players must continuously refine their movement skills in order to coordinate as a team . In this highly transitive game , NeuPL naturally represents a short sequence of best-responses without the need for a carefully chosen truncation criteria ( Section 2.4 ) . 1 METHODS . Our method is designed with two desiderata in mind . First , at convergence , the resulting population of policies should represent a sequence of iterative best-responses under reasonable conditions . Second , transfer learning can occur across policies throughout training . In this section , we define the problem setting of interests as well as necessary terminologies . We then describe NeuPL , our main conceptual algorithm as well as its theoretical properties . To make it concrete , we further consider deep RL specifically and offer two practical implementations of NeuPL for real-world games . 1.1 PRELIMINARIES . Approximate Best-Response ( ABR ) in Stochastic Games We consider a symmetric zero-sum Stochastic Game ( Shapley , 1953 ) defined by ( S , O , X , A , P , R , p0 ) with S the state space , O the observation space and X : S → O × O the observation function defining the ( partial ) views of the state for both players . Given joint actions ( at , a′t ) ∈ A × A , the state follows the transition distribution P : S ×A×A → Pr ( S ) . The reward functionR : S → R× R defines the rewards for both players in state st , denotedR ( st ) = ( rt , −rt ) . The initial state of the environment follows the distribution p0 . In a given state st , players act according to policies ( π ( ·|o≤t ) , π′ ( ·|o′≤t ) ) . Player π achieves an expected return of J ( π , π′ ) = Eπ , π′ [ ∑ t rt ] against π ′ . Policy π∗ is a best response to π′ if ∀π , J ( π∗ , π′ ) ≥ J ( π , π′ ) . We define π̂ ← ABR ( π , π′ ) with J ( π̂ , π′ ) ≥ J ( π , π′ ) . In other words , an ABR operator yields a policy π̂ that does no worse than π , in the presence of an opponent π′ . Meta-game Strategies in Population Learning Given a symmetric zero-sum game and a set of N policies Π : = { πi } Ni=1 , we define a normal-form meta-game where players ’ i-th action corresponds to executing policy πi for one episode . A meta-game strategy σ thus defines a probability assignment , or an action profile , over Π . Within Π , we define U ∈ RN×N ← EVAL ( Π ) to be the expected payoffs between pure strategies of this meta-game or equivalently , Uij : = J ( πi , πj ) in the underlying game . We further extend the ABR operator of the underlying game to mixture policies represented by σ , such that π̂ ← ABR ( π , σ , Π ) with Eπ′∼P ( σ ) [ J ( π̂ , π′ ) ] ≥ Eπ′∼P ( σ ) [ J ( π , π′ ) ] . Finally , we define f : R|Π|×|Π| → R|Π| to be a meta-strategy solver ( MSS ) with σ ← f ( U ) and F : RN×N → RN×N a meta-graph solver ( MGS ) with Σ ← F ( U ) . The former formulation is designed for iterative optimization of approximate best-responses as in Lanctot et al . ( 2017 ) whereas the latter is motivated by concurrent optimization over a set of population-level objectives as in Garnelo et al . ( 2021 ) . In particular , Σ ∈ RN×N : = { σi } Ni=1 defines N population-level objectives , with πi optimized against the mixture policy represented by σi and Π . As such , Σ ∈ RN×N corresponds to the adjacency matrix of an interaction graph . Figure 1 illustrates several commonly used population learning algorithms defined by Σ or equivalently , their interaction graphs . 1.2 NEURAL POPULATION LEARNING . We now present NeuPL and contrast it with Policy-Space Response Oracles ( PSRO , Lanctot et al . ( 2017 ) ) which similarly focuses on population learning with approximate best-responses by RL . Algorithm 1 Neural Population Learning ( Ours ) 1 : Πθ ( ·|s , σ ) . Conditional neural population net . 2 : Σ : = { σi } Ni=1 . Initial interaction graph . 3 : F : RN×N → RN×N . Meta-graph solver . 4 : while true do 5 : ΠΣθ ← { Πθ ( ·|s , σi ) } Ni=1 . Neural population . 6 : for σi ∈ UNIQUE ( Σ ) do 7 : Πσiθ ← Πθ ( ·|s , σi ) 8 : Πσiθ ← ABR ( Π σi θ , σi , Π Σ θ ) . Self-play . 9 : U ← EVAL ( ΠΣθ ) . ( Optional ) if F adaptive . 10 : Σ← F ( U ) . ( Optional ) if F adaptive . 11 : return Πθ , Σ Algorithm 2 PSRO ( Lanctot et al. , 2017 ) 1 : Π : = { π0 } . Initial policy population . 2 : σ ← UNIF ( Π ) . Initial meta-game strategy . 3 : f : R|Π|×|Π| → R|Π| . Meta-strategy solver . 4 : 5 : for i ∈ [ [ N ] ] do . N-step ABR . 6 : Initialize πθi . 7 : πθi ← ABR ( πθi , σ , Π ) 8 : Π← Π ∪ { πθi } 9 : U ← EVAL ( Π ) . Empirical payoffs . 10 : σ ← f ( U ) 11 : return Π NeuPL deviates from PSRO in two important ways . First , NeuPL suggests concurrent and continued training of all unique policies such that no good-response features in the population prematurely due to early truncation . Second , NeuPL represents an entire population of policies via a shared conditional network Πθ ( ·|s , σ ) with each policy Πθ ( ·|s , σi ) conditioned on and optimised against a meta-game mixture strategy σi , enabling transfer learning across policies . This representation also makes NeuPL general : it delegates the choice of effective population sizes |UNIQUE ( Σ ) | ≤ |Σ| = N to the meta-graph solver F as σi = σj implies Πθ ( ·|s , σi ) ≡ Πθ ( ·|s , σj ) ( cf . Section 2.1 ) . Finally , NeuPL allows for cyclic interaction graphs , beyond the scope of PSRO . We discuss the generality of NeuPL in the context of prior works in further details in Appendix D. N-step Best-Responses via Lower-Triangular Graphs A popular class of population learning algorithms seeks to converge to a sequence of N iterative best-responses where each policy πi is a best-response to an opponent meta-game strategy σi with support over a subset of the policy population Π < i = { πj } j < i . In NeuPL , this class of algorithms are implemented with meta-graph solvers that return lower-triangular adjacency matrices Σ with Σi≤j = 0 . Under this constraint , σ0 becomes a zero vector , implying that Πθ ( ·|s , σ0 ) does not seek to best-respond to any policies . Similar to the role of initial policies { π0 } in PSRO ( Algorithm 2 ) , Πθ ( ·|s , σ0 ) serves as a starting point for the sequence of N-step best-responses and any fixed policy can be used . We note that this property further allows for incorporating pre-trained policies in NeuPL , as we discuss in Appendix D.1 . Algorithm 3 A meta-graph solver implementing PSRO-NASH . 1 : function FPSRO-N ( U ) . U ∈ RN×N the empirical payoff matrix . 2 : Initialize meta-game strategies Σ ∈ RN×N with zeros . 3 : for i ∈ { 1 , . . . , N − 1 } do 4 : Σi+1,1 : i ← SOLVE-NASH ( U1 : i,1 : i ) . LP Nash solver , see Shoham & Leyton-Brown ( 2008 ) . 5 : return Σ One prominent example is PSRO-NASH , where πi is optimized to best-respond to the Nash mixture policy over Π < i . This particular meta-graph solver is shown in Algorithm 3 .

Summary: Population based training (PBT) algorithms progressively grow a set of policies by adding best-responses to mixtures of the existing population. When RL is used as a best-response method the new policy is generally not a best-response but instead a good response. Moreover, the good-response is initialized tabula rosa potentially relearning responses to policies responded to in previous iterations. To address these concerns, this work proposes NeuPL, where a single policy is trained to represent the entire population by conditioning it on the opponent's mixture. Moreover, this proposed algorithm introduces the use of interaction graphs as a means to codify match-making in a continuously training population. NeuPL is restricted to zero-sum symmetric games with fixed player sizes.