Iterative Empirical Game Solving via Single Policy Best Response

1 INTRODUCTION . In ( single-agent ) reinforcement learning ( RL ) , an agent repeatedly interacts with an environment until it achieves mastery , that is , can find no further way to improve performance ( measured by reward ) . In a multiagent system , the outcome of any learned policy may depend pivotally on the behavior of other agents . Direct search over a complex joint policy space for game solutions is daunting , and so more common in approaches combining RL and game theory is to interleave learning and game analysis iteratively . In empirical game-theoretic analysis ( EGTA ) , game reasoning is applied to approximate game models defined over restricted strategy sets . The strategies are selected instances drawn from a large space of possible strategies , for example , defined in terms of a fundamental policy space : mappings from observations to actions . For a given set of strategies ( policies ) ,1 we estimate an empirical game via simulation over combinations of these strategies . In iterative approaches to EGTA , analysis of the empirical game informs decisions about the sampling of strategy profiles or additions of strategies to the restricted sets . Schvartzman & Wellman ( 2009b ) first combined RL with EGTA , adding new strategies by learning a policy for one agent ( using tabular Q-learning ) , fixing other agent policies at a Nash equilibrium of the current empirical game . Lanctot et al . ( 2017 ) introduced a general framework , Policy-Space Response Oracles ( PSRO ) , interleaving empirical game analysis with deep RL . PSRO generalizes the identification of learning targets , employing an abstract method termed meta-strategy solver ( MSS ) that extracts a strategy profile from an empirical game . At each epoch of PSRO , a player uses RL to derive a new policy by training against the opponentstrategies in the MSS-derived profile . A particular challenge for the RL step in PSRO is that the learner must derive a response under uncertainty about opponent policies . The profile returned by the MSS is generally a mixed-strategy profile , as in strategically complex environments randomization is often a necessary ingredient for equilibrium . The opponent draws from this mixture are unobserved , adding uncertainty to the multiagent environment . 1The term policy as employed in the RL context corresponds exactly to the game-theoretic notion of strategy in our setting , and we use the terms interchangeably . We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies . We propose and evaluate two such methods , which work in qualitatively different ways : Mixed-Oracles learns separate best-responses ( BR ) to each pure strategy in a mixture and combines the results from learning to approximate a BR to the mixture . Mixed-Opponents constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy . Both of our methods employ the machinery of Q-Mixing ( Smith et al. , 2020 ) , which constructs policies based on an aggregation of Q-functions corresponding to component policies of a mixture . Our methods promise advantages beyond those of learning in a less stochastic environment . Mixed-Oracles transfers learning across epochs , exploiting the Q-functions learned against a particular opponent policy in constructing policies for any other epoch where that opponent policy is encountered . Mixed-Opponents applies directly over the joint opponent space , and so has the potential to scale beyond two-player games . We evaluate our methods in a series of experimental games , and find that both Mixed-Oracles and Mixed-Opponents are able to find solutions at least as good as an unmodified PSRO , and often better , while providing a substantial reduction in the required amount of training simulation . 2 PRELIMINARIES . An RL agent at time t ∈ T receives the state of the environment st ∈ S , or a partial state called an observation ot∈O . The agent then chooses an action at according to its policy , π : O→∆ ( A ) , effecting the environment and producing a reward signal rt∈R . An experience is a ( st , at , rt+1 , st+1 ) -tuple , and a sequence of experiences ending in a terminal state is an episode τ . How the environment changes as a result of the action is dictated by the environment ’ s transition dynamics p : S×A→S . The agent is said to act optimally when it maximizes returnGt= ∑∞ l γ lrt+l ( γ is a discount factor ) . In value-based RL , agents use return as an estimate of the quality of a state V ( ot ) =Eπ [ ∑∞ l=0γ lr ( ot+l , at+l ) ] , and/or taking an action in a stateQ ( ot , at ) =r ( ot , at ) +γEot+1∈O [ V ( ot+1 ) ] . A normal form games ( NFG ) Γ = ( Π , U , n ) describes a one-shot strategic interaction among n players . When there is more than one agent , we denote agent-specific components with subscripts ( e.g. , πi ) . Negated subscripts represent the joint elements of all other agents ( e.g. , π−i ) . Each of the players has a set of policies Πi= { π0i , ... , π k i } from which it may choose . Player imay select a pure strategy πi∈Πi , or may randomize play by sampling from a mixed strategy σi∈∆ ( Πi ) . At the end of the interaction , each player receives a payoff U : Π→Rn . An empirical NFG ( ENFG ) Γ̃= ( Π̃ , Ũ , n ) is an NFG induced by simulation of strategy profiles . An ENFG approximates an underlying game Γ , estimating the payoffs through simulation . ENFGs are often employed when the strategy set is too large for exhaustive representation , for example when the strategies are instances from a complex space of policies . The quality of a strategy profile in an ENFG can be evaluated using regret : how much players could gain by deviating from assigned policies . Regret is measured with respect to a set of available deviation policies . The regret of solution σ to player i when they are able to deviate to policies in Πi ⊆ Πi is : Regreti ( σ , Πi ) =maxπi∈ΠiUi ( πi , σ−i ) −Ui ( σi , σ−i ) . Example deviation sets Π employed in this paper are ΠPSRO , the strategies accumulated through a run of PSRO , and ΠEVAL , denoting a static set of held-out evaluation policies . The sum of regrets across all players SumRegret ( σ , Π ) = ∑ i∈nRegreti ( σ , Πi ) , sometimes called the Nash convergence , is a measure of how stable a solution is . 3 METHOD . At each epoch e of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile σ∗ , e−1−i from the currently constructed ENFG : π e i ∈BR ( σ ∗ , e−1 −i ) . These policies are then added to each player ’ s strategy set , Π̃ei ← Π̃ e−1 i ∪ { πei } , and the new profiles are simulated to expand the ENFG . Algorithm 1 presents the full PSRO algorithm as defined by Lanctot et al . ( 2017 ) . One of the key design choices in iterative empirical game-solving is choosing which policies to add to the ENFG . This was first studied by Schvartzman & Wellman ( 2009a ) , who termed it the strategy exploration problem . We want to add policies that both bring the solution to the ENFG closer to the solution of the full game and that can be calculated efficiently . In PSRO , the strategy exploration problem is decomposed into two steps : solution and BR via RL . In the solution step , PSRO derives a profile σ∗ , e from the current empirical game . The method for choosing this profile is termed the meta-strategy-solver ( MSS ) , formally a function from empirical games to solution profiles MSS : Γ̃e→σ∗ , e . For example , the MSS might compute a Nash equilibrium of Γ̃e.2 In the BR step , PSRO generates a new policy via RL ( the response oracle ) , training against the target opponent profile computed by the MSS . The choice of MSS and response oracle algorithm constitutes a strategy exploration approach and determines the convergence speed of PSRO . We observe and address two main problems with the standard version of PSRO ( Algorithm 1 ) . First , note that the only thing that changes from one epoch to the next is the other-agent strategy profile σ−i ; the transition dynamics remain the same , and the mixed profile σ−i itself likely contains other-agent policies encountered in previous epochs . This suggests a compelling opportunity to transfer learning across epochs ; however , the BR calculation step works by training anew . Furthermore , because it is responding to some of the same strategies , the new policy learnt may be similar to the strategies added in previous epochs , and so not the most useful addition to the ENFG . Secondly , the opponent profiles are mixtures ( i.e. , distribution over opponent policies ) . This makes the environment dynamics more stochastic from the perspective of the RL agent , making learning more difficult . In Section 3.1 we propose an algorithm that transfers knowledge between iterations , and only trains against the single new opponent . In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations , while also addressing the opponent variance issue and providing scalability to multiple other agents . 3.1 MIXED-ORACLES . The first problem we address is that during BR calculation there is a missed opportunity for transferring previously learned information . In each epoch , each player learns a BR to a mixed profile of opponent policies . This mixture typically involves the newly added strategies ( one per player ) for this epoch , but may also include strategies from previous epochs . Training in previous epochs already captured experience against those strategies , so including them in further training entails some redundancy . The Mixed-Oracles algorithm is a variant of PSRO for two-player games , with a modified BR oracle designed to transfer learning across epochs . This method works by learning and maintaining a collection of BRs to each opponent policy Λei = { λ1i , ... , λ e i } , where λei is the BR to π e−1 −i . During each epoch of MixedOracles , a BR is learned for the single new opponent policy , rather than for the mixed opponent-profile generated by the MSS . A BR to the MSS-generated target mixture is then constructed from the collection of BR results for constituent policies in the mixture . Constructing the new policy is done through a general Combine-Responses function that maps a set of policies and a distribution over the policies into a single policy . The resulting policy should approximately aggregate the behavior of the policies . By reusing learned behaviors from previous epochs , Mixed-Oracles allows us to focus training exclusively on new opponent policies . The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies . We provide a general description of Mixed-Oracles , where the Combine-Responses method is abstract , as Algorithm 2 . Smith et al . ( 2020 ) propose Q-Mixing as an approach for constructing policies against any mixture of opponent strategies . The method employs Q-values learned against each individual opponent strategy , and thus can support the transfer sought here . Specifically , Q-Mixing averages the Q-values learned against each opponent policy π−i , weighted by their likelihood in the opponent mixture σ−i : Qi ( oi , ai |σ−i ) = ∑ π−i ψi ( π−i |oi , σ−i ) Qi ( oi , ai |π−i ) , ( 1 ) where ψ determines the relative likelihood of playing an opponent ψi : Oi→∆ ( Π−i ) . In this study , we use Q-Mixing as our Combine-Responses , where ψ is the prior over the opponent distribution as given by an MSS .

Summary: The paper focuses on resolving the computational and sample efficiency challenges with current PSRO style approaches. To this end it proposes two different modifications to the standard PSRO setup: 1) Mixed Oracles, and 2) Mixed Opponents. These approaches allow avoiding resetting learning after each outer loop epoch and reduce the stochasticity of dynamics during training. Thee efficacy is demonstrated on relatively simple games but using Deep RL policies where the proposed approaches are at least on par with standard PSRO approach in terms of final performance while drastically improving the sample efficiency.