Iterative Empirical Game Solving via Single Policy Best Response

1 INTRODUCTION . In ( single-agent ) reinforcement learning ( RL ) , an agent repeatedly interacts with an environment until it achieves mastery , that is , can find no further way to improve performance ( measured by reward ) . In a multiagent system , the outcome of any learned policy may depend pivotally on the behavior of other agents . Direct search over a complex joint policy space for game solutions is daunting , and so more common in approaches combining RL and game theory is to interleave learning and game analysis iteratively . In empirical game-theoretic analysis ( EGTA ) , game reasoning is applied to approximate game models defined over restricted strategy sets . The strategies are selected instances drawn from a large space of possible strategies , for example , defined in terms of a fundamental policy space : mappings from observations to actions . For a given set of strategies ( policies ) ,1 we estimate an empirical game via simulation over combinations of these strategies . In iterative approaches to EGTA , analysis of the empirical game informs decisions about the sampling of strategy profiles or additions of strategies to the restricted sets . Schvartzman & Wellman ( 2009b ) first combined RL with EGTA , adding new strategies by learning a policy for one agent ( using tabular Q-learning ) , fixing other agent policies at a Nash equilibrium of the current empirical game . Lanctot et al . ( 2017 ) introduced a general framework , Policy-Space Response Oracles ( PSRO ) , interleaving empirical game analysis with deep RL . PSRO generalizes the identification of learning targets , employing an abstract method termed meta-strategy solver ( MSS ) that extracts a strategy profile from an empirical game . At each epoch of PSRO , a player uses RL to derive a new policy by training against the opponentstrategies in the MSS-derived profile . A particular challenge for the RL step in PSRO is that the learner must derive a response under uncertainty about opponent policies . The profile returned by the MSS is generally a mixed-strategy profile , as in strategically complex environments randomization is often a necessary ingredient for equilibrium . The opponent draws from this mixture are unobserved , adding uncertainty to the multiagent environment . 1The term policy as employed in the RL context corresponds exactly to the game-theoretic notion of strategy in our setting , and we use the terms interchangeably . We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies . We propose and evaluate two such methods , which work in qualitatively different ways : Mixed-Oracles learns separate best-responses ( BR ) to each pure strategy in a mixture and combines the results from learning to approximate a BR to the mixture . Mixed-Opponents constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy . Both of our methods employ the machinery of Q-Mixing ( Smith et al. , 2020 ) , which constructs policies based on an aggregation of Q-functions corresponding to component policies of a mixture . Our methods promise advantages beyond those of learning in a less stochastic environment . Mixed-Oracles transfers learning across epochs , exploiting the Q-functions learned against a particular opponent policy in constructing policies for any other epoch where that opponent policy is encountered . Mixed-Opponents applies directly over the joint opponent space , and so has the potential to scale beyond two-player games . We evaluate our methods in a series of experimental games , and find that both Mixed-Oracles and Mixed-Opponents are able to find solutions at least as good as an unmodified PSRO , and often better , while providing a substantial reduction in the required amount of training simulation . 2 PRELIMINARIES . An RL agent at time t ∈ T receives the state of the environment st ∈ S , or a partial state called an observation ot∈O . The agent then chooses an action at according to its policy , π : O→∆ ( A ) , effecting the environment and producing a reward signal rt∈R . An experience is a ( st , at , rt+1 , st+1 ) -tuple , and a sequence of experiences ending in a terminal state is an episode τ . How the environment changes as a result of the action is dictated by the environment ’ s transition dynamics p : S×A→S . The agent is said to act optimally when it maximizes returnGt= ∑∞ l γ lrt+l ( γ is a discount factor ) . In value-based RL , agents use return as an estimate of the quality of a state V ( ot ) =Eπ [ ∑∞ l=0γ lr ( ot+l , at+l ) ] , and/or taking an action in a stateQ ( ot , at ) =r ( ot , at ) +γEot+1∈O [ V ( ot+1 ) ] . A normal form games ( NFG ) Γ = ( Π , U , n ) describes a one-shot strategic interaction among n players . When there is more than one agent , we denote agent-specific components with subscripts ( e.g. , πi ) . Negated subscripts represent the joint elements of all other agents ( e.g. , π−i ) . Each of the players has a set of policies Πi= { π0i , ... , π k i } from which it may choose . Player imay select a pure strategy πi∈Πi , or may randomize play by sampling from a mixed strategy σi∈∆ ( Πi ) . At the end of the interaction , each player receives a payoff U : Π→Rn . An empirical NFG ( ENFG ) Γ̃= ( Π̃ , Ũ , n ) is an NFG induced by simulation of strategy profiles . An ENFG approximates an underlying game Γ , estimating the payoffs through simulation . ENFGs are often employed when the strategy set is too large for exhaustive representation , for example when the strategies are instances from a complex space of policies . The quality of a strategy profile in an ENFG can be evaluated using regret : how much players could gain by deviating from assigned policies . Regret is measured with respect to a set of available deviation policies . The regret of solution σ to player i when they are able to deviate to policies in Πi ⊆ Πi is : Regreti ( σ , Πi ) =maxπi∈ΠiUi ( πi , σ−i ) −Ui ( σi , σ−i ) . Example deviation sets Π employed in this paper are ΠPSRO , the strategies accumulated through a run of PSRO , and ΠEVAL , denoting a static set of held-out evaluation policies . The sum of regrets across all players SumRegret ( σ , Π ) = ∑ i∈nRegreti ( σ , Πi ) , sometimes called the Nash convergence , is a measure of how stable a solution is . 3 METHOD . At each epoch e of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile σ∗ , e−1−i from the currently constructed ENFG : π e i ∈BR ( σ ∗ , e−1 −i ) . These policies are then added to each player ’ s strategy set , Π̃ei ← Π̃ e−1 i ∪ { πei } , and the new profiles are simulated to expand the ENFG . Algorithm 1 presents the full PSRO algorithm as defined by Lanctot et al . ( 2017 ) . One of the key design choices in iterative empirical game-solving is choosing which policies to add to the ENFG . This was first studied by Schvartzman & Wellman ( 2009a ) , who termed it the strategy exploration problem . We want to add policies that both bring the solution to the ENFG closer to the solution of the full game and that can be calculated efficiently . In PSRO , the strategy exploration problem is decomposed into two steps : solution and BR via RL . In the solution step , PSRO derives a profile σ∗ , e from the current empirical game . The method for choosing this profile is termed the meta-strategy-solver ( MSS ) , formally a function from empirical games to solution profiles MSS : Γ̃e→σ∗ , e . For example , the MSS might compute a Nash equilibrium of Γ̃e.2 In the BR step , PSRO generates a new policy via RL ( the response oracle ) , training against the target opponent profile computed by the MSS . The choice of MSS and response oracle algorithm constitutes a strategy exploration approach and determines the convergence speed of PSRO . We observe and address two main problems with the standard version of PSRO ( Algorithm 1 ) . First , note that the only thing that changes from one epoch to the next is the other-agent strategy profile σ−i ; the transition dynamics remain the same , and the mixed profile σ−i itself likely contains other-agent policies encountered in previous epochs . This suggests a compelling opportunity to transfer learning across epochs ; however , the BR calculation step works by training anew . Furthermore , because it is responding to some of the same strategies , the new policy learnt may be similar to the strategies added in previous epochs , and so not the most useful addition to the ENFG . Secondly , the opponent profiles are mixtures ( i.e. , distribution over opponent policies ) . This makes the environment dynamics more stochastic from the perspective of the RL agent , making learning more difficult . In Section 3.1 we propose an algorithm that transfers knowledge between iterations , and only trains against the single new opponent . In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations , while also addressing the opponent variance issue and providing scalability to multiple other agents . 3.1 MIXED-ORACLES . The first problem we address is that during BR calculation there is a missed opportunity for transferring previously learned information . In each epoch , each player learns a BR to a mixed profile of opponent policies . This mixture typically involves the newly added strategies ( one per player ) for this epoch , but may also include strategies from previous epochs . Training in previous epochs already captured experience against those strategies , so including them in further training entails some redundancy . The Mixed-Oracles algorithm is a variant of PSRO for two-player games , with a modified BR oracle designed to transfer learning across epochs . This method works by learning and maintaining a collection of BRs to each opponent policy Λei = { λ1i , ... , λ e i } , where λei is the BR to π e−1 −i . During each epoch of MixedOracles , a BR is learned for the single new opponent policy , rather than for the mixed opponent-profile generated by the MSS . A BR to the MSS-generated target mixture is then constructed from the collection of BR results for constituent policies in the mixture . Constructing the new policy is done through a general Combine-Responses function that maps a set of policies and a distribution over the policies into a single policy . The resulting policy should approximately aggregate the behavior of the policies . By reusing learned behaviors from previous epochs , Mixed-Oracles allows us to focus training exclusively on new opponent policies . The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies . We provide a general description of Mixed-Oracles , where the Combine-Responses method is abstract , as Algorithm 2 . Smith et al . ( 2020 ) propose Q-Mixing as an approach for constructing policies against any mixture of opponent strategies . The method employs Q-values learned against each individual opponent strategy , and thus can support the transfer sought here . Specifically , Q-Mixing averages the Q-values learned against each opponent policy π−i , weighted by their likelihood in the opponent mixture σ−i : Qi ( oi , ai |σ−i ) = ∑ π−i ψi ( π−i |oi , σ−i ) Qi ( oi , ai |π−i ) , ( 1 ) where ψ determines the relative likelihood of playing an opponent ψi : Oi→∆ ( Π−i ) . In this study , we use Q-Mixing as our Combine-Responses , where ψ is the prior over the opponent distribution as given by an MSS .

Summary: The paper proposes two new methods in the Policy-Space Response Oracle framework. These approaches permit to reuse past knowledge in order to reduce the amount of data required for the RL training. The first algorithm Mixed-Oracles transfers the previous iteration of Deep RL, instead of the second one, Mixed-Opponents, transfers existing strategy action-value estimates.