Efficient Competitive Self-Play Policy Optimization

1 INTRODUCTION . Reinforcement learning ( RL ) from self-play has drawn tremendous attention over the past few years . Empirical successes have been observed in several challenging tasks , including Go ( Silver et al. , 2016 ; 2017 ; 2018 ) , simulated hide-and-seek ( Baker et al. , 2020 ) , simulated sumo wrestling ( Bansal et al. , 2017 ) , Capture the Flag ( Jaderberg et al. , 2019 ) , Dota 2 ( Berner et al. , 2019 ) , StarCraft II ( Vinyals et al. , 2019 ) , and poker ( Brown & Sandholm , 2019 ) , to name a few . During RL from selfplay , the learner collects training data by competing with an opponent selected from its past self or an agent population . Self-play presumably creates an auto-curriculum for the agents to learn at their own pace . At each iteration , the learner always faces an opponent that is comparably in strength to itself , allowing continuous improvement . The way the opponents are selected often follows human-designed heuristic rules in prior work . For example , AlphaGo ( Silver et al. , 2016 ) always competes with the latest agent , while the later generation AlphaGo Zero ( Silver et al. , 2017 ) and AlphaZero ( Silver et al. , 2018 ) generate self-play data with the maintained best historical agent . In specific tasks , such as OpenAI ’ s sumo wrestling , competing against a randomly chosen historical agent leads to the emergence of more diverse behaviors ( Bansal et al. , 2017 ) and more stable training than against the latest agent ( Al-Shedivat et al. , 2018 ) . In population-based training ( Jaderberg et al. , 2019 ; Liu et al. , 2019 ) and AlphaStar ( Vinyals et al. , 2019 ) , an elite or random agent is picked from the agent population as the opponent . Unfortunately , these rules may be inefficient and sometimes ineffective in practice since they do not necessarily enjoy last-iterate convergence to the “ average-case optimal ” solution even in tabular matrix games . In fact , in the simple Matching Pennies game , self-play with the latest agent fails to converge and falls into an oscillating behavior , as shown in Sec . 5 . In this paper , we develop an algorithm that adopts a principle-derived opponent-selection rule to alleviate some of the issues mentioned above . This requires clarifying first what the solution of self-play RL should be . From the game-theoretical perspective , Nash equilibrium is a fundamental solution concept that characterizes the desired “ average-case optimal ” strategies ( policies ) . When each player assumes other players also play their equilibrium strategies , no one in the game can gain more by unilaterally deviating to another strategy . Nash , in his seminal work ( Nash , 1951 ) , has established the existence result of mixed-strategy Nash equilibrium of any finite game . Thus solving for a mixed-strategy Nash equilibrium is a reasonable goal of self-play RL . We consider the particular case of two-player zero-sum games as the model for the competitive selfplay RL environments . In this case , the Nash equilibrium is the same as the ( global ) saddle point and as the solution of the minimax program minx∈X maxy∈Y f ( x , y ) . We denote x , y as the strategy profiles ( in RL terminology , policies ) and f as the loss for x or utility/reward for y . A saddle point ( x∗ , y∗ ) ∈ X × Y , where X , Y are the sets of all possible mixed-strategies ( stochastic policies ) of the two players , satisfies the following key property f ( x∗ , y ) ≤ f ( x∗ , y∗ ) ≤ f ( x , y∗ ) , ∀x ∈ X , ∀y ∈ Y . ( 1 ) Connections to the saddle problem and game theory inspire us to borrow ideas from the abundant literature for finding saddle points in the optimization field ( Arrow et al. , 1958 ; Korpelevich , 1976 ; Kallio & Ruszczynski , 1994 ; Nedić & Ozdaglar , 2009 ) and for finding equilibrium in the game theory field ( Zinkevich et al. , 2008 ; Brown , 1951 ; Singh et al. , 2000 ) . One particular class of method , i.e. , the perturbation-based subgradient methods to find the saddles ( Korpelevich , 1976 ; Kallio & Ruszczynski , 1994 ) , is especially appealing . This class of method directly builds upon the inequality properties in Eq . 1 , and has several advantages : ( 1 ) Unlike some algorithms that require knowledge of the game dynamics ( Silver et al. , 2016 ; 2017 ; Nowé et al. , 2012 ) , it requires only subgradients ; thus , it is easy to be adapted to policy optimization with estimated policy gradients . ( 2 ) For convexconcave functions , it is guaranteed to converge in its last iterate instead of an average iterate , hence alleviates the need to compute any historical averages as in Brown ( 1951 ) ; Singh et al . ( 2000 ) ; Zinkevich et al . ( 2008 ) , which can get complicated when neural nets are involved ( Heinrich & Silver , 2016 ) . ( 3 ) Most importantly , it prescribes a simple principled way to adversarially choose self-play opponents , which can be naturally instantiated with a concurrently-trained agent population . To summarize , we apply ideas from the perturbation-based methods of classical saddle point optimization to the model-free self-play RL regime . This results in a novel population-based policy gradient method with a principled adversarial opponent-selection rule . Analogous to the standard model-free RL setting , we assume only “ naive ” players ( Jafari et al. , 2001 ) where the game dynamic is hidden and only rewards for their own actions are revealed . This enables broader applicability to problems with mismatched or unknown game dynamics than many existing algorithms ( Silver et al. , 2016 ; 2017 ; Nowé et al. , 2012 ) . In Sec . 4 , we provide an approximate convergence theorem for convex-concave games as a sanity check . Sec . 5 shows extensive experiment results favoring our algorithm ’ s effectiveness in several games , including matrix games , grid-world soccer , a board game , and a challenging simulated robot sumo game . Our method demonstrates higher per-agent sample efficiency than baseline methods with alternative opponent-selection rules . Our trained agents also outperform the baseline agents on average in competitions . 2 RELATED WORK . Reinforcement learning trains a single agent to maximize the expected return in an environment ( Sutton & Barto , 2018 ) . Multiagent reinforcement learning ( MARL ) , of which two-agent is a special case , concerns multiple agents taking actions in the same environment ( Littman , 1994 ) . Self-play is a training paradigm to generate data for MARL and has led to great successes , achieving superhuman performance in several domains ( Tesauro , 1995 ; Silver et al. , 2016 ; Brown & Sandholm , 2019 ) . Applying RL algorithms naively as independent learners in MARL sometimes produces strong agents ( Tesauro , 1995 ) but not always . People have studied ways to extend RL algorithms specifically to MARL , e.g. , minimax-Q ( Littman , 1994 ) , Nash-Q ( Hu & Wellman , 2003 ) , WoLF-PG ( Bowling & Veloso , 2002 ) , etc . However , most of these methods are designed for tabular RL only , therefore not readily applicable to continuous state action spaces or complex policy functions where gradient-based policy optimization methods are preferred . Recently , Bai & Jin ( 2020 ) , Lee et al . ( 2020 ) and Zhang et al . ( 2020 ) provide theoretical regret or convergence analyses under tabular or other restricted self-play settings , which complement our empirical effort . There are algorithms developed from the game theory and online learning perspective ( Lanctot et al. , 2017 ; Nowé et al. , 2012 ; Cardoso et al. , 2019 ) , notably Tree search , Fictitious self-play ( Brown , 1951 ) , Regret minimization ( Jafari et al. , 2001 ; Zinkevich et al. , 2008 ) , and Mirror descent ( Mertikopoulos et al. , 2019 ; Rakhlin & Sridharan , 2013 ) . Tree search such as minimax and alpha-beta pruning is particularly effective in small-state games . Monte Carlo Tree Search ( MCTS ) is also effective in Go ( Silver et al. , 2016 ) . However , Tree search requires learners to know ( or at least learn ) the game dynamics . The other ones typically require maintaining some historical quantities . In Fictitious play , the learner best-responds to a historical average opponent , and the average strategy converges . Similarly , the total historical regrets in all ( information ) states are maintained in ( counterfactual ) regret minimization ( Zinkevich et al. , 2008 ) . Furthermore , most of those algorithms are designed only for discrete state action games . Special care has to be taken with neural net function approximators ( Heinrich & Silver , 2016 ) . On the contrary , our method does not require the complicated computation of averaging neural nets , and is readily applicable to continuous environments . In two-player zero-sum games , the Nash equilibrium coincides with the saddle point . This enables the techniques developed for finding saddle points . While some saddle-point methods also rely on time averages ( Nedić & Ozdaglar , 2009 ) , a class of perturbation-based gradient method is known to converge under mild convex-concave assumption for deterministic functions ( Kallio & Ruszczynski , 1994 ; Korpelevich , 1976 ; Hamm & Noh , 2018 ) . We develop a sampling version of them for stochastic RL objectives , which leads to a more principled and effective way of choosing opponents in self-play . Our adversarial opponent-selection rule bears a resemblance to Gleave et al . ( 2019 ) . However , our goal is to develop an effective self-play RL algorithm , while Gleave et al . ( 2019 ) aims at attacking deep self-play learned policies . A recent work by Prajapat et al . ( 2020 ) tackles the self-play policy optimization problem differently from ours by employing a bilinear approximation to the game . Finally , although the algorithm presented here builds upon policy gradient , the same framework may be extended to other RL algorithms such as MCTS thanks to a recent interpretation of MCTS as policy optimization ( Grill et al. , 2020 ) . Our way of leveraging Eq . 1 in a population may potentially work beyond gradient-based RL , e.g. , in training generative adversarial networks similarly to Hamm & Noh ( 2018 ) due to the same minimax formulation . 3 METHOD . Classical game theory defines a two-player zero-sum game as a tuple ( X , Y , f ) where X , Y are the sets of possible strategies of Players 1 and 2 respectively , and f : X × Y 7→ R is a mapping from a pair of strategies to a real-valued utility/reward for Player 2 . The game is zero-sum ( fully competitive ) , so Player 1 ’ s reward is −f . This is a special case of the Stochastic Game formulation for Multiagent RL ( Shapley , 1953 ) which itself is an extension to Markov Decision Processes ( MDP ) . We consider mixed strategies induced by stochastic policies πx and πy . The policies can be parameterized functions in which case X , Y are the sets of all possible policy parameters . Denote at as the action of Player 1 and bt as the action of Player 2 at time t , let T be the time limit of the game , then the stochastic payoff f writes as f ( x , y ) = E at∼πx , bt∼πy , st+1∼P ( ·|st , at , bt ) T∑ t=0 γtr ( st , at , bt ) . ( 2 ) The state sequence { st } Tt=0 follows a transition dynamic P ( st+1|st , at , bt ) . Actions are sampled according to action distributions πx ( ·|st ) and πy ( ·|st ) . And r ( st , at , bt ) is the reward ( payoff ) for Player 2 at time t , determined jointly by the state and actions . We use the term ‘ agent ’ and ‘ player ’ interchangeably . While we consider an agent pair ( x , y ) in this paper , in some cases ( Silver et al. , 2016 ) , x = y can be enforced by sharing parameters if the game is impartial . The discounting factor γ weights between short- and long-term rewards and is optional . Note that when one agent is fixed , taking y as an example , the problem x is facing reduces to an MDP if we define a new state transition dynamic Pnew ( st+1|st , at ) = ∑ bt P ( st+1|st , at , bt ) πy ( bt|st ) and a new reward rnew ( st , at ) = ∑ bt r ( st , at , bt ) πy ( bt|st ) . This leads to the naive gradient descentascent algorithm , which provably works in strictly convex-concave games ( where f is strictly convex in x and strictly concave in y ) under some assumptions ( Arrow et al. , 1958 ) . However , in general , it does not enjoy last-iterate convergence to the Nash equilibrium . Even for simple games such as Matching Pennies and Rock Paper Scissors , as we shall see in our experiments , the naive algorithm generates cyclic sequences of xk , yk that orbit around the equilibrium . This motivates us to study the perturbation-based method which converges under weaker assumptions . Algorithm 1 : Perturbation-based self-play policy optimization of an n agent population . Input : N : No iterations ; ηk : learning rates ; mk : sample size ; n : population size ; l : No inner updates ; Result : n pairs of policies ; 1 Initialize ( x0i , y 0 i ) , i = 1 , 2 , . . . n ; 2 for k = 0 , 1 , 2 , . . . N − 1 do 3 Evaluate f̂ ( xki , y k j ) , ∀i , j ∈ 1 . . . n with Eq . 4 and sample size mk ; 4 for i = 1 , . . . n do 5 Construct candidate opponent sets Ckyi = { y k j : j = 1 . . . n } and Ckxi = { x k j : j = 1 . . . n } ; 6 Find perturbed vki = argmaxy∈Ckyi f̂ ( xki , y ) , perturbed u k i = argminx∈Ckxi f̂ ( x , yki ) ; 7 Invoke a single-agent RL algorithm ( e.g. , A2C , PPO ) on xki for l times that : 8 Estimate policy gradients ĝkxi = ∇̂xf ( x k i , v k i ) with sample size mk ( e.g. , Eq . 5 ) ; 9 Update policy by xk+1i ← x k i − ηkĝkxi ( or RmsProp ) ; 10 Invoke a single-agent RL algorithm ( e.g. , A2C , PPO ) on yki for l times that : 11 Estimate policy gradients ĝkyi = ∇̂yf ( u k i , y k i ) with sample size mk ; 12 Update policy by yk+1i ← yi k + ηkĝ k yi ( or RmsProp ) ; 13 return { ( xNi , yNi ) } ni=1 ; Recall that the Nash equilibrium has to satisfy the saddle constraints Eq . 1 : f ( x∗ , y ) ≤ f ( x∗ , y∗ ) ≤ f ( x , y∗ ) . The perturbation-based methods build upon this property ( Nedić & Ozdaglar , 2009 ; Kallio & Ruszczynski , 1994 ; Korpelevich , 1976 ) and directly optimize for a solution that meets the constraints . They find perturbed points u of Player 1 and v of Player 2 , and use gradients at ( x , v ) and ( u , y ) to optimize x and y respectively . Under some regularity assumptions , gradient direction from a single perturbed point is adequate for proving convergence for ( not strictly ) convex-concave functions ( Nedić & Ozdaglar , 2009 ) . They can be easily extended to accommodate gradient based policy optimization and the stochastic RL objective in Eq . 4 . We propose to find the perturbations from an agent population , resulting in the algorithm outlined in Alg . 1 . The algorithm trains n pairs of agents simultaneously . At each rounds of training , we first run n2 pairwise competitions as the evaluation step ( Alg . 1 L3 ) , costing n2mk trajectories . To save sample complexity , we can use these rollouts to do one policy update as well . Then a simple adversarial rule ( Eq . 3 ) is adopted in Alg . 1 L6 to choose the opponents adaptively . The intuition is that vki and u k i are the most challenging opponents in the population for the current xi and yi . vki = arg max y∈Ckyi f̂ ( xki , y ) , u k i = arg min x∈Ckxi f̂ ( x , yki ) . ( 3 ) The perturbations vki and u k i always satisfy f ( x k i , v k i ) ≥ f ( uki , yki ) , since maxy∈Ckyi f̂ ( x k i , y ) ≥ f̂ ( xki , y k i ) ≥ minx∈Ckxi f̂ ( x , y k i ) . Then we run gradient descent on x k i with the perturbed v k i as opponent to minimize f ( xki , v k i ) , and run gradient ascent on y k i to maximize f ( u k i , y k i ) . Intuitively , the duality gap between minx maxy f ( x , y ) and maxy minx f ( x , y ) , approximated by f ( xki , v k i ) − f ( uki , y k i ) , is reduced , leading ( x k i , y k i ) to converge to the saddle point ( equilibrium ) . We build the candidate opponent sets in L5 of Alg . 1 simply as the concurrently-trained n-agent population . Specifically , Ckyi = { yk1 , . . . , y k n } and Ckxi = { xk1 , . . . , x k n } . This is due to the following considerations . An alternative source of candidates is the fixed known agents such as a rule-based agent , which may not be available in practice . Another source is the extragradient methods ( Korpelevich , 1976 ; Mertikopoulos et al. , 2019 ) , where extra gradient steps are taken on y before optimizing x . The extragradient method can be thought of as a local approximation to Eq . 3 with a neighborhood opponent set , thus is related to our method . However , this method could be less efficient because the trajectory sample used in the extragradient steps is wasted as it does not contribute to actually optimizing y . Yet another source is the past agents . This choice is motivated by Fictitious play and ensures that the current learner always defeats a past self . However , as we shall see in the experiments , self-play with a random past agent may learn slower than our method . We expect all agents in the population in our algorithm to be strong , thus provide stronger learning signals . Finally , we use Monte Carlo estimation to compute the values and gradients of f . In the classical game theory setting where the game dynamic and payoff are known , it is possible to compute the exact values and gradients of f . But in the model-free MARL setting , we have to collect roll-out trajectories to estimate both the function values through policy evaluation and gradients through the Policy gradient theorem ( Sutton & Barto , 2018 ) . After collecting m independent trajectories { { ( sit , ait , rit ) } Tt=0 } m i=1 , we can estimate f ( x , y ) by f̂ ( x , y ) = 1 m m∑ i=1 T∑ t=0 γtrit . ( 4 ) And given estimates Q̂x ( s , a ; y ) to the state-action value Qx ( s , a ; y ) ( assuming an MDP with y as a fixed opponent of x ) , we construct an estimator for∇xf ( x , y ) ( and similarly for∇yf given Q̂y ) by ∇̂xf ( x , y ) ∝ 1 m m∑ i=1 T∑ t=0 ∇x log πx ( ait|sit ) Q̂x ( sit , ait ; y ) . ( 5 )

Summary: The paper “Efficient Competitive Self-Play Policy Optimization” introduces a new self-play scheme for solving zero-sum two-player games. It is suggested to train a population of N agents in parallel, where each agent is matched against the comparatively strongest opponent in the next round of training. As baselines, the paper considers self-play against the best, the latest and random snapshots from the training history of only a single agent.