Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization

We propose a simple , general and effective technique , Reward Randomization for discovering diverse strategic policies in complex multi-agent games . Combining reward randomization and policy gradient , we derive a new algorithm , RewardRandomized Policy Gradient ( RPG ) . RPG is able to discover multiple distinctive human-interpretable strategies in challenging temporal trust dilemmas , including grid-world games and a real-world game Agar.io , where multiple equilibria exist but standard multi-agent policy gradient algorithms always converge to a fixed one with a sub-optimal payoff for every player even using state-of-the-art exploration techniques . Furthermore , with the set of diverse strategies from RPG , we can ( 1 ) achieve higher payoffs by fine-tuning the best policy from the set ; and ( 2 ) obtain an adaptive agent by using this set of strategies as its training opponents . The source code and example videos can be found in our website : https : //sites.google . com/view/staghuntrpg . 1 INTRODUCTION . Games have been a long-standing benchmark for artificial intelligence , which prompts persistent technical advances towards our ultimate goal of building intelligent agents like humans , from Shannon ’ s initial interest in Chess ( Shannon , 1950 ) and IBM DeepBlue ( Campbell et al. , 2002 ) , to the most recent deep reinforcement learning breakthroughs in Go ( Silver et al. , 2017 ) , Dota II ( OpenAI et al. , 2019 ) and Starcraft ( Vinyals et al. , 2019 ) . Hence , analyzing and understanding the challenges in various games also become critical for developing new learning algorithms for even harder challenges . Most recent successes in games are based on decentralized multi-agent learning ( Brown , 1951 ; Singh et al. , 2000 ; Lowe et al. , 2017 ; Silver et al. , 2018 ) , where agents compete against each other and optimize their own rewards to gradually improve their strategies . In this framework , Nash Equilibrium ( NE ) ( Nash , 1951 ) , where no player could benefit from altering its strategy unilaterally , provides a general solution concept and serves as a goal for policy learning and has attracted increasingly significant interests from AI researchers ( Heinrich & Silver , 2016 ; Lanctot et al. , 2017 ; Foerster et al. , 2018 ; Kamra et al. , 2019 ; Han & Hu , 2019 ; Bai & Jin , 2020 ; Perolat et al. , 2020 ) : many existing works studied how to design practical multi-agent reinforcement learning ( MARL ) algorithms that can provably converge to an NE in Markov games , particularly in the zero-sum setting . Despite the empirical success of these algorithms , a fundamental question remains largely unstudied in the field : even if an MARL algorithm converges to an NE , which equilibrium will it converge to ? The existence of multiple NEs is extremely common in many multi-agent games . Discovering as many NE strategies as possible is particularly important in practice not only because different NEs can produce drastically different payoffs but also because when facing unknown players who are trained to play an NE strategy , we can gain advantage by identifying which NE strategy the opponent is playing and choosing the most appropriate response . Unfortunately , in many games where multiple distinct NEs exist , the popular decentralized policy gradient algorithm ( PG ) , which has led to great successes in numerous games including Dota II and Stacraft , always converge to a particular NE with non-optimal payoffs and fail to explore more diverse modes in the strategy space . Consider an extremely simple example , a 2-by-2 matrix game Stag-Hunt ( Rousseau , 1984 ; Skyrms , 2004 ) , where two pure strategy NEs exist : a “ risky ” cooperative equilibrium with the highest payoff ∗Equal contribution . † Work done as an intern at Institute for Interdisciplinary Information Sciences ( IIIS ) , Tsinghua University . for both agents and a “ safe ” non-cooperative equilibrium with strictly lower payoffs . We show , from both theoretical and practical perspectives , that even in this simple matrix-form game , PG fails to discover the high-payoff “ risky ” NE with high probability . The intuition is that the neighborhood that makes policies converge to the “ risky ” NE can be substantially small comparing to the entire policy space . Therefore , an exponentially large number of exploration steps are needed to ensure PG discovers the desired mode . We propose a simple technique , Reward Randomization ( RR ) , which can help PG discover the “ risky ” cooperation strategy in the stag-hunt game with theoretical guarantees . The core idea of RR is to directly perturb the reward structure of the multi-agent game of interest , which is typically low-dimensional . RR directly alters the landscape of different strategy modes in the policy space and therefore makes it possible to easily discover novel behavior in the perturbed game ( Fig . 1 ) . We call this new PG variant Reward-Randomized Policy Gradient ( RPG ) . To further illustrate the effectiveness of RPG , we introduce three Markov games – two gridworld games and a real-world online game Agar.io . All these games have multiple NEs including both “ risky ” cooperation strategies and “ safe ” non-cooperative strategies . We empirically show that even with state-of-the-art exploration techniques , PG fails to discover the “ risky ” cooperation strategies . In contrast , RPG discovers a surprisingly diverse set of human-interpretable strategies in all these games , including some non-trivial emergent behavior . Importantly , among this set are policies achieving much higher payoffs for each player compared to those found by PG . This “ diversityseeking ” property of RPG also makes it feasible to build adaptive policies : by re-training an RL agent against the diverse opponents discovered by RPG , the agent is able to dynamically alter its strategy between different modes , e.g. , either cooperate or compete , w.r.t . its test-time opponent ’ s behavior . We summarize our contributions as follow • We studied a collection of challenging multi-agent games , where the popular multi-agent PG algorithm always converges to a sub-optimal equilibrium strategy with low payoffs . • A novel reward-space exploration technique , reward randomization ( RR ) , for discovering hard-to-find equilibrium with high payoffs . Both theoretical and empirical results show that reward randomization substantially outperforms classical policy/action-space exploration techniques in challenging trust dilemmas . • We empirically show that RR discovers surprisingly diverse strategic behaviors in complex Markov games , which further provides a practical solution for building an adaptive agent . • A new multi-agent environment Agar.io , which allows complex multi-agent strategic behavior . We released the environment to the community as a novel testbed for MARL research . 2 A MOTIVATING EXAMPLE : STAG HUNT Stag Hare Stag a , a c , b Hare b , c d , d Table 1 : The stag-hunt game , a > b ≥ d > c. We start by analyzing a simple problem : finding the NE with the optimal payoffs in the Stag Hunt game . This game was originally introduced in Rousseau ’ s work , “ A discourse on inequality ” ( Rousseau , 1984 ) : a group of hunters are tracking a big stag silently ; now a hare shows up , each hunter should decide whether to keep tracking the stag or kill the hare immediately . This leads to the 2-by-2 matrix-form stag-hunt game in Tab . 1 with two actions for each agent , Stag ( S ) and Hare ( H ) . There are two pure strategy NEs : the Stag NE , where both agents choose S and receive a high payoff a ( e.g. , a = 4 ) , and the Hare NE , where both agents choose H and receive a lower payoff d ( e.g. , d = 1 ) . The Stag NE is “ risky ” because if one agent defects , they still receives a decent reward b ( e.g. , b = 3 ) for eating the hare alone while the other agent with an S action may suffer from a big loss c for being hungry ( e.g. , c = −10 ) . Formally , let A = { S , H } denote the action space , πi ( θi ) denote the policy for agent i ( i ∈ { 1 , 2 } ) parameterized by θi , i.e. , P [ πi ( θi ) = S ] = θi and P [ πi ( θi ) = H ] = 1− θi , and R ( a1 , a2 ; i ) denote the payoff for agent i when agent 1 takes action a1 and agent 2 takes action a2 . Each agent i optimizes its expected utility Ui ( π1 , π2 ) = Ea1∼π1 , a2∼π2 [ R ( a1 , a2 ; i ) ] . Using the standard policy gradient algorithm , a typical learning procedure is to repeatedly take the following two steps until convergence1 : ( 1 ) estimate gradient ∇i = ∇Ui ( π1 , π2 ) via self-play ; ( 2 ) update the policies by θi ← θi + α∇i with learning rate α . Although PG is widely used in practice , the following theorem shows in certain scenarios , unfortunately , the probability that PG converges to the Stag NE is low . Theorem 1 . Suppose a− b = ( d− c ) for some 0 < < 1 and initialize θ1 , θ2 ∼ Unif [ 0 , 1 ] . Then the probability that PG discovers the high-payoff NE is upper bounded by 2 + 2 1+2 + 2 . Theorem 1 shows when the risk is high ( i.e. , c is low ) , then the probability of finding the Stag NE via PG is very low . Note this theorem applies to random initialization , which is standard in RL . Remark : One needs at least N = Ω ( 1 ) restarts to ensure a constant success probability . Fig . 2 shows empirical studies : we select 4 value assignments , i.e. , c ∈ { −5 , −20 , −50 , −100 } and a=4 , b=3 , d=1 , and run a state-of-the-art PG method , proximal policy optimization ( PPO ) ( Schulman et al. , 2017 ) , on these games . The Stag NE is rarely reached , and , as c becomes smaller , the probability of finding the Stag NE significantly decreases . Peysakhovich & Lerer ( 2018b ) provided a theorem of similar flavor without analyzing the dynamics of the learning algorithm whereas we explicitly characterize the behavior of PG . They studied a prosocial reward-sharing scheme , which transforms the reward of both agents toR ( a1 , a2 ; 1 ) +R ( a1 , a2 ; 2 ) . Reward sharing can be viewed as a special case of our method and , as shown in Sec . 5 , it is insufficient for solving complex temporal games . 2.1 REWARD RANDOMIZATION IN THE MATRIX-FORM STAG-HUNT GAME . 9 Thm . 1 suggests that the utility function R highly influences what strategy PG might learn . Taking one step further , even if a strategy is difficult to learn with a particular R , it might be easier in some other function R′ . Hence , if we can define an appropriate spaceR over different utility functions and draw samples from R , we may possibly discover desired novel strategies by running PG on some sampled utility function R′ and evaluating the obtained policy profile on the original game with R. We call this procedure Reward Randomization ( RR ) . Concretely , in the stag-hunt game , R is parameterized by 4 variables ( aR , bR , cR , dR ) . We can define a distribution over R4 , draw a tuple R′ = ( aR′ , bR′ , cR′ , dR′ ) from this distribution , and run PG on R′ . Denote the original stag-hunt game where the Stag NE is hard to discover as R0 . Reward randomization draws N perturbed tuples R1 , . . . , RN , runs PG on each Ri , and evaluates each of the obtained strategies on R0 . The theorem below shows it is highly likely that the population of the N policy profiles obtained from the perturbed games contains the Stag NE strategy . Theorem 2 . For any Stag-Hunt game , suppose in the i-th run of RR we randomly generate aRi , bRi , cRi , dRi ∼ Unif [ −1 , 1 ] and initialize θ1 , θ2 ∼ Unif [ 0 , 1 ] , then with probability at least 1− 0.6N = 1− exp ( −Ω ( N ) ) , the aforementioned RR procedure discovers the high-payoff NE . Here we use the uniform distribution as an example . Other distributions may also help in practice . Comparing Thm . 2 and Thm . 1 , RR significantly improves standard PG w.r.t . success probability . Remark 1 : For the scenario studied in Thm . 1 , to achieve a ( 1− δ ) success probability for some 0 < δ < 1 , PG requires at least N = Ω ( 1 log ( 1 δ ) ) random restarts . For the same scenario , RR only requires to repeat at most N = O ( log ( 1/δ ) ) which is independent of . When is small , this is a huge improvement . Remark 2 : Thm . 2 suggests that comparing with policy randomization , perturbing the payoff matrix makes it substantially easier to discover a strategy that can be hardly reached in the original game . Note that although in Stag Hunt , we particularly focus on the Stag NE that has the highest payoff for both agents , in general RR can also be applied to NE selection in other matrix-form games using a payoff evaluation functionE ( π1 , π2 ) . For example , we can setE ( π1 , π2 ) = U1 ( π1 , π2 ) +U2 ( π1 , π2 ) for a prosocial NE , or look for Pareto-optimal NEs by setting E ( π1 , π2 ) = βU1 ( π1 , π2 ) + ( 1 − β ) U2 ( π1 , π2 ) with 0 ≤ β ≤ 1 . 1In general matrix games beyond stag hunt , the procedure can be cyclic as well ( Singh et al. , 2000 ) . Algorithm 1 : RPG : Reward-Randomized Policy Gradient Input : original game M , search spaceR , evaluation function E , population size N ; draw samples { R ( 1 ) , . . . , R ( N ) } fromR ; { π ( i ) 1 , π ( i ) 2 } ← PG on induced games { M ( R ( i ) ) } i in parallel ; // RR phase select the best candidate π ( k ) 1 , π ( k ) 2 by k = arg maxiE ( π ( i ) 1 , π ( i ) 2 ) ; // evaluation phase π ? 1 , π ? 2 ← fine-tune π ( k ) 1 , π ( k ) 2 on M via PG ( if necessary ) ; // fine-tuning phase return π ? 1 , π ? 2 ;

Summary: This paper considers the problem of finding a nash equilibrium in two player games where each of the algorithm runs an RL algorithm. In this paper they ask the question -- which nash equilibria does the dynamics converge to in this two player game (where each player optimizes based on a policy gradient algorithm). They construct two player games with multiple nash equilibria; one is a favorable nash equilibria where both players get high rewards while the other is a less favorable nash equilibria where both player only get medium rewards. In such games they first show that in general simply running policy gradient on the natural reward function i.e., the observed payoff will not lead to the desirable nash equilibria. The goal of this paper is to ameliorate this by considering perturbations in the reward space. At a high level, the algorithm learns multiple policies on a class of games generated by sampling multiple reward functions from a family and training one policy per sampled reward function using PG. Then using an evaluation function, the best policy is picked by evaluating each of the learnt policies on the original game.