Towards Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games

1 Introduction . Many zero-sum games have a strong non-transitive component [ 2 , 4 ] in the policy space , and thus each player must acquire a diverse set of winning strategies to achieve high unexploitability [ 36 ] , which has been widely validated by recent studies of constructing AIs with superhuman performance in sophisticated tasks , such as StarCraft [ 30 , 28 ] and DOTA2 [ 25 , 39 ] . The non-transitivity in games means there is not a dominating strategy and the set of strategies form a cycle ( e.g. , the endless cycles among Rock , Paper and Scissors ) . It is the presence of this special structure in games that requires players to maintain a diverse set of policies . Otherwise , we only need to seek the strongest one . Formally , the necessity of diversity for zero-sum games lies in three ways : ( 1 ) policy evaluation : with the presence of non-transitivity , one can not justify the strength or weakness of a strategy through the outcome of the interaction with a single type of opponent ; ( 2 ) avoiding being exploited [ 24 ] : since in non-transitive games a single strategy can be always beaten by another one , a diverse set of strategies ⇤Correspondence to Ying Wen < ying.wen @ sjtu.edu.cn > . Code available at https : //github.com/ sjtu-marl/bd_rd_psro 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . allows players to make corresponding responses when encountering different opponents ; ( 3 ) training adaptable strategies [ 32 ] : a diverse set of training opponents helps gradually eliminate the weakness of a strategy , which can adapt to a wide range of opponents with very few interactions at test time . The open-ended learning framework is a promising direction towards inducing a population of distinct policies in zero-sum games via auto-curricula . Although various open-ended algorithms have been proposed to derive diverse strategies [ 2 , 24 , 6 , 22 ] , there are no consistent definitions for diversity . One of the most intuitive principles to characterize diversity is to build metrics over the trajectory or state-action distribution [ 6 , 20 ] . However , this perspective only focuses on the policy behaviors and ignores the reward attributes inherited from the Markov decision process . We argue that this is not reasonable since sometimes a slight difference in the policy can result in a huge difference in the final reward like , in maze . Contrary to this , another line of works builds the diversity measure over empirical payoffs [ 24 , 2 , 10 ] , thus revealing the underlying diverse behaviors of a strategy through the responses when encountering distinct opponents . In this work , based on all previous diversity concepts , we work towards offering a unified view for diversity in an open-ended learning framework by combining both the behavioral attribute and the response attribute of a strategy . The behavioral diversity is formulated through the occupancy measure , which is an equivalent representation of a policy . We hypothesize that the diversity in policy behaviors should be revealed by differences in the state-action distribution , and we use a general divergence family f -divergence to indicate the novelty of a new policy . On the other hand , gamescape [ 2 ] has been proposed to represent the response capacity of a population of strategies . Based on gamescape , we formulate a new geometric perspective to treat the response diversity by considering the distance to the gamescape . To summarize , in this paper , we provide the following contributions : • We formulate the concept of behavioral diversity in the state-action space as the discrepancies of occupancy measures and analyze the optimization methods in both normal-form games and general Markov games . • We provide a new geometric perspective on the response diversity as a form of Euclidean projection onto the convex hull of the meta-game to enlarge the gamescape directly and propose the optimization lower bound for practical implementation . • We analyze the limitation of exploitability as the evaluation metric and introduce a new metric with theoretical soundness called population effectivity , which is a fairer way to represent the effectiveness of a population than exploitability [ 16 ] . 2 Preliminaries . 2.1 Markov Games . The extension of Markov decision processes ( MDPs ) with more than one agents is commonly modelled as Markov games [ 18 ] . A Markov game with N agents is defined by a tuple < N , S , { Ai } Ni=1 , P , { ri } Ni=1 , ⌘ , > , where S denotes the state space and Ai is the action space for agent i . The function P controls the state transitions by the current state and one action from each agent : P : S ⇥A1 ⇥ · · · ⇥AN ! P ( S ) , where P ( S ) denotes the set of probability distributions over the state space S. Given the current state st and the joint action ( a1 , . . . , aN ) , the transition probability to st+1 is given by P ( st+1|st , a1 , . . . , aN ) . The initial state distribution is given by ⌘ : S ! [ 0 , 1 ] . Each agent i also has an associated reward function ri : S ⇥Ai ⇥ · · ·⇥AN ! R. Each agent ’ s goal is to maximize the -discounted expected return Ri = E [ P1 t=0 tri ( st , ati , a t i ) ] , where i is a compact representation of all complementary agents of i . Specifically , for zero-sum games , the rewards satisfy that PN i=1 ri ( s , a ) = 0 , and players need to behave competitively to achieve higher rewards . In multi-agent reinforcement learning ( MARL ) [ 37 ] , each agent is equipped with a policy ⇡i : S ⇥ Ai ! [ 0 , 1 ] and the joint policy is defined by ⇡ ( a|s ) = ⇧Ni=1⇡i ( ai|s ) . In single-agent reinforcement learning , occupancy measure is a principled way to characterize a policy , which indicates how a policy covers the state-action space . Inspired by the definition from the single-agent setting , we define the joint occupancy measure in MARL induced by the joint policy ⇡ ( a|s ) as : Definition 1 . ( Joint Occupancy Measure in MARL ) Let ⇢⇡ ( s ) : S ! R denote the normalized distribution of state visitation by following the joint policy ⇡ = ( ⇡i , ⇡ i ) in the environment : ⇢⇡ ( s ) = ( 1 ) 1X t=0 tP ( st = s|⇡ ) . ( 1 ) Then the distribution of state-action pairs ⇢⇡ ( s , a ) = ⇢⇡ ( s ) ⇡ ( a|s ) is called occupancy measure of the joint policy ⇡ . 2.2 Policy Space Response Oracle . Adapted from double oracle [ 23 ] , policy space response oracle ( PSRO ) [ 16 , 5 ] has been serving as a powerful tool to solve the nash equilibrium ( NE ) in zero-sum games . In PSRO , each player maintains a pool of policies , say Pi = { ⇡1i , . . . , ⇡Mi } for player i and P i = { ⇡1 i , . . . , ⇡N i } for player i . The so-called meta game APi⇥P i has its ( k , j ) entry as i ( ⇡ki , ⇡ j i ) , where the function i encapsulates the reward outcome for player i like the winning rate or expected return . When player i adds a new policy ⇡M+1i , it will compute the best response to the mixture of its opponents : Br ( P i ) = max ⇡M+1i X j j iE⇡M+1i , ⇡j i [ ri ( s , a ) ] . where = ( i , i ) is a distribution over policies in Pi and P i , which is usually a NE of APi⇥P i . The empirical gamescape is introduced by [ 2 ] to represent the expressiveness of a population Pi in the reward outcome level given the opponent population P i : Definition 2 . Given population Pi and P i with evaluation matrix APi⇥P i , the corresponding empirical gamescape ( EGS ) for Pi is defined as GPi|P i : = { convex mixtures of rows of APi⇥P i } . 2.3 Existing Diversity Measures . As the metric to measure the differences between models , diversity is an important topic in many fields of machine learning , including generative modelling [ 6 ] , latent variable models [ 33 ] , and robotics [ 1 ] . Specifically , in reinforcement learning ( RL ) , diversity is a useful tool for learning transferable skills [ 7 ] , boosting explorations [ 27 ] , or collecting near-optimal policies that are distinct in a meaningful way . Despite the importance of diversity , as shown in Table 1 , there has not been a consistent definition of diversity for RL , and various diversity concepts are used . [ 22 ] investigated behavioral diversity in multi-agent reinforcement learning through expected action variation , which is modeled as the average total variation distance of two action distributions under certain sampled states . Considering the geometric perspective that the determinant of the kernel matrix represents the volume of a parallelepiped spanned by feature maps , DvD [ 26 ] proposed the concept of population diversity using the determinant of the kernel matrix composed by the behavioral embeddings by multiple policies . Thanks to the tools of empirical game theory analysis , diversity can be modeled from the perspective of the empirical game . Effective diversity [ 2 ] is formulated as the weighted L1,1 norm of the empirical payoff matrix , which emphasizes what opponents a policy can win against . Also inspired by determinantal point process ( DPP ) [ 14 , 38 ] , [ 24 ] uses the expected cardinality to measure the diversity of a population . 3 A Unified Diversity Measure . Motivated by bisimulation metrics [ 8 ] to measure the similarity of two states in MDPs : d ( si , sj ) = maxa2A ( 1 c ) · Rasi R a sj + c ·W1 ⇣ Pasi , P a sj ; d ⌘ that considers both the immediate reward and the following transition dynamics , where c 2 [ 0 , 1 ] and W1 is the 1-Wasserstein distance , we want to build a metric to measure the similarity of two policies in a given Markov game through the task-specific reward attributes and the interaction between policy behaviors and transition dynamics . We will firstly model the interaction between policy behaviors and transition dynamics through the principled occupancy measure in MDPs , which encodes how a policy behaves in a given state and how the state will transit . On the reward side , the interaction responses with different opponents feature a policy , which can be used for common diversity measures like DPP [ 24 ] and rectified Nash [ 2 ] . 3.1 Behavioral Diversity via Occupancy Measure Mismatching . One fundamental way to characterize a policy in MDPs is through the distribution of the state-action pairs ( s , a ) . Formally , we define the occupancy measure in multi-agent learning as the distribution of the joint state-action distribution . It has been shown that there is a one-to-one correspondence between the joint policy ⇡ and the occupancy measure ⇢⇡ . Proposition 1 ( Theorem 2 of [ 29 ] ) . If ⇢ is a valid occupancy measure , then ⇢ is the occupancy measure for ⇡⇢ ( a | s ) = ⇢ ( s , a ) / P a0 ⇢ ( s , a 0 ) , and ⇡⇢ is the only policy whose occupancy measure is ⇢ . Usually , the policy ⇡ is parameterized as a neural network , and tackling the policy in the parameter space is intractable . However , due to the one-to-one correspondence between the policy and occupancy measure , the occupancy measure ⇢⇡ serves as a unique and informative representation for the policy ⇡ . Therefore , we are justified in considering diversity from a perspective of the occupancy measure . Next , we will consider how to promote diversity in the framework of policy space response oracle . Suppose after t iterations of PSRO , the joint policy aggregated according the distribution of nash is ⇡E = ( ⇡i , ⇡E i ) . The occupancy measure is given by ⇢⇡E . For player i in the t+ 1 iteration , it will seek the new policy ⇡0i , which can maximize the discrepancy between old ⇢⇡E and ⇢⇡0i , ⇡E i . max ⇡0i Divocc ( ⇡ 0 i ) = Df ( ⇢⇡0i , ⇡E i ||⇢⇡i , ⇡E i ) , ( 2 ) where we use the general f -divergence to measure the discrepancy of two distributions . We firstly investigate the objective under the one-step game by giving the following theorem : Theorem 1 . By assuming the game is a one-step game ( normal-form games , functional-form games , etc . ) and policies among players are independent , the behavioral diversity can be simplified by : Df ( ⇢⇡0i , ⇡E i ||⇢⇡i , ⇡E i ) = Es0⇠⌘ ( s ) [ Df ( ⇡ 0 i ( ·|s0 ) ||⇡i ( ·|s0 ) ) ] , ( 3 ) where ⌘ ( s ) is the initial state distribution . Proof . See Appendix A.1 . For more general Markov games , computing the exact occupancy measure is intractable . However , notice that we are maximizing a f -divergence objective of occupancy measures , while occupancy measure matching algorithms in imitation learning try to minimize the same objective [ 13 , 11 , 9 ] . Leveraging the powerful tool from occupancy measure matching , we here propose an approximate method to maximize Divocc . Prediction Error for Approximate Optimization . Inspired by random expert distillation [ 31 ] , a neural network f✓̂ ( s , a ) is trained to fit a randomly initialized fixed network f✓ ( s , a ) on the dataset of state-action pair ( s , a ) ⇠ ⇢⇡E . Then we can assign an intrinsic reward rinti ( s , a ) = ||f✓̂ ( s , a ) f✓ ( s , a ) || to the player , which will encourage the agent to visit the state-action with large prediction errors , thus pushing occupancy measure of the new policy to be different from the old one . Alternative Solutions . There are also many other practical occupancy measure matching algorithms . One popular paradigm is learning a discriminator D ( s , a ) to classify the state-action pair ( s , a ) from the distribution ⇢⇡0i , ⇡E i and the distribution ⇢⇡E . Then the trained D ( s , a ) can be used to construct different intrinsic rewards , which will correspond to different choices of f -divergence [ 13 , 11 , 9 ] . One major drawback of this paradigm is that the discriminator depends on the new policy ⇡0i and needs re-training once the policy ⇡0i is updated . Another popular paradigm is to learn an intrinsic reward directly from the target distribution ⇢⇡E like the prediction error . Besides using the prediction error , there are also other choices , including energy-based models ( EBM ) [ 19 ] and expert variance [ 3 ] . However , those methods usually require specialized training techniques .

Summary: This paper is concerned with learning in zero-sum Markov games, in which there are often non-transitive dynamics between policies/strategies (e.g., Rock beats Scissors beats Paper beats Rock ...). A common approach to avoid getting stuck in cycles or local optima when training in such domains is to encourage algorithms to discover *diverse* solutions or policies.   This paper discusses and formalises how diversity measures can be split up in two broad categories: **behavioral diversity** measures, which measure whether different policies visit different state-action distributions, and **response diversity** measures, which measure whether different policies obtain different vectors of empirical outcomes against a population of opponents. For each of these two categories, the paper proposes a concrete measure and (approximate) methods to optimise them within a Policy Space Response Oracle (PSRO) framework. Additionally, the paper proposes a new Population Effectivity metrid, which is argued to be more fair and informative for evaluating the effectiveness of a population of policies than the Exploitability metric.  The additional diversity objectives are shown to improve training in several different domains.