Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination

1 INTRODUCTION . Deep Reinforcement Learning ( RL ) has gained many successes against humans in competitive games , such as Go ( Silver et al. , 2017 ) , Dota ( OpenAI , 2019 ) , and StarCraft ( Vinyals et al. , 2019a ) . However , it still remains a challenge to build AI agents that can coordinate and collaborate with humans that the agents have not seen during training ( Kleiman-Weiner et al. , 2016 ; Lerer & Peysakhovich , 2017 ; Carroll et al. , 2019 ; Shum et al. , 2019 ; Hu et al. , 2020 ; Knott et al. , 2021 ) . Zero-shot human-AI coordination is particularly important in real-world applications , such as cooperative games ( Carroll et al. , 2019 ) , communicative agents ( Foerster et al. , 2016 ) , self-driving vehicles ( Resnick et al. , 2018 ) , and assistant robots ( Andrychowicz et al. , 2018 ) . Ultimately , we want to build AI systems that can assist humans and augmenting our capabilities ( Engelbart , 1962 ; Carter & Nielsen , 2017 ) to make our life better . The mainstream method for building state-of-the-art AI agents is through self-play reinforcement learning ( Tesauro , 1994 ; Silver et al. , 2017 ) . Self-play-trained agents are very specialized , and therefore suffered significantly from distributional shift when paired with humans . For example , in the Overcooked game , the self-play-trained agents only use a specific pot and ignore the others . However , humans use all pots . The AI agent ends up waiting unproductively for the human to deliver a soup from the specific pot , while the human has instead decided to fill up the other pots ( Carroll et al. , 2019 ) . Since the AI agent has only encountered its own policy during training , it undergoes a distributional shift when it is paired with human players . We explore how to design a robust and efficient approach to train agents for zero-shot human-AI coordination . To that end , we draw on the advances in maximum entropy RL ( Haarnoja et al. , 2018b ) , diversity ( Eysenbach et al. , 2019 ) , and Multi-agent RL ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) . First , maximum entropy RL augments the standard reward function with an entropy maximization term and provides a substantial improvement in exploration and robustness ( Ziebart et al. , 2008 ; Toussaint , 2009 ; Ziebart , 2010 ; Rawlik et al. , 2013 ; Fox et al. , 2015 ; Haarnoja et al. , 2017 ; 2018b ; Zhao et al. , 2019 ) . For a population of agents , we use the maximum entropy bonus to encourage individual diversity , i.e. , each agent ’ s policy to be diverse and exploratory . Secondly , to acquire diverse and distinguishable behaviors ( Eysenbach et al. , 2019 ) , we further utilize the sum of crossentropy terms across all agent pairs in the population to encourage pairwise diversity . We define the combination of individual diversity and pairwise diversity as population diversity . Thirdly , analog to multi-agent RL ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) , the population diversity is calculated in a centralized fashion . Each agent in the population is rewarded to maximize the centralized population diversity . Subsequently , we derive a safe and computationally efficient surrogate objective , i.e. , population entropy , which is proven to be a lower bound of the original population diversity objective . The population entropy is defined as the entropy of the averaged action probability distribution conditioned on the state across all agents in the population . Eventually , we train a new agent to collaborate with each agent in the diversified population using prioritized sampling of agents policies from the population based on the learning progress ( Schaul et al. , 2016 ; Vinyals et al. , 2019a ; b ; Han et al. , 2020 ) . The newly trained AI agent encounters a diverse set of strategies and is in general more robust to human behaviors ( Pan & Yang , 2009 ; Tobin et al. , 2017 ; Andrychowicz et al. , 2018 ) . We evaluate the proposed Maximum Entropy Population-based training ( MEP ) framework in a testbed based on the popular Overcooked game ( Ghost Town Games , 2016 ) . Our contributions are three-fold . First , based on the novel population diversity objective that considers both agents ’ individual diversity and pairwise diversity , we derive a safe and computationally efficient surrogate objective , i.e. , the population entropy . Secondly , we develop the MEP framework , which comprises training a diversified population and using this population to train a robust AI agent . Last but not least , in the experiments , we show MEP ’ s superior performance , by comparing it with state-of-the-art baseline methods . To further verify the improvements , we conduct online experiments with real humans . 2 PRELIMINARIES . Markov Decision Process : A two-player Markov Decision Process ( MDP ) is defined by a tuple M = 〈S , { A ( i ) } , P , γ , R〉 ( Boutilier , 1996 ) , where S is a set of states ; A ( i ) is a set of the i-th agent ’ s actions , where i ∈ [ 1 , 2 ] ; P is a transition function that maps the current state and all agents ’ actions to the next state ; γ is the discount factor ; R is a reward function . The i-th agent ’ s policy is π ( i ) . A trajectory is denoted by τ . The shared objective is to maximize the expected sum rewards , which is Eτ [ ∑ tR ( st , at ) ] , where at = ( a ( 1 ) t , a ( 2 ) t ) . We can extend the objective to infinite horizon problems by the discount factor γ to ensure that the sum of expected rewards is finite . AI Agent , Population , and Human : Throughout this paper , we use the phrase AI agent to explicitly mean the agent that plays the AI role in human-AI coordination . The population of agents is used to train the AI agent to make it capable of cooperating with different partner agents . The human policy is represented as π ( H ) and a model of the human policy is π̂ ( H ) . The AI agent is denoted as π ( A ) . Environment : We use the Overcooked environment ( Carroll et al. , 2019 ) as the human-AI coordination testbed , see Figure 2 . In the Overcooked game , to have a high score , it naturally requires coordination and collaboration between the two players . The players are tasked to cook soups . Maximum Entropy RL : Standard RL maximizes the expected sum of rewards Eτ [ ∑ tR ( st , at ) ] . At the beginning of learning , almost all actions have equal probability . After some training , some actions have a higher probability in the direction of accumulating more rewards . Subsequently , the entropy of the policy is reduced over time during training ( Mnih et al. , 2016 ) . Maximum entropy RL augments the standard RL objective with a maximum entropy term ( Ziebart , 2010 ; Haarnoja et al. , 2018b ) , which gives a reward to the agent if it selects the non-dominate actions during training , and the higher the reward favors more exploration . The maximum entropy RL objective is defined as : J ( π ) = Eτ [ ∑ t R ( st , at ) + αH ( π ( · |st ) ) ] . ( 1 ) The parameter α adjusts the relative importance of the entropy bonus against the reward and controls the stochasticity of the optimal policy . The maximum entropy RL objective has a number of advantages . First , the policy is incentivized to explore more widely . Prior works have demonstrated improved exploration using the maximum entropy RL objective ( Haarnoja et al. , 2017 ; Schulman et al. , 2017a ) . Secondly , the policy can capture multiple modes of optimal behaviors . In situations where multiple actions are equally important , the policy will give equal probability mass to those actions . Lastly , recent works have shown improved robustness of the policy trained with maximum entropy RL ( Haarnoja et al. , 2018a ; 2019 ) . 3 METHOD . In this section , we first define the population diversity objective , which includes the policy entropy and the pairwise difference among policies . Secondly , we derive a safe and computationally efficient surrogate objective , i.e. , the population entropy , for optimization . Thirdly , we illustrate the MEP framework , which comprises two parts : training a maximum entropy population and training a robust AI agent via prioritized sampling using the population . 3.1 POPULATION DIVERSITY . Motivated by maximum entropy RL , we want to make the policy population exploratory and diverse . First , by utilizing the maximum entropy bonus , we encourage each policy itself to be diverse and multi-modal . Secondly , to encourage the policies in the population , { π ( 1 ) , π ( 2 ) , ... , π ( n ) } , to be complementary and mutually different , we naturally utilize the cross entropy objective ( Murphy , 2012 ) . Formally , we define the Population Diversity ( PD ) as the combination of the sum of the entropy of each agent ’ s policy and the sum of the Cross Entropy ( CE ) between each pair of agents in the population . Mathematically , PD ( { π ( 1 ) , π ( 2 ) , ... , π ( n ) } , st ) : = n∑ i=1 n∑ j 6=i CE ( π ( i ) ( · |st ) , π ( j ) ( · |st ) ) + n∑ i=1 H ( π ( i ) ( · |st ) ) , ( 2 ) where cross entropy ( CE ) and entropy ( H ) are defined as follows : CE ( π ( i ) ( · |st ) , π ( j ) ( · |st ) ) = − ∑ a∈A π ( i ) ( at|st ) log π ( j ) ( at|st ) , ( 3 ) H ( π ( i ) ( · |st ) ) = − ∑ a∈A π ( i ) ( at|st ) log π ( i ) ( at|st ) . ( 4 ) This objective not only captures single agent ’ s diversity but also encourage multiple agents to be far from each other . However , the population diversity objective is not safe to be maximized as a reward function , since cross entropy is unbounded . Besides , evaluating the population diversity objective has a quadratic runtime complexity of O ( n2 ) , where n is the population size . 3.2 POPULATION ENTROPY . To improve the stability and the runtime complexity , we derive a safe and efficient surrogate objective for optimization , which is named the Population Entropy ( PE ) . Population entropy is defined as the entropy of the mean of all policies in the population . Mathematically , PE ( { π ( 1 ) , π ( 2 ) , ... , π ( n ) } , st ) : = H ( π̄ ( · |st ) ) , where π̄ ( at|st ) : = 1 n n∑ i=1 π ( i ) ( at|st ) . ( 5 ) The population entropy serves as a lower bound of the population diversity objective . Theorem 1 . Let the population diversity be defined as Equation ( 2 ) . Let the population entropy be defined as Equation ( 5 ) . Then , we have PD ( { π ( 1 ) , π ( 2 ) , ... , π ( n ) } , st ) ≥ n2PE ( { π ( 1 ) , π ( 2 ) , ... , π ( n ) } , st ) , ( 6 ) where n is the population size . Proof . See Appendix A . Compared to the population diversity objective , the population entropy objective , H ( π̄ ( · |st ) ) , is safe to maximize and it has only a linear runtime complexity O ( n ) . Therefore , we use the derived population entropy objective for optimization . Taking a closer look at the population entropy objective , we find that it can be written into the Jensen-Shannon Divergence ( JSD ) and entropy form as follows : H ( π̄ ( · |st ) ) = JSD ( π ( 1 ) ( · |st ) , ... , π ( n ) ( · |st ) ) + 1 n n∑ i=1 H ( π ( i ) ( · |st ) ) , ( 7 ) A step-by-step derivation is in Appendix B . From Equation ( 7 ) , we can see that when we maximize the population entropy objective , it attempts to push each policy away from each other via the JSD term in Equation 7 , as well as increase each policy ’ s entropy via the entropy term in Equation 7 .

Summary: The paper aims to address the important problem of training AI agents in multi-agent games. Standard self-play training leads to agents that overfit to a particular partner, and even naive population based training may not help much if the population is not very diverse. A number of recent works have looked into generating diversity in the partner strategies so that population based training methods can work better. By covering a diverse set of trajectories/policies, it is more likely that coordinating with a new partner (perhaps a human) at test time will fall in-distribution relative to our training set partners.  The paper proposes a population diversity metric based on cross-entropy between different pairs of agents in the population. They optimize a lower bound to this metric based on the population entropy. They also propose a 'prioritized sampling' procedure that determines which partner in the population the ego-agent should train with. They experiment on the game of Overcooked, compare with TrajeDi, and run a human experiment to check performance of coordination with humans at test time.