Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination

1 INTRODUCTION . Deep Reinforcement Learning ( RL ) has gained many successes against humans in competitive games , such as Go ( Silver et al. , 2017 ) , Dota ( OpenAI , 2019 ) , and StarCraft ( Vinyals et al. , 2019a ) . However , it still remains a challenge to build AI agents that can coordinate and collaborate with humans that the agents have not seen during training ( Kleiman-Weiner et al. , 2016 ; Lerer & Peysakhovich , 2017 ; Carroll et al. , 2019 ; Shum et al. , 2019 ; Hu et al. , 2020 ; Knott et al. , 2021 ) . Zero-shot human-AI coordination is particularly important in real-world applications , such as cooperative games ( Carroll et al. , 2019 ) , communicative agents ( Foerster et al. , 2016 ) , self-driving vehicles ( Resnick et al. , 2018 ) , and assistant robots ( Andrychowicz et al. , 2018 ) . Ultimately , we want to build AI systems that can assist humans and augmenting our capabilities ( Engelbart , 1962 ; Carter & Nielsen , 2017 ) to make our life better . The mainstream method for building state-of-the-art AI agents is through self-play reinforcement learning ( Tesauro , 1994 ; Silver et al. , 2017 ) . Self-play-trained agents are very specialized , and therefore suffered significantly from distributional shift when paired with humans . For example , in the Overcooked game , the self-play-trained agents only use a specific pot and ignore the others . However , humans use all pots . The AI agent ends up waiting unproductively for the human to deliver a soup from the specific pot , while the human has instead decided to fill up the other pots ( Carroll et al. , 2019 ) . Since the AI agent has only encountered its own policy during training , it undergoes a distributional shift when it is paired with human players . We explore how to design a robust and efficient approach to train agents for zero-shot human-AI coordination . To that end , we draw on the advances in maximum entropy RL ( Haarnoja et al. , 2018b ) , diversity ( Eysenbach et al. , 2019 ) , and Multi-agent RL ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) . First , maximum entropy RL augments the standard reward function with an entropy maximization term and provides a substantial improvement in exploration and robustness ( Ziebart et al. , 2008 ; Toussaint , 2009 ; Ziebart , 2010 ; Rawlik et al. , 2013 ; Fox et al. , 2015 ; Haarnoja et al. , 2017 ; 2018b ; Zhao et al. , 2019 ) . For a population of agents , we use the maximum entropy bonus to encourage individual diversity , i.e. , each agent ’ s policy to be diverse and exploratory . Secondly , to acquire diverse and distinguishable behaviors ( Eysenbach et al. , 2019 ) , we further utilize the sum of crossentropy terms across all agent pairs in the population to encourage pairwise diversity . We define the combination of individual diversity and pairwise diversity as population diversity . Thirdly , analog to multi-agent RL ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) , the population diversity is calculated in a centralized fashion . Each agent in the population is rewarded to maximize the centralized population diversity . Subsequently , we derive a safe and computationally efficient surrogate objective , i.e. , population entropy , which is proven to be a lower bound of the original population diversity objective . The population entropy is defined as the entropy of the averaged action probability distribution conditioned on the state across all agents in the population . Eventually , we train a new agent to collaborate with each agent in the diversified population using prioritized sampling of agents policies from the population based on the learning progress ( Schaul et al. , 2016 ; Vinyals et al. , 2019a ; b ; Han et al. , 2020 ) . The newly trained AI agent encounters a diverse set of strategies and is in general more robust to human behaviors ( Pan & Yang , 2009 ; Tobin et al. , 2017 ; Andrychowicz et al. , 2018 ) . We evaluate the proposed Maximum Entropy Population-based training ( MEP ) framework in a testbed based on the popular Overcooked game ( Ghost Town Games , 2016 ) . Our contributions are three-fold . First , based on the novel population diversity objective that considers both agents ’ individual diversity and pairwise diversity , we derive a safe and computationally efficient surrogate objective , i.e. , the population entropy . Secondly , we develop the MEP framework , which comprises training a diversified population and using this population to train a robust AI agent . Last but not least , in the experiments , we show MEP ’ s superior performance , by comparing it with state-of-the-art baseline methods . To further verify the improvements , we conduct online experiments with real humans . 2 PRELIMINARIES . Markov Decision Process : A two-player Markov Decision Process ( MDP ) is defined by a tuple M = 〈S , { A ( i ) } , P , γ , R〉 ( Boutilier , 1996 ) , where S is a set of states ; A ( i ) is a set of the i-th agent ’ s actions , where i ∈ [ 1 , 2 ] ; P is a transition function that maps the current state and all agents ’ actions to the next state ; γ is the discount factor ; R is a reward function . The i-th agent ’ s policy is π ( i ) . A trajectory is denoted by τ . The shared objective is to maximize the expected sum rewards , which is Eτ [ ∑ tR ( st , at ) ] , where at = ( a ( 1 ) t , a ( 2 ) t ) . We can extend the objective to infinite horizon problems by the discount factor γ to ensure that the sum of expected rewards is finite . AI Agent , Population , and Human : Throughout this paper , we use the phrase AI agent to explicitly mean the agent that plays the AI role in human-AI coordination . The population of agents is used to train the AI agent to make it capable of cooperating with different partner agents . The human policy is represented as π ( H ) and a model of the human policy is π̂ ( H ) . The AI agent is denoted as π ( A ) . Environment : We use the Overcooked environment ( Carroll et al. , 2019 ) as the human-AI coordination testbed , see Figure 2 . In the Overcooked game , to have a high score , it naturally requires coordination and collaboration between the two players . The players are tasked to cook soups . Maximum Entropy RL : Standard RL maximizes the expected sum of rewards Eτ [ ∑ tR ( st , at ) ] . At the beginning of learning , almost all actions have equal probability . After some training , some actions have a higher probability in the direction of accumulating more rewards . Subsequently , the entropy of the policy is reduced over time during training ( Mnih et al. , 2016 ) . Maximum entropy RL augments the standard RL objective with a maximum entropy term ( Ziebart , 2010 ; Haarnoja et al. , 2018b ) , which gives a reward to the agent if it selects the non-dominate actions during training , and the higher the reward favors more exploration . The maximum entropy RL objective is defined as : J ( π ) = Eτ [ ∑ t R ( st , at ) + αH ( π ( · |st ) ) ] . ( 1 ) The parameter α adjusts the relative importance of the entropy bonus against the reward and controls the stochasticity of the optimal policy . The maximum entropy RL objective has a number of advantages . First , the policy is incentivized to explore more widely . Prior works have demonstrated improved exploration using the maximum entropy RL objective ( Haarnoja et al. , 2017 ; Schulman et al. , 2017a ) . Secondly , the policy can capture multiple modes of optimal behaviors . In situations where multiple actions are equally important , the policy will give equal probability mass to those actions . Lastly , recent works have shown improved robustness of the policy trained with maximum entropy RL ( Haarnoja et al. , 2018a ; 2019 ) . 3 METHOD . In this section , we first define the population diversity objective , which includes the policy entropy and the pairwise difference among policies . Secondly , we derive a safe and computationally efficient surrogate objective , i.e. , the population entropy , for optimization . Thirdly , we illustrate the MEP framework , which comprises two parts : training a maximum entropy population and training a robust AI agent via prioritized sampling using the population . 3.1 POPULATION DIVERSITY . Motivated by maximum entropy RL , we want to make the policy population exploratory and diverse . First , by utilizing the maximum entropy bonus , we encourage each policy itself to be diverse and multi-modal . Secondly , to encourage the policies in the population , { π ( 1 ) , π ( 2 ) , ... , π ( n ) } , to be complementary and mutually different , we naturally utilize the cross entropy objective ( Murphy , 2012 ) . Formally , we define the Population Diversity ( PD ) as the combination of the sum of the entropy of each agent ’ s policy and the sum of the Cross Entropy ( CE ) between each pair of agents in the population . Mathematically , PD ( { π ( 1 ) , π ( 2 ) , ... , π ( n ) } , st ) : = n∑ i=1 n∑ j 6=i CE ( π ( i ) ( · |st ) , π ( j ) ( · |st ) ) + n∑ i=1 H ( π ( i ) ( · |st ) ) , ( 2 ) where cross entropy ( CE ) and entropy ( H ) are defined as follows : CE ( π ( i ) ( · |st ) , π ( j ) ( · |st ) ) = − ∑ a∈A π ( i ) ( at|st ) log π ( j ) ( at|st ) , ( 3 ) H ( π ( i ) ( · |st ) ) = − ∑ a∈A π ( i ) ( at|st ) log π ( i ) ( at|st ) . ( 4 ) This objective not only captures single agent ’ s diversity but also encourage multiple agents to be far from each other . However , the population diversity objective is not safe to be maximized as a reward function , since cross entropy is unbounded . Besides , evaluating the population diversity objective has a quadratic runtime complexity of O ( n2 ) , where n is the population size . 3.2 POPULATION ENTROPY . To improve the stability and the runtime complexity , we derive a safe and efficient surrogate objective for optimization , which is named the Population Entropy ( PE ) . Population entropy is defined as the entropy of the mean of all policies in the population . Mathematically , PE ( { π ( 1 ) , π ( 2 ) , ... , π ( n ) } , st ) : = H ( π̄ ( · |st ) ) , where π̄ ( at|st ) : = 1 n n∑ i=1 π ( i ) ( at|st ) . ( 5 ) The population entropy serves as a lower bound of the population diversity objective . Theorem 1 . Let the population diversity be defined as Equation ( 2 ) . Let the population entropy be defined as Equation ( 5 ) . Then , we have PD ( { π ( 1 ) , π ( 2 ) , ... , π ( n ) } , st ) ≥ n2PE ( { π ( 1 ) , π ( 2 ) , ... , π ( n ) } , st ) , ( 6 ) where n is the population size . Proof . See Appendix A . Compared to the population diversity objective , the population entropy objective , H ( π̄ ( · |st ) ) , is safe to maximize and it has only a linear runtime complexity O ( n ) . Therefore , we use the derived population entropy objective for optimization . Taking a closer look at the population entropy objective , we find that it can be written into the Jensen-Shannon Divergence ( JSD ) and entropy form as follows : H ( π̄ ( · |st ) ) = JSD ( π ( 1 ) ( · |st ) , ... , π ( n ) ( · |st ) ) + 1 n n∑ i=1 H ( π ( i ) ( · |st ) ) , ( 7 ) A step-by-step derivation is in Appendix B . From Equation ( 7 ) , we can see that when we maximize the population entropy objective , it attempts to push each policy away from each other via the JSD term in Equation 7 , as well as increase each policy ’ s entropy via the entropy term in Equation 7 .

Summary: The authors focus on training agents for zero-shot human-AI coordination on Overcooked. They propose to first train a population of agents with a population entropy bonus, and then train another agent as the best response to the population, using a prioritized sampling approach that focuses more training on the worst performing pairs. They evaluate their agent and some baselines and ablations both with a human proxy model (i.e. an agent trained on human-human gameplay data with behavioral cloning) and also a smaller subset of their agent and baselines with real humans on Mechanical Turk. Their method performs as well or better in terms of game score than the baselines and ablations presented.