K-level Reasoning for Zero-Shot Coordination in Hanabi

1 Introduction . Research into multi-agent reinforcement learning ( MARL ) has recently seen a flurry of activity , ranging from large-scale multiplayer zero-sum settings such as StarCraft [ 35 ] to partially observable , fully cooperative settings , such as Hanabi [ 1 ] . The latter ( cooperative ) setting is of particular interest , as it covers human-AI coordination , one of the longstanding goals of AI research [ 10 , 6 ] . However , most work in the cooperative setting—typically modeled as a Dec-POMDPs—has approached the problem in the self-play ( SP ) setting , where the only goal is to find a team of agents that works well together . Unfortunately , optimal SP policies in Dec-POMDPs commonly communicate information through arbitrary handshakes ( or conventions ) , which fail to generalize to other , independently trained , AI agents or humans at test time . To address this , the zero-shot coordination setting [ 18 ] was recently introduced , where the goal is to find training strategies that allow independently trained agents to coordinate at test time . The main idea of this line of work is to develop learning algorithms that can use the structure of the DecPOMDP itself to independently find mutually compatible policies , a necessary step towards human-AI coordination . Related coordination problems have also been studied by different communities , in ∗Work done while at Facebook AI Research 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) particular behavioural game theory . One of the best-known approaches in this area is the cognitivehierarchies ( CH ) framework [ 4 ] , in which a hierarchy of agents is trained . For this method , an agent at level-k models other agents as coming from a distribution up to level k − 1 and best-responds accordingly . The CH framework has been shown to model human behavior in games for which equilibrium theory does not match empirical data [ 4 ] ; thus , in principle , the CH framework could be leveraged to facilitate human-AI coordination in complex settings . A specific instance of CH that is relevant to our work is K-level reasoning ( KLR ) [ 7 ] , wherein the level-k agent models the other agents as level- ( k − 1 ) . However , KLR , like many of the ideas developed in these works , has not been successfully scaled to large scale coordination problems [ 18 ] . In this paper we show that k-level reasoning can indeed be scaled to large partially observable coordination problems , like Hanabi . We identify two key innovations that both increase training speed and improve the performance of the method . First , rather than training the different levels of the hierarchy sequentially , as would be suggested by a literal interpretation of the method ( as was done as a baseline in [ 18 ] ) , we instead develop a synchronous version , where all levels are trained in parallel ( see figure 1 ) . The obvious advantage is that the wall-clock time can be reduced from linear in the number of levels to constant , taking advantage of parallel training . The more surprising finding is that synchronous training also acts as a regularizer on the policies , stabilizing training . The second innovation is that in parallel we also train a best response ( BR ) to the entire KLR hierarchy , with more weight being placed on the highest two levels . This constitutes a hybrid approach between CH and KLR , and the resulting BR is our final test time policy . Our method , synchronous-k-level reasoning with a best response ( SyKLRBR ) , obtains high scores when evaluating independently trained agents in cross-play ( XP ) . Importantly , this method also improves ad-hoc teamplay performance , indicating a robust policy that plays well with various conventions . Lastly , we evaluate our SyKLRBR agents paired with a proxy human policy and establish new stateof-the-art performance , beating recent strong algorithms that , in contrast to our approach , require additional information beyond the game-provided observations [ 18 , 17 ] . Our results show that indeed KLR can be adapted to address large scale coordination problems , in particular those in which the main challenge is to prevent information exchange through arbitrary conventions . Our analysis shows that synchronous training regularizes the training process and prevents level-k from overfitting to the now changing policy at level-k − 1 . In contrast , in sequential training each agent overfits to the static agent at the level below , leading to arbitrary handshakes and brittle conventions . Furthermore , training a best response to the entire hierarchy improves the final ZSC performance and robustness in ad-hoc settings . This is intuitive since the BR can carry out on-the-fly adaptation in the ZSC setting . Our results show that the exact graph-structure used , which were similarly studied in [ 13 ] , and the type of training regime ( synchronous vs sequential ) can have a major impact on the final outcome when adapting ideas from the cognitive hierarchy literature to the deep MARL setting . We hope our findings will encourage other practitioners to seek inspiration in the game theory literature and to scale those ideas to high dimensional problems , even when there is precedent of unsuccessful attempts in prior work . 2 Related Work . A significant portion of research in MARL has been focused on creating agents that do well in fully cooperative , partially observable settings . A standard approach is through variations of self-play ( SP ) methods [ 9 , 8 , 12 , 24 , 15 ] ; however , as shown in [ 18 ] generally optimal SP agents learn highly arbitrary policies , which are incompatible with independently trained agents . Clearly , test-time coordination with other , independently trained agents including humans is an important requirement for AI agents that is not captured in the SP problem setting . To address this , [ 18 ] introduced the zero-shot coordination ( ZSC ) setting , where the explicit goal is to develop learning algorithms that allow independently trained agents to collaborate at test time . Another recent area of work trains an RL agent separately , and then evaluate its performance in a new group of AI agents or humans assuming access to a small amount of test-time data [ 24 , 33 ] . These methods build SP policies that are compatible with the test time agents by guiding the learning to the nearest equilibria [ 24 , 33 ] . Other methods use human data to build a human model and then train an approximate best response to this human model , making it compatible with human play [ 5 ] . While this presents a promising near-term approach for learning human-like policies in specific settings where we have enough data , it does not enable us to understand the fundamental principles of coordination that lead to this behavior in the first place , which is the goal of the ZSC setting . Our paper shows that even relatively simple training methods can lead to drastically improved ZSC performance when combined with modern engineering best practices and , importantly , that the performance gains directly translate into better coordination with a human-like proxy and ad-hoc teamplay ( without requiring human data in the training process ) . Our approach for scaling KLR to Hanabi relies heavily on the parallel training of all of the different levels , where each level is trained on one machine and the models are exchanged periodically through a central server . This architecture draws inspiration from population based training ( PBT ) , which was first popularized for hyperparameter turning [ 20 ] and then applied in the multi-agent context to train more robust policies in two player zero-sum team settings [ 19 ] . PBT has also been used to obtain better self-play policies in Hanabi , both in [ 12 ] and [ 1 ] . In contrast to prior work , we do not use PBT to avoid local optima or train less exploitable agents but instead leverage this framework to implement a KLR and a best response to this KLR that is geared towards ZSC and coordination with human-like proxies . There are a few other methods directly addressing the ZSC framework . The first , other-play ( OP ) [ 18 ] requires access to the ground truth symmetries of the problem setting and then learns policies that avoid breaking these symmetries during training . OP has previously been applied to Hanabi and KLR compares favorably to the OP results ( see Section 4 ) . We also note , that KLR does not require access to the symmetries and can be applied in settings where no symmetries are present . The next method , Ridge Rider ( RR ) [ 27 ] uses the connection between symmetries in an environment and repeated eigenvalues of the Hessian , to solve ZSC problems . Like KLR , RR does not require prior ground truth access . However , unlike KLR , RR is extremely computationally expensive and has not been scaled to large scale RL problems . Life-Long Learning ( LLL ) has been studied for ZSC [ 26 ] . However , LLL requires access to a pool of pre-trained agents , and in this case they had access to symmetries , whereas our method never required access to such symmetries and our method compares favorably in the ZSC setting . Lastly , Off-Belief Learning ( OBL ) [ 17 ] has been shown to provide well-grounded play in hanabi and strong results in the ZSC setting , but requires simulator access to train . We note that KLR doesn ’ t require simulator access and also matches or even outperforms OBL on various metrics . 3 Background . 3.1 Dec-POMDPs . This work considers a class of problems , Dec-POMDPs [ 2 ] , where N agents interact in a partially observable environment . The partial observability implies that every agent i ∈ { 1 , · · · , N } has an observation oit = O ( st ) obtained from via the observation function O from the underlying state st ∈ S. In our setting , at each timestep t the acting agent i samples an action uit from policy πi , uit ∼ πiθ ( ui|τ it ) , where θ are the weights of the neural networks , and all other agents take no-op actions . Here we use action-observation histories ( AOH ) which we denote as τ i = { oi0 , ui0 , r1 · · · , rT−1 , oiT } , where T is the length of the trajectory , and rt is the common reward at timestep t defined by the reward function R ( s , u ) . The goal of the agents is to maximize the total reward Eτ∼P ( τ |s , u ) [ Rt ( τ ) ] ; here we consider Rt ( τ ) to be the discounted sum of rewards , i.e . Rt ( τ ) = ∑∞ t′=t γ t′−trt′ , where γ is the discount factor . Additionally , the environments in this work are turn-based and bounded in length at tmax . 3.2 Deep Multi-Agent Reinforcement Learning . Deep reinforcement learning has been applied to a multitude of multi-agent learning problems with great success . Cooperative MARL is readily addressed with extensions of Deep Q-learning [ 25 ] , where the Q-function is parameterized by neural networks to learn to predict the expected return based on the current state s and action u , Q ( τt , ut ) = Eτ∼P ( τ |s , u ) Rt ( τ ) . Our work also builds off of state of the state of the art algorithm Recurrent Replay Distributed Deep Q-Network ( R2D2 ) [ 21 ] . R2D2 also incorporates other recent advancements such as using a dueling network architecture [ 36 ] , prioritized replay experience [ 29 ] , and double DQN learning [ 34 ] . Additionally , we use a similar architecture as the one proposed in [ 14 ] and run many environments in parallel , each of which has actors with varying exploration rates that add to a centralized replay buffer . The simplest way to adapt deep Q-learning to the Dec-POMDP setting is through Independent Q-learning ( IQL ) as proposed by [ 32 ] . In IQL , every agent individually estimates the total return and treats other agents as part of the environment . There are other methods that explicitly account for the multi-agent structure by taking advantage of the centralized training with decentralized control regime [ 31 , 28 ] . However , since our work is based on learning best responses , here we only consider IQL .

Summary: This paper demonstrates an approach to hierarchically train agents for Hanabi, adapted from a classical approach called k-level reasoning. The authors demonstrate that training these hierarchies in parallel improves performance by reducing overfitting. They present a novel training schema they call SyKLRBR, which demonstrates SOTA performance with a couple of agents. The paper ends with a large number of experiments and discussion.