K-level Reasoning for Zero-Shot Coordination in Hanabi

1 Introduction . Research into multi-agent reinforcement learning ( MARL ) has recently seen a flurry of activity , ranging from large-scale multiplayer zero-sum settings such as StarCraft [ 35 ] to partially observable , fully cooperative settings , such as Hanabi [ 1 ] . The latter ( cooperative ) setting is of particular interest , as it covers human-AI coordination , one of the longstanding goals of AI research [ 10 , 6 ] . However , most work in the cooperative setting—typically modeled as a Dec-POMDPs—has approached the problem in the self-play ( SP ) setting , where the only goal is to find a team of agents that works well together . Unfortunately , optimal SP policies in Dec-POMDPs commonly communicate information through arbitrary handshakes ( or conventions ) , which fail to generalize to other , independently trained , AI agents or humans at test time . To address this , the zero-shot coordination setting [ 18 ] was recently introduced , where the goal is to find training strategies that allow independently trained agents to coordinate at test time . The main idea of this line of work is to develop learning algorithms that can use the structure of the DecPOMDP itself to independently find mutually compatible policies , a necessary step towards human-AI coordination . Related coordination problems have also been studied by different communities , in ∗Work done while at Facebook AI Research 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) particular behavioural game theory . One of the best-known approaches in this area is the cognitivehierarchies ( CH ) framework [ 4 ] , in which a hierarchy of agents is trained . For this method , an agent at level-k models other agents as coming from a distribution up to level k − 1 and best-responds accordingly . The CH framework has been shown to model human behavior in games for which equilibrium theory does not match empirical data [ 4 ] ; thus , in principle , the CH framework could be leveraged to facilitate human-AI coordination in complex settings . A specific instance of CH that is relevant to our work is K-level reasoning ( KLR ) [ 7 ] , wherein the level-k agent models the other agents as level- ( k − 1 ) . However , KLR , like many of the ideas developed in these works , has not been successfully scaled to large scale coordination problems [ 18 ] . In this paper we show that k-level reasoning can indeed be scaled to large partially observable coordination problems , like Hanabi . We identify two key innovations that both increase training speed and improve the performance of the method . First , rather than training the different levels of the hierarchy sequentially , as would be suggested by a literal interpretation of the method ( as was done as a baseline in [ 18 ] ) , we instead develop a synchronous version , where all levels are trained in parallel ( see figure 1 ) . The obvious advantage is that the wall-clock time can be reduced from linear in the number of levels to constant , taking advantage of parallel training . The more surprising finding is that synchronous training also acts as a regularizer on the policies , stabilizing training . The second innovation is that in parallel we also train a best response ( BR ) to the entire KLR hierarchy , with more weight being placed on the highest two levels . This constitutes a hybrid approach between CH and KLR , and the resulting BR is our final test time policy . Our method , synchronous-k-level reasoning with a best response ( SyKLRBR ) , obtains high scores when evaluating independently trained agents in cross-play ( XP ) . Importantly , this method also improves ad-hoc teamplay performance , indicating a robust policy that plays well with various conventions . Lastly , we evaluate our SyKLRBR agents paired with a proxy human policy and establish new stateof-the-art performance , beating recent strong algorithms that , in contrast to our approach , require additional information beyond the game-provided observations [ 18 , 17 ] . Our results show that indeed KLR can be adapted to address large scale coordination problems , in particular those in which the main challenge is to prevent information exchange through arbitrary conventions . Our analysis shows that synchronous training regularizes the training process and prevents level-k from overfitting to the now changing policy at level-k − 1 . In contrast , in sequential training each agent overfits to the static agent at the level below , leading to arbitrary handshakes and brittle conventions . Furthermore , training a best response to the entire hierarchy improves the final ZSC performance and robustness in ad-hoc settings . This is intuitive since the BR can carry out on-the-fly adaptation in the ZSC setting . Our results show that the exact graph-structure used , which were similarly studied in [ 13 ] , and the type of training regime ( synchronous vs sequential ) can have a major impact on the final outcome when adapting ideas from the cognitive hierarchy literature to the deep MARL setting . We hope our findings will encourage other practitioners to seek inspiration in the game theory literature and to scale those ideas to high dimensional problems , even when there is precedent of unsuccessful attempts in prior work . 2 Related Work . A significant portion of research in MARL has been focused on creating agents that do well in fully cooperative , partially observable settings . A standard approach is through variations of self-play ( SP ) methods [ 9 , 8 , 12 , 24 , 15 ] ; however , as shown in [ 18 ] generally optimal SP agents learn highly arbitrary policies , which are incompatible with independently trained agents . Clearly , test-time coordination with other , independently trained agents including humans is an important requirement for AI agents that is not captured in the SP problem setting . To address this , [ 18 ] introduced the zero-shot coordination ( ZSC ) setting , where the explicit goal is to develop learning algorithms that allow independently trained agents to collaborate at test time . Another recent area of work trains an RL agent separately , and then evaluate its performance in a new group of AI agents or humans assuming access to a small amount of test-time data [ 24 , 33 ] . These methods build SP policies that are compatible with the test time agents by guiding the learning to the nearest equilibria [ 24 , 33 ] . Other methods use human data to build a human model and then train an approximate best response to this human model , making it compatible with human play [ 5 ] . While this presents a promising near-term approach for learning human-like policies in specific settings where we have enough data , it does not enable us to understand the fundamental principles of coordination that lead to this behavior in the first place , which is the goal of the ZSC setting . Our paper shows that even relatively simple training methods can lead to drastically improved ZSC performance when combined with modern engineering best practices and , importantly , that the performance gains directly translate into better coordination with a human-like proxy and ad-hoc teamplay ( without requiring human data in the training process ) . Our approach for scaling KLR to Hanabi relies heavily on the parallel training of all of the different levels , where each level is trained on one machine and the models are exchanged periodically through a central server . This architecture draws inspiration from population based training ( PBT ) , which was first popularized for hyperparameter turning [ 20 ] and then applied in the multi-agent context to train more robust policies in two player zero-sum team settings [ 19 ] . PBT has also been used to obtain better self-play policies in Hanabi , both in [ 12 ] and [ 1 ] . In contrast to prior work , we do not use PBT to avoid local optima or train less exploitable agents but instead leverage this framework to implement a KLR and a best response to this KLR that is geared towards ZSC and coordination with human-like proxies . There are a few other methods directly addressing the ZSC framework . The first , other-play ( OP ) [ 18 ] requires access to the ground truth symmetries of the problem setting and then learns policies that avoid breaking these symmetries during training . OP has previously been applied to Hanabi and KLR compares favorably to the OP results ( see Section 4 ) . We also note , that KLR does not require access to the symmetries and can be applied in settings where no symmetries are present . The next method , Ridge Rider ( RR ) [ 27 ] uses the connection between symmetries in an environment and repeated eigenvalues of the Hessian , to solve ZSC problems . Like KLR , RR does not require prior ground truth access . However , unlike KLR , RR is extremely computationally expensive and has not been scaled to large scale RL problems . Life-Long Learning ( LLL ) has been studied for ZSC [ 26 ] . However , LLL requires access to a pool of pre-trained agents , and in this case they had access to symmetries , whereas our method never required access to such symmetries and our method compares favorably in the ZSC setting . Lastly , Off-Belief Learning ( OBL ) [ 17 ] has been shown to provide well-grounded play in hanabi and strong results in the ZSC setting , but requires simulator access to train . We note that KLR doesn ’ t require simulator access and also matches or even outperforms OBL on various metrics . 3 Background . 3.1 Dec-POMDPs . This work considers a class of problems , Dec-POMDPs [ 2 ] , where N agents interact in a partially observable environment . The partial observability implies that every agent i ∈ { 1 , · · · , N } has an observation oit = O ( st ) obtained from via the observation function O from the underlying state st ∈ S. In our setting , at each timestep t the acting agent i samples an action uit from policy πi , uit ∼ πiθ ( ui|τ it ) , where θ are the weights of the neural networks , and all other agents take no-op actions . Here we use action-observation histories ( AOH ) which we denote as τ i = { oi0 , ui0 , r1 · · · , rT−1 , oiT } , where T is the length of the trajectory , and rt is the common reward at timestep t defined by the reward function R ( s , u ) . The goal of the agents is to maximize the total reward Eτ∼P ( τ |s , u ) [ Rt ( τ ) ] ; here we consider Rt ( τ ) to be the discounted sum of rewards , i.e . Rt ( τ ) = ∑∞ t′=t γ t′−trt′ , where γ is the discount factor . Additionally , the environments in this work are turn-based and bounded in length at tmax . 3.2 Deep Multi-Agent Reinforcement Learning . Deep reinforcement learning has been applied to a multitude of multi-agent learning problems with great success . Cooperative MARL is readily addressed with extensions of Deep Q-learning [ 25 ] , where the Q-function is parameterized by neural networks to learn to predict the expected return based on the current state s and action u , Q ( τt , ut ) = Eτ∼P ( τ |s , u ) Rt ( τ ) . Our work also builds off of state of the state of the art algorithm Recurrent Replay Distributed Deep Q-Network ( R2D2 ) [ 21 ] . R2D2 also incorporates other recent advancements such as using a dueling network architecture [ 36 ] , prioritized replay experience [ 29 ] , and double DQN learning [ 34 ] . Additionally , we use a similar architecture as the one proposed in [ 14 ] and run many environments in parallel , each of which has actors with varying exploration rates that add to a centralized replay buffer . The simplest way to adapt deep Q-learning to the Dec-POMDP setting is through Independent Q-learning ( IQL ) as proposed by [ 32 ] . In IQL , every agent individually estimates the total return and treats other agents as part of the environment . There are other methods that explicitly account for the multi-agent structure by taking advantage of the centralized training with decentralized control regime [ 31 , 28 ] . However , since our work is based on learning best responses , here we only consider IQL .

Summary: An important problem in multi-agent RL is how to train agents to cooperate and coordinate well with humans. One approach has been to train agents that can coordinate well with themselves, using self-play. Recently more attention has been paid to the problem of coordinating well with independent training runs: referred to as Zero-Shot Coordination, as an easier proxy for coordinating with particular other policies, such as humans. A benchmark game for this has been Hanabi, which is fully cooperative and, for strong play, requires sending signals whose meanings require multi-step chains of reasoning to deduce, resulting in complex conventions. One way to understand these chains has been via the so-called “cognitive hierarchy” (CH) from behavioural game theory, that was invented to characterize human non-equilibrium play in some settings: it’s a hierarchy in which higher levels select optimal strategies assuming other players are following strategies from lower levels.  Past work has tried to tackle ZSC in Hanabi using this idea directly (training a chain of policies that each best-respond to the preceding one), but the resulting training process did not manage to avoid arbitrary conventions, hurting ZSC performance. Recent work has explored more sophisticated ways of rooting out arbitrary conventions, eg by using carefully grounded priors and an expensive simulator to reason about the state distribution, or by ensuring that policies don’t learn to rely on a particular way of breaking symmetries in the game (via data augmentation).  This paper revisits the chain-of-best-responders method, and modifies the training schema by training each policy in the chain simultaneously. This results in policies that have near-SOTA ZSC performance; behavioural analysis is shown to support the explanation that this is because the moving-target opponents strongly regularize away the overfitting that otherwise occurs when training best-responders against fixed opponents. With another tweak (learning a best-response to a mixture of the chain of policies), a policy is trained that is even stronger in ZSC and self-play, and achieves SOTA in playing alongside a human-play proxy policy (as well as consistently good performance in some other ad-hoc teams). The presented method seems to be the first to achieve good ZSC in Hanabi without requiring bespoke components such as a model of all of the symmetries or a simulator to use for deriving belief distributions, and is therefore an important contribution that’s likely to be applicable to many other settings for learning coordination and strategic behaviour.