Zero-Shot Coordination via Semantic Relationships Between Actions and Observations

1 INTRODUCTION . Successful collaboration between agents requires coordination ( Tomasello et al. , 2005 ; Misyak et al. , 2014 ; Kleiman-Weiner et al. , 2016 ) , which is challenging because coordinated strategies can be arbitrary ( Lewis , 1969 ; Young , 1993 ; Lerer & Peysakhovich , 2018 ) . A priori , one can neither deduce which side of the road to drive , nor which utterance to use to refer to ♥ ( Pal et al. , 2020 ) . In these cases coordination can arise from actors best responding to what others are already doing i.e. , following a convention . For example , Americans drive on the right side of the road and say “ heart ” to refer to ♥ while Japanese drive on the left and say “ shinzo ” . Yet in many situations prior conventions may not be available and agents may be faced with entirely novel situations or partners . In this work we study ways that agents may learn to leverage abstract relations between observations and actions to coordinate with agents they have had no experience interacting with before . To illustrate , consider the following situations where people can figure out how to coordinate without prior experienced or shared conventions . Imagine a store that sells strawberries and blueberries . You want to buy strawberries but you don ’ t share any common language with the clerk . You are however wearing a red hat and you wave the hat at the clerk to hint that the strawberries are what you want . The clerk has two baskets of strawberries remaining , and so you raise a single finger to indicate that you only want one of the baskets . The clerk produces a paper and plastic bag and you point to the paper bag to indicate that you want the paper one . These examples are so simple that they seem obvious : the red hat matches the colors of the strawberries , the number of fingers matches the number of baskets you want , and you extend a finger in the direction of the desired packaging ( Grice , 1975 ) . While obvious to people , who rely on a theory-of-mind in understanding others , we will show that these inferences remain a challenge for multi-agent reinforcement learning agents . Less obvious examples are common in the cognitive science literature . Consider the shapes in Fig . 1 . When asked to assign the names “ Boubo ” and “ Kiki ” to the two shapes people name the jagged object “ Kiki ” and the curvy object “ Boubo ” ( Köhler , 1929 ) . This finding is robust across different linguistic communities and cultures and is even found in young children ( Maurer et al. , 2006 ) . The causal explanation is that people match a “ jaggedness ” -feature and “ curvey ” -feature in both the visual and auditory data . Across the above these cases , there seem to be a generalized mechanism for mapping the features of the persons action with the features of the desired action . All are examples of where in the absence of norms or conventions , people minimize the distance between features when making a choice . This basic form of zero-shot coordination predates verbal behavior ( Tomasello et al. , 2007 ) and this capability has been hypothesized as a key predecessor to more sophisticated language development and acquisition ( Tomasello et al. , 2005 ) . Modeling these capacities is key for building machines that can robustly coordinate with other agents and with people ( Kleiman-Weiner et al. , 2016 ; Dafoe et al. , 2020 ) . However , as we will show , naively training reinforcement learning ( RL ) agents with self-play fails to learn to coordinate even in these obvious ways . Instead , they develop arbitrary private languages that are uninterpretable to both the same models trained with a different random seed as well as to human partners ( Hu et al. , 2020 ) . For instance in the examples above , they would be equally likely to wave a red-hat to hint they want strawberries as they would to indicate that they want blueberries . These problems also emerge at scale in the decentralized partially observable Markov decision process ( Dec-POMDP ) benchmark Hanabi ( Bard et al. , 2019 ) . When agents are trained with self-play using standard architectures they do not develop strategies that take into account the correspondence between the features of the actions ( colored and numbered cards ) and the observation of the game state ( other colored and numbered cards ) . Unfortunately , describing the kind of abstract knowledge that these agents lack in closed form is challenging . Rather than attempting to do so , we take a learning-based approach . Our aim is to build an agent with the capacity to develop these kinds of abstract correspondences during self-play such that they can robustly succeed during cross-play or during play with humans . Our contributions are as follows : ( 1 ) We extend Dec-POMDPs to allow actions and observations to be represented using shared features and develop a novel human-interpretable environment for studying coordination in this setting . ( 2 ) We evaluate the role of neural network ( NN ) architectures including feedforward , recurrent , and attention mechanisms on both cross-play generalization and ability to create human-interpretable conventions . ( 3 ) We show that an attention architecture which takes both the action and observations as input allows the agent to exploit the semantic relationships for coordination , resulting in strong cross-play and human compatible policies that outperform baseline ZSC methods . This model also demonstrates sophisticated coordination patterns that exploit mutual exclusivity and implicature , two well-known phenomena studied in cognitive science ( Markman & Wachtel , 1988 ; Grice , 1975 ) . 2 RELATED WORK . Cooperative MARL . The standard method for training multi-agent reinforcement learning ( MARL ) agents in fully cooperative , partially observable settings is self-play ( SP ) . However , the failure of SP policies in cross-play ( XP ) has been recently explored . Carroll et al . ( 2019 ) used grid-world MDPs to show that both SP and population-based training fail when paired with human collaborators . In Hanabi , agents trained via SP develop conventions that fail to generalize to independently trained agents from the same algorithm with different random seeds ( Bard et al. , 2019 ; Hu et al. , 2020 ) . Zero-Shot Coordination . To address this issue , Hu et al . ( 2020 ) formally introduced the zeroshot coordination ( ZSC ) framework , where the goal is to maximize the XP returns of independently trained agents , allowing them to coordinated at test time . Thus formulated , ZSC is an alternative to ad-hoc teamplay , a framework for measuring coordinated team success when faced with players with unknown behavior ( Stone et al. , 2010 ; Barrett et al. , 2011 ) , which can be formalized as playing a best response to a distribution of a-priori known agents . A few methods have attempted to address the ZSC framework . Other-Play ( OP ) ( Hu et al. , 2020 ) exploits the symmetries in a given Dec-POMDP to prevent agents from learning equivalent but mutually incompatible policies . OP prohibits arbitrary tie-breaking , thereby preventing equivalent conventions from forming . However , OP requires experimenter-coded symmetries , and discovering such symmetries is computationally challenging . In contrast , our learning based approach requires no experimenter-coding . Another recent method , Off-Belief Learning ( OBL ) ( Hu et al. , 2021 ) , regularizes agents ’ ability to make inferences based on the behavior of others . Compared to prior work on Hanabi where SP scores were high but XP scores were near chance , both of OP and OBL drastically improve XP scores and show promising preliminary results in play with people . However , neither of these algorithms can exploit the correspondences between features of an action and the observation of the state as we show in this work , unless this falls out of the environment dynamics . Attention for Modeling Input-Output Relationships . Attention ( Vaswani et al. , 2017 ; Bahdanau et al. , 2015 ; Xu et al. , 2016 ) is an important tool for large sequence models , and exploiting semantic relationships between inputs and outputs via an attention-based model has been studied in the deep learning literature . In natural language processing , such an idea is commonly applied to question answering models ( dos Santos et al. , 2016 ; Tan et al. , 2016 ; Yang et al. , 2016 ) . For instance , Yang et al . ( 2016 ) form a matrix that represents the semantic matching information of term pairs from a question and answer pair , and then use dot-product attention to model question term importance . For regression tasks , Kim et al . ( 2019 ) proposed Attentive Neural Processes ( ANP ) that use dot-product attention to allow each input location to attend to the relevant context points for the prediction , and applied ANP to vision problems . However , to our knowledge , we are the first to apply attention to exploit shared features of actions and observations in a Dec-POMDP setting for coordination . Human Coordination . Our work is also inspired by how humans coordinate in cooperative settings . Theory-of-mind , the mechanism people use to infer intentions from the actions of others , plays a key role in structuring coordination ( Wu et al. , 2021 ; Shum et al. , 2019 ) . In particular , Rational Speech Acts ( RSA ) is a influential model of pragmatic implicature ( Frank & Goodman , 2012 ; Goodman & Stuhlmüller , 2013 ) . At the heart of these approaches are probabilistic representations of belief which allow for the modeling of uncertainty and recursive reasoning about each others beliefs , enabling higher-order mental state inferences . This recursive reasoning step also underlies the cognitive hierarchy and level-K reasoning models , and is useful for explaining certain focal points ( Camerer , 2011 ; Stahl & Wilson , 1995 ; Camerer et al. , 2004 ) . However , constructing recursive models of players beliefs and behavior is computationally expensive as each agent must construct an exponentially growing number of models of each agent modeling each other agent . As a result , recursive models are often limited to one or two levels of recursion . Furthermore , none of these approaches can by itself take advantage of the shared features across actions and observations . 3 BACKGROUND . Dec-POMDPs . We use decentralized partially observable Markov decision processes ( DecPOMDPs ) to formalize our setting ( Nair et al. , 2003 ) . In a Dec-POMDP , each player i observes the underlying state s partially through an observation function Ωi ( s ) ∈ Oi , and takes action ai ∈ Ai . Players receive a common reward R ( s , a ) and the state follows the transition function T ( s , a ) . The historical trajectory is denoted as τ = ( s1 , a1 , . . . , at−1 , st ) . Player i ’ s action-observation history ( AOH ) is denoted as ( Ωi ( s1 ) , ai1 , . . . , a i t−1 , Ω i ( st ) ) . The policy for player i takes as input an AOH and outputs a distribution over actions , denoted by πi ( ai | τ it ) . The joint policy is denoted by π. Dot-Product Attention . Given a sequence of input vectors ( x1 , ... , xm ) , dot-product attention uses three weight matrices ( Q , K , V ) to obtain triples ( Qxi , Kxi , V xi ) for each i ∈ { 1 , . . . , m } , called query vectors , key vectors , and value vectors . We abbreviate these as ( qi , ki , vi ) . Next , for each i , j , dot-product attention computes logits using dot products qi · kj . These logits are in turn used to compute an output matrix [ softmax ( qi · k1/ √ m , . . . , qi · km/ √ m ) · vj ] i , j . We denote this output matrix as Attention ( x1 , . . . , xm ) .

Summary: This paper explores several architectures for reinforcement-learning in the context of a newly proposed card game (Hinter-Guesser). A number of the architectures seem to have poor performance on the game. Moreover, whatever they learn is not compatible with what other learners learn, so that they cannot mutually understand Hints & Guesses. The stated aim is: "to build an agent with the capacity to develop these kinds of abstract correspondences during self-play such that they can robustly succeed during cross-play or during play with people." The architectures generally succeed (70-90%) at playing against themselves (with the shared learned policy) but perform badly when playing with independently trained agents, except when trained with the desired action as part of the input (unclear how this can be tested on unknown cases when 'answer' is given as part of the inputs).