Zero-Shot Coordination via Semantic Relationships Between Actions and Observations

1 INTRODUCTION . Successful collaboration between agents requires coordination ( Tomasello et al. , 2005 ; Misyak et al. , 2014 ; Kleiman-Weiner et al. , 2016 ) , which is challenging because coordinated strategies can be arbitrary ( Lewis , 1969 ; Young , 1993 ; Lerer & Peysakhovich , 2018 ) . A priori , one can neither deduce which side of the road to drive , nor which utterance to use to refer to ♥ ( Pal et al. , 2020 ) . In these cases coordination can arise from actors best responding to what others are already doing i.e. , following a convention . For example , Americans drive on the right side of the road and say “ heart ” to refer to ♥ while Japanese drive on the left and say “ shinzo ” . Yet in many situations prior conventions may not be available and agents may be faced with entirely novel situations or partners . In this work we study ways that agents may learn to leverage abstract relations between observations and actions to coordinate with agents they have had no experience interacting with before . To illustrate , consider the following situations where people can figure out how to coordinate without prior experienced or shared conventions . Imagine a store that sells strawberries and blueberries . You want to buy strawberries but you don ’ t share any common language with the clerk . You are however wearing a red hat and you wave the hat at the clerk to hint that the strawberries are what you want . The clerk has two baskets of strawberries remaining , and so you raise a single finger to indicate that you only want one of the baskets . The clerk produces a paper and plastic bag and you point to the paper bag to indicate that you want the paper one . These examples are so simple that they seem obvious : the red hat matches the colors of the strawberries , the number of fingers matches the number of baskets you want , and you extend a finger in the direction of the desired packaging ( Grice , 1975 ) . While obvious to people , who rely on a theory-of-mind in understanding others , we will show that these inferences remain a challenge for multi-agent reinforcement learning agents . Less obvious examples are common in the cognitive science literature . Consider the shapes in Fig . 1 . When asked to assign the names “ Boubo ” and “ Kiki ” to the two shapes people name the jagged object “ Kiki ” and the curvy object “ Boubo ” ( Köhler , 1929 ) . This finding is robust across different linguistic communities and cultures and is even found in young children ( Maurer et al. , 2006 ) . The causal explanation is that people match a “ jaggedness ” -feature and “ curvey ” -feature in both the visual and auditory data . Across the above these cases , there seem to be a generalized mechanism for mapping the features of the persons action with the features of the desired action . All are examples of where in the absence of norms or conventions , people minimize the distance between features when making a choice . This basic form of zero-shot coordination predates verbal behavior ( Tomasello et al. , 2007 ) and this capability has been hypothesized as a key predecessor to more sophisticated language development and acquisition ( Tomasello et al. , 2005 ) . Modeling these capacities is key for building machines that can robustly coordinate with other agents and with people ( Kleiman-Weiner et al. , 2016 ; Dafoe et al. , 2020 ) . However , as we will show , naively training reinforcement learning ( RL ) agents with self-play fails to learn to coordinate even in these obvious ways . Instead , they develop arbitrary private languages that are uninterpretable to both the same models trained with a different random seed as well as to human partners ( Hu et al. , 2020 ) . For instance in the examples above , they would be equally likely to wave a red-hat to hint they want strawberries as they would to indicate that they want blueberries . These problems also emerge at scale in the decentralized partially observable Markov decision process ( Dec-POMDP ) benchmark Hanabi ( Bard et al. , 2019 ) . When agents are trained with self-play using standard architectures they do not develop strategies that take into account the correspondence between the features of the actions ( colored and numbered cards ) and the observation of the game state ( other colored and numbered cards ) . Unfortunately , describing the kind of abstract knowledge that these agents lack in closed form is challenging . Rather than attempting to do so , we take a learning-based approach . Our aim is to build an agent with the capacity to develop these kinds of abstract correspondences during self-play such that they can robustly succeed during cross-play or during play with humans . Our contributions are as follows : ( 1 ) We extend Dec-POMDPs to allow actions and observations to be represented using shared features and develop a novel human-interpretable environment for studying coordination in this setting . ( 2 ) We evaluate the role of neural network ( NN ) architectures including feedforward , recurrent , and attention mechanisms on both cross-play generalization and ability to create human-interpretable conventions . ( 3 ) We show that an attention architecture which takes both the action and observations as input allows the agent to exploit the semantic relationships for coordination , resulting in strong cross-play and human compatible policies that outperform baseline ZSC methods . This model also demonstrates sophisticated coordination patterns that exploit mutual exclusivity and implicature , two well-known phenomena studied in cognitive science ( Markman & Wachtel , 1988 ; Grice , 1975 ) . 2 RELATED WORK . Cooperative MARL . The standard method for training multi-agent reinforcement learning ( MARL ) agents in fully cooperative , partially observable settings is self-play ( SP ) . However , the failure of SP policies in cross-play ( XP ) has been recently explored . Carroll et al . ( 2019 ) used grid-world MDPs to show that both SP and population-based training fail when paired with human collaborators . In Hanabi , agents trained via SP develop conventions that fail to generalize to independently trained agents from the same algorithm with different random seeds ( Bard et al. , 2019 ; Hu et al. , 2020 ) . Zero-Shot Coordination . To address this issue , Hu et al . ( 2020 ) formally introduced the zeroshot coordination ( ZSC ) framework , where the goal is to maximize the XP returns of independently trained agents , allowing them to coordinated at test time . Thus formulated , ZSC is an alternative to ad-hoc teamplay , a framework for measuring coordinated team success when faced with players with unknown behavior ( Stone et al. , 2010 ; Barrett et al. , 2011 ) , which can be formalized as playing a best response to a distribution of a-priori known agents . A few methods have attempted to address the ZSC framework . Other-Play ( OP ) ( Hu et al. , 2020 ) exploits the symmetries in a given Dec-POMDP to prevent agents from learning equivalent but mutually incompatible policies . OP prohibits arbitrary tie-breaking , thereby preventing equivalent conventions from forming . However , OP requires experimenter-coded symmetries , and discovering such symmetries is computationally challenging . In contrast , our learning based approach requires no experimenter-coding . Another recent method , Off-Belief Learning ( OBL ) ( Hu et al. , 2021 ) , regularizes agents ’ ability to make inferences based on the behavior of others . Compared to prior work on Hanabi where SP scores were high but XP scores were near chance , both of OP and OBL drastically improve XP scores and show promising preliminary results in play with people . However , neither of these algorithms can exploit the correspondences between features of an action and the observation of the state as we show in this work , unless this falls out of the environment dynamics . Attention for Modeling Input-Output Relationships . Attention ( Vaswani et al. , 2017 ; Bahdanau et al. , 2015 ; Xu et al. , 2016 ) is an important tool for large sequence models , and exploiting semantic relationships between inputs and outputs via an attention-based model has been studied in the deep learning literature . In natural language processing , such an idea is commonly applied to question answering models ( dos Santos et al. , 2016 ; Tan et al. , 2016 ; Yang et al. , 2016 ) . For instance , Yang et al . ( 2016 ) form a matrix that represents the semantic matching information of term pairs from a question and answer pair , and then use dot-product attention to model question term importance . For regression tasks , Kim et al . ( 2019 ) proposed Attentive Neural Processes ( ANP ) that use dot-product attention to allow each input location to attend to the relevant context points for the prediction , and applied ANP to vision problems . However , to our knowledge , we are the first to apply attention to exploit shared features of actions and observations in a Dec-POMDP setting for coordination . Human Coordination . Our work is also inspired by how humans coordinate in cooperative settings . Theory-of-mind , the mechanism people use to infer intentions from the actions of others , plays a key role in structuring coordination ( Wu et al. , 2021 ; Shum et al. , 2019 ) . In particular , Rational Speech Acts ( RSA ) is a influential model of pragmatic implicature ( Frank & Goodman , 2012 ; Goodman & Stuhlmüller , 2013 ) . At the heart of these approaches are probabilistic representations of belief which allow for the modeling of uncertainty and recursive reasoning about each others beliefs , enabling higher-order mental state inferences . This recursive reasoning step also underlies the cognitive hierarchy and level-K reasoning models , and is useful for explaining certain focal points ( Camerer , 2011 ; Stahl & Wilson , 1995 ; Camerer et al. , 2004 ) . However , constructing recursive models of players beliefs and behavior is computationally expensive as each agent must construct an exponentially growing number of models of each agent modeling each other agent . As a result , recursive models are often limited to one or two levels of recursion . Furthermore , none of these approaches can by itself take advantage of the shared features across actions and observations . 3 BACKGROUND . Dec-POMDPs . We use decentralized partially observable Markov decision processes ( DecPOMDPs ) to formalize our setting ( Nair et al. , 2003 ) . In a Dec-POMDP , each player i observes the underlying state s partially through an observation function Ωi ( s ) ∈ Oi , and takes action ai ∈ Ai . Players receive a common reward R ( s , a ) and the state follows the transition function T ( s , a ) . The historical trajectory is denoted as τ = ( s1 , a1 , . . . , at−1 , st ) . Player i ’ s action-observation history ( AOH ) is denoted as ( Ωi ( s1 ) , ai1 , . . . , a i t−1 , Ω i ( st ) ) . The policy for player i takes as input an AOH and outputs a distribution over actions , denoted by πi ( ai | τ it ) . The joint policy is denoted by π. Dot-Product Attention . Given a sequence of input vectors ( x1 , ... , xm ) , dot-product attention uses three weight matrices ( Q , K , V ) to obtain triples ( Qxi , Kxi , V xi ) for each i ∈ { 1 , . . . , m } , called query vectors , key vectors , and value vectors . We abbreviate these as ( qi , ki , vi ) . Next , for each i , j , dot-product attention computes logits using dot products qi · kj . These logits are in turn used to compute an output matrix [ softmax ( qi · k1/ √ m , . . . , qi · km/ √ m ) · vj ] i , j . We denote this output matrix as Attention ( x1 , . . . , xm ) .

Summary: The paper exploits semantic relationships between features of observations and an action for zero-shot coordination in multi-agent reinforcement learning. It extends decentralized partially observable Markov decision process (Dec-POMDP) by representing observations and actions with shared features. Technically it uses attention mechanism of deep learning to exploit the semantic relationships between observations and actions. Besides, the paper develops a novel human interpretable environment for analysis.