Learning Intuitive Policies Using Action Features
Authors: Mingwei Ma
Jizhou Liu
Samuel Sokota
Max Kleiman-Weiner
Jakob Nicolaus Foerster
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-agent coordination
attention
inductive bias

Abstract: An unaddressed challenge in multi-agent coordination is to enable AI agents to exploit the semantic relationships between the features of actions and the features of observations. Humans take advantage of these relationships in highly intuitive ways. For instance, in the absence of a shared language, we might point to the object we desire or hold up our fingers to indicate how many objects we want. To address this challenge, we investigate the effect of network architecture on the propensity of learning algorithms to exploit these semantic relationships. Across a procedurally generated coordination task, we find that attention-based architectures that jointly process a featurized representation of observations and actions have a better inductive bias for zero-shot coordination. Through fine-grained evaluation and scenario analysis, we show that the resulting policies are human-interpretable. Moreover, such agents coordinate with people without training on any human data. 

Under review as a conference paper at ICLR 2023
LEARNING INTUITIVE POLICIES USING ACTION FEA- TURES
Anonymous authors Paper under double-blind review
ABSTRACT
An unaddressed challenge in multi-agent coordination is to enable Al agents to exploit the semantic relationships between the features of actions and the features of observations. Humans take advantage of these relationships in highly intuitive ways. For instance, in the absence of shared language, we might point to the object we desire or hold up our fingers to indicate how many objects we want. To address this challenge, we investigate the effect of network architecture on the propensity of learning algorithms exploit these semantic relationships. Across a procedurally generated coordination task, we find that attention -based architec- tures that jointly process featurized representation of observations and actions have a better inductive bias for learning intuitive policies. Through fine-grained evaluation and scenario analysis, show that the resulting policies are human- interpretable. Moreover, such agents coordinate with people without training on any human data.
INTRODUCTION
Successful collaboration between agents requires coordination (Tomasello et al. 2005 Misyak al., 2014 Kleiman- Weiner et al.) 2016). which is challenging because coordinated strategies can be arbitrary (Lewis) 1969 Young 1993 Lerer & Peysakhovich 2018). A priori, one can neither which side of the road to drive, nor what utterance to use to refer to V (Pal et 20200). In these cases coordination can arise from actors best responding to what others are already doing doing-i.e.. following convention. For example, Americans drive on the right side of the road and say "heart" to refer to while Japanese drive on the left and say "shinzo". Yet in many situations prior conventions may not be available and agents may be faced with entirely novel situations or partners. In this work, we study ways that agents may learn to leverage semantic relations between observations and actions to coordinate with agents they have had experience interacting with before.
Consider the shapes Fig. I When asked to assign the names "Bouba" and "Kiki" to the two shapes, people name the jagged object "Kiki" and the curvy object "Bouba" (Köhler) 1929). This finding is robust across different linguistic com- munities and cultures and is even found in young children (Maurer et al., 2006). The causal explanation is that people match a "jaggedness" -fetuure and visual and auditory data. Across the above cases, there seems to be a generalized mechanism for mapping the features of the person's action with the features of the action that the person desires the other agent to take. In the absence of norms or conventions, people may minimize the distance between these features when making a choice. This basic form of feature utilization in humans predates verbal behavior (Tomasello et al. (007). and this capability has been hypothesized as a key predecessor to more sophisticated language development and acquisition (Tomasello et al.) 2005). Modeling these capacities is key for building machines that can robustly coordinate with other agents and with people (Kleiman Weiner et al. 2016 Dafoe et al. 2020).
in both the
Figure 1: The "Bouba" (right) and "Kiki" (left) effect.
Might this general mechanism emerge through multi- agent reinforcement learning across range of tasks? As we will show, reinforcement learning agents naively trained with self-play fail to learn to coordinate even in these obvious ways. Instead, they develop arbitrary private languages that

---

Under review as a conference paper at ICLR 2023
uninterpretable to both the same models trained with different random seed and to human partners (Hu et al.] 2020). For instance, in the examples above, they would be equally likely to wave a red-hat to hint they want strawberries as they would to indicate that they want blueberries.
Unfortunately, developing an inductive bias that might take into account these correspondences is not straightforward because describing the kind of abstract knowledge that these agents lack in closed form is challenging. Rather than attempting to do so, we take a learning-baser approach. Our aim is to build an agent with the capacity to develop these kinds of abstract correspondences during self-play, such that it can robustly succeed during cross play, a setting in which a model is paired with a partner (human or AI) that it did not train with.
Toward this end, we extend the Dec-POMDP formalism to allow actions and observations to be rep- resented using shared features and design human- interprtantt environment for studying coordi- nation with these enrichments. Using this formalism, we examine the inductive bias of a collection of five network architectures- two feedforward based and three attention based-ij in procedurally generated coordination tasks. Our main contribution is finding that a self-attention architecture that takes both the action and observations as input has a strong inductive bias toward using the relation- ship between actions and observations in intuitive ways. This inductive bias manifests in: High intra-algorithm pros--ply scores compared to both other architecures and to algorithms specifically designed to maximize angra-algorithm cross play: 2) Sophisticated human-lit coordination patterss that exploit mutual exclusivity and implicature two well-known phenomena studied in cognitive science (Markman & Wachtell 1988, Grice [975): 3) Human-level performance ad-hoc coordi- nating with humans. We hypothesize that the success of this architecture can be attributed to the fact that it processes observation features and action features using the same weights. Our finding sug- gests that this kind of attention architecture is the most sensible starting point for learning intuitive policies in settings in which action features play an important role.
2 BACKGROUND AND RELATED WORK
Dec-POMDPs. We start with decentralized partially observable Markov decision processes (Dec- POMDPs) to formalize our setting (Nair et al. 2003). In a Dec-POMDP, each player receives an observation D (2)(s) Oi generated by the underlying state s, and takes action at E AA.. Players receive a common reward R(s, a) and the state transitions according to the function T(s, a). The historical trajectory is t = (s1, a1 Q,-1, st). Player i's action-observation history (AOH) is denoted as T. (5* ), ........ (f The policy for player takes input an AOH and outputs distribution over actions, denoted by r'aa' (T)). The joint policy is denoted by π.
MARL and Coordination. The standard paradigm for training multi-agent reinforcement learning (MARL) agents in Dec-POMDPs is self play (SP). However, the failure of such policies to achieve high reward when evaluated in cross play (XP) is well documented. Carroll et al. (2019) used grid-world MDPs to show that both SP and population-basee training fail when paired with human collaborators. |Bard et al.] (2019): Hu et al. (2020) showed that agents perform significantly worse when paired with independently trained agents than they do at training time in Hanabi, even though the agents are trained under identical circumstances. This drop in XP performance directly results in poor human-AI coordination, as shown in (Hu al., 2020). Lanctot et al. (011)) found similar qualitative XP results in a partially-cooperative laser tag game.
To address this issue, Hu al. (2020) introduced setting in which the goal is maximize the XP turns of independently trained agents using the same algorithm. We call this setting intra-algorithmm cross play (intra-AXP). Hu et al. (2020) argue that high intra-AXP is necessary for successful co- ordination with humans: If agents trained from independent runs or random seeds using the same algorithm cannot coordinate well with each other, is unlikely they will be able to coordinate with agents with different model architectures, not to mention humans. However, while there has been recent progress in developing algorithms that achieve high intra- AXP scores in some settings Hu et al. (2000 2021), ), this success does not carry over to settings in which the correspondence between actions and observations is important for coordination, as we will show.
Beyond performing well in intra-AXP, more ambitious goal is to perform well with agents that are externally determined, such as humans, and not observed during training time. This setting has been referred to both as ad-hoc coordination (Stone al. 2010 Barrett al. 2011) and zero-shot
2

---

Under review as a conference paper at ICLR 2023
coordination (Hu et al., 2020; Strouse al. 2021). However, both terms carry some ambiguity. as ad-hoc coordination is sometimes evaluated in a setting in which the externally determined partners known at training time (Stone et al.] 2010 and zero-shot coordination is sometimes used to refer to the setting in which the goal is to maximize intra-AXP (Hu et al. 2020 We use ad-hoc coordination to refer to this setting.
Dot-Product Attention, As we will see in our experiments, one way to leverage the correspon- dences between action features and observation features is by using attention mechanisms (Vaswani et al., 2017; Bahdanau et al. 2015 Xu et al. 2016). Given a set of input vectors , Tm self- uses three weight matrices (Q. K. to obtain triples (Qr,, Kx,, Vx;) for each called query vectors, key vectors, and value vectors. We abbreviate these as
i E (4, Ai, 11). Next, for each J. dot-product attention computes logits using dot products d; -ky. These logits are in turn used to compute an output matrix [softmax /tm........ Cross-attention similar, accepts inputs that are partitioned into disjoint blocks (yl,.... .M). and (II, Im). and only applies Q to y and only applied K, V to H. We refer to this as CA(YI..... Mn: 11, T.,)
km/ km/Vm) u1 uyle..-
We denote this output matrix as SA(r1,....,2
Attention for Input-Output Relationships. Exploiting semantic relationships between inputs and outputs via an attention aateetion-bbsed model has been studied in the deep learning literature. In natural language processing, such an idea is commonly used in question answering models (dos Santos et al. 2016 Tan et a 2016 Yang et al.] 2016). For instance, Yang et al. 2016) form a matrix that represents the semantic matching information of term pairs from question and answer pair, and then use dot-product attention to model question term importance. For regression tasks, Kim et al. (2019) proposed attentive neural processes (ANP) that use dot-product attention allow each input location to attend to the relevant context points for the prediction, and applied ANP to vision problems.
Human Coordination. Our work is also inspired by how humans coordinate in cooperative settings. Theory-of-mind, the mechanism people use to infer intentions from the actions of others, plays key role structuring coordination (Wu al.) 202T, Shum al. 2019). In particular. rational speech acts (RSA) is an influential model of pragmatic implicature (Frank& & Goodman 2012 Goodman & Stuhlmüller 2013). At the heart of these approaches are probabilistic representations of beliefs that allow for modeling uncertainty and recursive reasoning about the beliefs of others, enabling higher- order mental state inferences. This recursive reasoning step also underlies the cognitive hierarchy and level-K K reasoning models, and is useful for explaining certain focal points (Camerer) 2011 Stahl & Wilson| 1995] Cameeer et al.) 2004). However, constructing recursive models of players' beliefs and behaviors is computationally expensive as each agent must construct an exponentially growing number of models of each agent modeling each other agent. As a result, recursive models are often limited to one or levels of recursion. Furthermore, none of these approaches by itself take advantage of the shared features across actions and observations.
3 DEC-POMDPS WITH SHARED ACTION AND OBSERVATION FEATURES
common to describe the states and observations in Dec-POMDPs using features, e.g. in card games each card has rank and suit. These featurized observations can be exploited by function approximators. In contrast, in typical MARL implementations the actions are merely outputs of the neural network and the models do not take advantage of features of the actions. In the standard representation of Dec-POMDPs, actions are defined solely through their effect on the environment through the reward and the state transition functions. In contrast, in real world environments actions are often grounded and can be described with semantic features that refer to the object they act on, e.g. "I pull the red lever".
To allow action features to be used by MARL agents, we formalize the concept of observation and action features in Dec-POMDPs. We say a Dec-POMDP has observation features if, for at least one player i, we can represent the observation (*(s) as a set of objects (2(ss) = (. Oc), where each object O2 (f1.... fn,) is described by a vector of n, features. Each of these features fú exists in a feature space F.. Similarly, a Dec- POMDP has action features if one can factor the representation of the actions into features at = (fi...,, fm), where each action feature fr E F..
1, b. m. and 8, is the action feature space.
3

---

Under review as a conference paper at ICLR 2023
In some Dec-POMDPs, actions can be described using some of the same features that describe the observations. For example, an agent might observe the "red" light and take the action of pulling the "red" lever where "red" is shared feature between observations and actions. In such cases there a non-empty intersection between F& and F, ("shared action-ohservation features") which may be exploited for coordination. Even in the absence of an exact match, the distance between similar features (e.g., "pink" and "red" and vs. "green" and "red") might also be useful for coordination.
4 THE HINT-GUES GAME
To study Dec-POMDPss with shared call hint-guess. Hint-guess is two-player game where players must coordinate to successfully guess a target card. The game consists of a hinter and a guesser. Both players are given a hand of cards, H1 [C} y.. } for the hinter and H2 {CR,... (1 where Ji E F1 and f2 E F2. Cards in each hand are drawn independently and randomly with replacement, with equal probability for each combination of features. Both hands, H1 and H2, are public information exposed to both players. Before each game, one of the guesser's 's cards, C? is randomly chosen be the target card and its features are revealed the hinter, but not the guesser.
features, we introduce a novel setting that we
} for the guesser. Each card has two features
In the first round, the hinter (who observes H1, H2, CP²) chooses a card in its own hand, which we refer as CJ., to show to the guesser. In the second round, the guesser (who observes H1, H2, C}) guesses which of its cards is the target. Both players receive a common reward r = 1 if the features of the card played match those of the target, otherwise r 0 for both players.
Fig. 2 shows some simple scenarios that probe key dimensions of coordination with 3 and F2 (,, Each of these scenarios has a human-compatible and intuitive solution. The first scenario (exact match) is the most simple-tie the hinter has copy of the target card (2B) so it can simply hint 2B. The next scenario (feature similarity) requires rea- soning about the features under some am- biguity since neither of the cards in the two hands are direct match. In this case, both cards in the hinter's hand share one fea- ture with the guesser. Thus, the human- compatible strategy would to match the cards that share features to each other. The third and fourth examples (labeled implica- tures in Fig. 2 require understanding the action embedded within its context, e.g. what the hinter would have done had the goal been dif- ferent. The third scenario invokes simple kind of implicature: mutual exclusivity. In this scenario, human-compatible intuitive reasoning follows the logic of: "if the target card was 1B, the hinter would choose IB. So that means IB is taken and 3C should correspond share no common feature overlap". The final scenario combines feature similarity and mutual ex- clusivity. These scenarios are particularly interesting as deep learning models often struggle to effectively grapple with mutual exclusivity (Gandhi & Lake 2020
Impli Implicatures
N 2. F1
Exact Ma Match
Feature Simi larity
Matual Exclusivity by
Similarity Exclusivity
2 B
a C
2
?
Hinter
A
B
B
A
C
2 B
3 A
2
1
Guesser
A
Figure 2: Example scenarios in hinf-guess. Shown above are four hand-crafted scenarios that test distinct dimensions important for ZSC. The card in the red dotted lines is a tar- get card and the highlighted yellow card corresponds to numan-compatible choice. The two right scenarios require agents to reason about implicatures, i.c., the intuitive choice has zero fcature overlap with the target card. Model perfor- mance in these scenario types is shown in Table
2A even though they
5 ARCHITECTURES EXAMINED
We consider the following architectures to investigate the effect of policy parameterization on the agents' ability to learn intuitive relationships between actions and observation. For details about the model architectures, see Appendix I
Feedforward Networks (MLPs). The most basic architecture test is standard fully connected feedforward network with ReLU activations. All featurized representation of objects in the obser- vation concatenated and fed into the network, which outputs the estimated Q-value for each
4

---

Under review as a conference paper at ICLR 2023
action. There is no explicit representation of action-observation relationships in this model, since observations are inputs and actions are outputs.
Feedforward Networks with Action as Input (MLP Action In) We also examined variant of the MLP architectures which featurized representation of objects in the observation and one particular action concatenated and fed into the network, which outputs the estimated Q-value for the action that was fed in. Note that this requires separate forward pass to compute the Q-value for each action.
Attention (Attn). We also investigate three attention- based models shown in Fig. [ The first model processes the observations using attention, takes the object-wise mean, and feeds the output into feedfor- ward network, which produces vector with Q-value for each action
deservattions
Mizn
Q4
Q MLP(MMun(SAAOO On)).
There is explicit representation of action- observation relationships in this model, since observations are inputs and actions are outputs.
Cross-Attn
Actione
Cross Attention with Action as Input (CA2I). In the second attention model, the featurized actions are fed into a cross-attention block as queries and the featur- ized observations are fed in as keys and values, which produces vector with Q-value for each action
Reduoe
Self Attnn
Q = MLP(MMcan( (ClcannCA(Aii) (A(AA,................... Am, .I....,O.)))..
Repeat for seliee
Self Attention with Action as Input (SA2I). Lastly, look attention-based architecture similar to Attn, where a featurized action is passed as input to the attention module(s) along with the observations. This outputs single scalar value at a time, the esti- mated Q-value for the specific action being fed into the network
Figure 3: Model architecture for the attention- based models. Top: Attention (Attn). Middle: Cross Attention with Action as Input (CA2I). Bottom: Self Attention with Action as Input (SA2T). The red blocks denote featurized ob- jects in the observation, c.g. cards in the deck. The cyan blocks denote fcaturized actions, c.g. cards in the hand that can be hinted/guessed. Cross-Attm, Self-Atin and MLP denote the cross attention, self attention, and fully-connected layers, respectively.
Q+ = MMLP(Mcnn(SA(O)....
Ax)).
for k = 1,..., m. Indeed, SA2I requires a forward pass for each action to caleulate the Q-value vector.
Note that, of all the architectures examined here, only SA2I processes the featurized actions and observations using the same weights. Indeed, MLP and Attn do not consider action features at all; MLP Action In uses a different vector of weights for each input dimension; and CA2T uses a different set of weights for the queries (i.e., the actions) than it uses for the keys and values (i.e., the observations). We hypothesize that this contrast may explain the strong performance of SA2I that we will describe in the following sections.
EXPERIMENTS
6.1 EXPERIMENT SETUP
We experimentally evaluate the architectures in the hint-guess game introduced in Section 4 In
Sections 6.2 6.4 we fix the hand size to be N = 5 and the features to be Fj = {1, 2, 3 and F2 (A, CJ. We use one-hot encoding for features; more specifically, we use two-hot vector to represent the two features of each card. In Sec. 6.5 we examine qualitatively different and there is only one feature, Fi = 10, 19). In this version,
version of the game where
'There is nothing particular about the hand size, and as shown in Appendix obtained with either larger or smaller hand size.
A.5 similar results can be
5

---

Under review as a conference paper at ICLR 2023
investigate whether is possible to capture ordinal relationships between actions using sinusoidal positional encodings. For these experiments, we encode each number as a 500-dimensional vector consisting of sine and cosine functions of different frequencies, as in Vaswani et al. (2017).
For both variants of the game, the observation input is a sequence of card representations for both hands H1 and H2, well the representation of the target card, C? (for the hinter) or the hinted card C) (for the guesser); a binary feature is added to each card's featurization specifying to which player belongs. We train agents in the standard self-play setting using independent Q-learning Tan 1993 IQL), where the hinter and guesser are jointly trained to maximize their score in ran- domly initialized games; the hinter and guesser do not share any weights. To avoid giving the set-based attention architectures an unfair advantage, we also permute the cards in the hands ob- served by all agents so that agents are not able to coordinate using the position of the cards. The action space of the architectures that output multiple Q-values is of the form "hint f2)" for each (fi, f2): we mask the Q-values so that only features corresponding to cards in the player's hand can be chosen. To evaluate success, we consider the agents' performance and behavior in both SP and the intra-AXP setting. We also provide fine-grained examination of their policies and investigate their ability to match the human-compatible response in different scenarios. See Appendix A.I for training details.
6.2 CROSS-PLAY PERFORMANCE.
First, we evaluate model cross-play (XP) performance for each architecture in the intra-AXP setting. In this setting, agents from independent training runs with different random seeds are paired together. Fig. 4 records the scores obtained by each pair of agents, where the diagonal entries are the within-pair SP scores and the off-diagonal entries are XP scores. Table I summarizes average SP and XP scores across agents.
Comparison Across Architectures. Fig. 4 shows that the XP matrix of all architectures except SA21 (Self Attention with Action as
Model Architectures Cross-Play
Model
Self-Play
MLP
0.27 + 0.04
0.85 0.02
Input) lack an interpretable pattern. The XP score is near chance for these architectures as shown in Table []] In contrast, the XP matrix for the SA2T model shows two clear clusters. Within the clusters, agents show XP performance nearly identical that of their SP. implying that they coordinate nearly perfectly with other agents trained with a different seed, whereas outside the clusters they achieve a re- turn close to zero. As we will show in the next section, the upper cluster, which has a higher average XP score, corresponds to a highly interpretable and human-like strategy where agents maximize the "similarity" between the target and the hint card (as well between the hint card and the guess card). In the lower, second cluster, agents do the opposite. They try to hint/guess cards that share no common feature with the target/hint cards. In the rest of the paper, we will refer to the cluster where agents maximize the similarity between cards as SA2I Sim, and the cluster where agents maximize the dissimilarity as SA2T Dissim. in section 6.3 the However, as will maxinize/mininize SA21 agents do not just feature similarity; they also demonstrate more sophisticated coordination patterns that exploit implicature.
MLP Action In
0.28 + 0.05
0.02
Attn
0.27
0.87
CA2I
0.29 + 0.03
0.85 0.02
SA2I
0.37 0.12
0.76 0.02
SA2I Sim
0.77 + 0.01
0.82 + 0.01
SA2T Dissim
0.71 + 0.01
0.72 0.01
Baseline Training Algorithms
Algothrim OP
Cross-Play
Self-Play
0.35 + 0.02
0.35 + 0.02
OBL (level 1) OBL (level 2)
0.27 + 0.05
0.29 0.06
0.28 + 0.04
0.28 + 0.05
Cross-play performance in the intra-AXP set- ting. Each entry is the average performance of 20 pairs of agents that are trained with different random seeds. The XP score is the off-diagonal mean of each grid. The SP score is the diagonal mean, i.e. the score attained when agents play with the peer they are trained with. Note that "chance agent" that acts randomly ex- pected to obtain a score of 0.28 in this setting. All models in the "Model Architecture" part are trained with IQL (Tan 1993), and all algorithms in the "Base- line Training Algorithm" section use MLP architec- ture.
Table I:
Comparison with intra-AXP Baselines. The bottom part of Table T contains the SP and XP results for two recent intra-AXP algorithms, other-play (Hu et al. 2020 OP) and bef-b learning (Hu et al. 2021 OBL... For details and implementation of the baseline algorthims, see Appendix A.3

---

Under review as a conference paper at ICLR 2023
As shown, the XP scores for OP agents only show marginal improvement over MLP agents. By preventing arbitrary symmetry breaking, OP improves XP performance, but only to limited extent. In contrast, the OBL agents fail to obtain scores beyond chance both in XP and SP. This is expected, as OBL is designed to explicitly prevent the interpretation of cheap cheap talk, i.e., costless messages between players, which is important for coordination in hint-guess.
6.3 POLICY EXAMINATION
Figure 4: Cross-play matrices. Cluserting visualization of paired evaluation of different agents trained under the same method. The y-axis represents the agent index of the hinter and the x-axis represents the agent index of the guesser. Each block in the grid is obtained evaluating the pair of agents 10K games with different random seeds. Numerical performance shown in Table !
Diaiin
Cat
Conditional probability matrices. We show Pr (Guess) Hint), for the guesser guess particular card (x-ax axis) when the hinted card the card on the y-axis Each subplot is sample average 20 agent- pairs with different seeds for 1K games within pair. Pr [Hint]Target look almost identical so is omitted.
Figure 5:
Conditional Probability Analysis. In Fig. 5 we provide the conditional probability for the guesser to guess card given the hinted card (bottom row). One crucial thing to analyze is whether agents assign different probabilities to actions based on the features they share with the observation. One that for MLP, MLP Action In, Attn, and CA2T the probability matrices for both target- hint and hint-guess are nearly uniform. This implies that the SP policies across seeds each form their own private language for arbitrary and undecipherable coordination.
In contrast, for the two clusters of SA2T agents, the correlation (or anti-correlation) between the action features and target/hint card features much stronger. For SA2I Sim, both the hinter and the guesser prioritize exact matches when they are present. If the exact match is not present, they turn to cards that share one feature in common. The SA2T Dissim agents the exact aoposite-matchin cards together that share as few features as possible.
Human Compatibility Analysis. However, we find that the nuance with which these clusters play goes beyond simply maximizing or minimizing feature similarity. To demonstrate this, we run simulations on the four scenarios (exact match, feature similarity, mutual exclusivity, exclusivit++similariy) shown in Fig. 2 and described in Section 4 In Table 2 we record the percentage of times where SA2I agents in each cluster chose the human-compatible actions in Fig. We find that SA21 Sim agents demonstrate coordination patterns that are nearly identical to a human-compatible policy. These results are surprising given that our models have never been trained with any human data. Furthermore, mutual exclusivity was thought to be hard for deep learning models to learn (Gandhi & Lake 2020). In contrast, while the SA21 Dissim agents always perform actions that are the opposite the human-compatible policy, these conventions still interpretable and non aabrrbitrrry.
6.4 HUMAN-AI EXPERIMENTS
We recruited 10 university students to play hint-guess. Each subject played as hinter for 15 randomly generated games, totaling 150 different games. These subjects are then cross-matched to play as
7

---

Under review as a conference paper at ICLR 2023
Self-Play (SA21 Sim))
Cross-Play (SA2) Stan)
Selt-Play (SA2) Dissim)
Cross-Play (SA21 Dissim)
Scenario
Homan (%)
(%)
Human (%)
(in
Human
Win (%)
Human (%)
Win (%)
Exact match
10000
100.0
100.0
100.0
0.0
1000
0.0
10C.
Feature similarity
100.0
100.0
100.0
100.0
0.0
100.0
0.0
100.0
Mutual exclusivity
100.0
100.0
100.0
100.0
9.3
91.2
9.3
92.2
Similarity + Exclusivity
92.0
91.7
97.9
99.5
3.2
98.44
0.0
99.9
Table 2:
Behavioral analysis for the SA2I model in the Fig. scenarios. We randomly chose 20 agent-pairs from each cluster simulated the same scenario times. Human (%) denotes the fraction of games where the hinter hints the card that corresponds to human-compatible choice (highlighted yellow in Fig. and Win (%) denotes the fraction where the guesser correctly guesses.
guessers with the hints their peers generated. The human hints are also fed into randomly sampled MLP and SA21 Sim guesser-agents to test Al performance against human partners. The experiment was carefully designed so that the hinter is never informed of the guesser's s guess and the guesser is never informed of the true target card. This experimental design ensures that the human participants generate zero-shot data, and do not optimize their play using previous experience. Further details of the experiment are in Appendix A.2
Ad-Hoc Performance. In the right table of Fig. we report average ad-hoe coordination scores obtained by hinier-guesser pairs for human-human, human-MLP, and human-SA21 Sim. Humans obtained an average ad-hoc coordination score of 0.75 with their peers. As a baseline, the MLP guessers show poor performance in understanding human -emaangeeerrted hints, barely outperforming random guessing. In contrast, the SA2I Sim guessers achieve human-level performance with an average ZSC score of 0.77 with humans. Note that this score is very close the average ZSC score where SA2I Sim agents cross-played among themselves.
in Table [
kinds of correlations between human play and Al play. The right table of Fig. shows the percentage of games where model guessers chose the same action the human guessers. In 80.7% of the games, the SA2T Sim agents and human guessers agree on the same action across many different scenarios. In contrast, the MLP guessers deviate from human guessers, with only 40.7% 7% agreement.
Human-Al Behavior Correlation. We also investigate
Human-Al Performance Correlation. The left plot of Fig. 6 shows the cor- relations between human-human play and human-Al play. Across humans we observed a range of skill levels at the game, with some hinters not even achieving 50% guess accuracy when paired with other humans, while others exceeded 90% (as measured along the x-axis). We see the performance of human-SA2T Sim pairs increased substantially with the skill level of the human, whereas the performance of human-MLP pairs was less sensitive along this axis. Taken together, these results suggest that the SA2I Sim agent both better at coordinating with people than baseline model and is also better at coordinating with the people who are better at coordinating with people.
Haman-SAAi Humon-ML MLP
Gusser Type
Cross-Play
Agreement
0.30
40.7%
SAZI
0.77
80.7%
Human
0.75
(100%)
Hummar-ituman Pair Score
Figure 6: Human-Al ZSC results comparing human pairs, human-SA22 Sim pairs and human-MLA pairs. In the left plot, each point corresponds to particular human hinter. In the table, "agree" measures the percentage of games in which the guesser selected the same card the human guesser.
6.5 SINUSOIDAL ENCODING, MULTI-HEAD AND MULTI-LAYER ATTENTION
In previous settings, the SA2I models were able to learn sophisticated cognitive patterns like mutual exclusivity from one-hot encoding of inputs (equal or non-equal). However, one-hot encoding does not capture richer semantic relationships between features. For instance, in hint hint-guess, if cards are encoded oot, the agent can only "know" that the card 1A has the same first label the card IB, but cannot "know" whether the number is closer to 2 than to 5. Thus, in this section, investigate whether
expressive encoding enables the SA2T model to learn to leverage
8

---

Under review as a conference paper at ICLR 2023
the ordinal relationship between features. Specifically, we examine the performance of sinusoidal positional encodings in a variant of hint-guess in which the only card feature is a number between and 19, as described in the experiment setup.
TableB] SP and XP performance of SA2I agents with one-hot encoding and sinusoidal encoding in this ängle-feature setting. Agents with one-hot encoding are near chance in XP.
Encoding
One-hot
Sinusoidal
SP
0.81 = 0.02
0.92 = 0.01
XP
0.36 = 0.10
0.52 # 0.16
XP (SA21 Sim)
0.92 # 0.01
They do not form clusters as observed be- fore. We hypothesize that the failure of one-hot agents is because of the large feature space (20 numbers) relative to the small number of fea- tures (1). Specifically, since the feature space large and the agents are only sensitive to ex- act overlap, they degenerate into using arbitrary conventions, resulting in a large performance gap between SP and XP.
XP (SA21 Dissim)
0.93 = 0.01
Table SP and XP scores for SA2T agents with one- hot and sinusoidal encodings. Agents with sinusoidal encoding form two clusters we also show the within- cluster results.
Agents with sinusoidal encodings. in contrast, split into two clusters (named SA21 Sim and SA2T Dissim as before), wherein each cluster has near-perfect SP and XP scores with no sig- nificant performance gap. We find that these agents learn exploit the ordering and distance information between the numbers for coordina- tion. SA2I Sim agents rank the hinter's and guesser's 's hands in the same order and match the corresponding numbers as hint-guess pairs. SA2T Dissim agents, on the other hand, rank one hand in ascending order and the other hand in descending order for matching. See Fig. 7 for a concrete example. For both strategies, if the hinter does not have duplicate numbers in its hand, agents obtain a near-perfect play score. Indeed, in a XP simulation across 15 agent-pairs with IK games per pair, where each agent's hand is drawn without replacement (so no duplicates), 99.9% of the time, the SA21 Sim agents hint/guess exactly according to the same order matching scheme. Also in 99.9% of the time, the SA21 Dissim agents hint/guess according to the reversed order matching scheme.
Bano
Beversed
Naive Fenture
Ordde
2
Order
3
Match
Similarity
3
Figure 7: A scenario illustrating the behavior of SA2I agents. In this scenario, the hinter's hand is 2,3 and the guexser's 's hand is (2.3, 4) (the actual hands seen by agents will be permuted). We find that with probabil- ity close to SA21 Sim agents use a strategy that ex- ploits same order matching (left). They sort both hands in the same order and match 1-2, 2-3, 3-4, etc. Also with probability close to 1, SA2T Dissim agents versed order matching (middle). They sort one hand in ascending order and the other in descending order and match. To compare we also show naive feature similar- ity (right) that solely maximizes feature similarity; this strategy will match 1-2, 2-2, 3-3 and leave out 4.
re-
We also find that the results for SA2I architectures are qualitatively similar when using multi- head or multi-layer attention or both in Appendix[A.4 This suggests that SA2I may also be able to produce human-compatible policies in settings where larger architectures are required for effective learning.
CONCLUSION AND FUTURE WORK
We investigated the effect of network architecture on the ability of learning algorithms to exploit the semantic relationship between shared features across actions and observations for coordination. We found that celf-attention-bare based architectures which jointly process featurized representation of observations and actions have a better inductive bias for exploiting this relationship; we hypothesize that this difference in performance may be explained by whether the architectures use the same weights to process both the action features and the observation features.
For future work, we think it would be interesting to investigate the behavior of the SA2T architecture in settings with more abstract features (compared to the categorical and ordinal features investigated in this work), like word embeddings would be to use SA2I on a sequence of actions to capture the abstract correspondence between actions across different time steps. One limitation of this work is that action features need to be provided to the agent, which is a stronger assumption than is typical for PoMMPS In some situations, it may be possible to learn action features, and this is another direction for future work.
multi-modal features. Another interesting future direction

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. International Conference on Learning Representations, 2015.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, lain Dunning, Shibl Mourad, Hugo Larochelle, Mare G Bellemare, and Michael Bowling. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 2019. doi: 10.1016/f.artint.200 201..1011.......
Samuel Barrett, Peter Stone, and Sarit Kraus. Empirical evaluation of ad hoc teamwork in the pursuit domain. AAMAS, pp. 567-574, 2011.
Colin Camerer. Behavioral game theory: Experiments in strategic interaction. Princeton univer- sity press, 2011.
Colin F Camerer, Teck-Hua Ho, and Juin-Kuan Chong. cognitive hierarchy model of games. The Quarterly Journal of Economics, 119(3)8611889) 2004.
Micah Carroli, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances Neural Information Processing Systems, 32:5174-5185, 2019.
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.086630, 2020.
Cícero Nogueira dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou, Attentive pooling networks. CoRR, abs/1602.03609, 2016. http:/t:://aatt /ttppi///axiv..../ wwtgg/bb//eee 03609
Michael C Frank and Noah D Goodman. Predicting pragmatic reasoning in language games. Sci- ence, 336(6084):998 99889999999999999999999988998..9 2012.
Kanishk Gandhi and Brenden M Lake. Mutual exclusivity as a challenge for deep neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Noah D Goodman and Andreas Stuhlmüller. Knowledge and implicature: Modeling language un- derstanding as social cognition. Topics in cognitive science, 5(1):173-184, 2013.
Herbert P Grice. Logic and conversation. In Speech acts, pp. 41-58. Brill, 1975.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-Play" for Zero-Shot Coordination. International Conference on Machine Learning, 2020.
Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, David Wu, Noam Brown, and Jakob Foerster. Off-belief learning. International Conference on Machine Learning, 2021.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami. Dan Rosenbaum, Oriol Vinyals, and Yee Whye Tch. Attentive neural processes. International Conference on Learning Representations, 2019.
Max Seeiman-Weiness Mark K Ho, Joseph Austerweil, Michael L. Littman, and Joshua Tenen- baum. Coordinate to cooperate or compete: abstract goals and joint intentions in social interac- tion. In CogSci, 2016.
W Köhler. Gestalt psychology. Liveright, 1929.
Mare Lanctot, Vinícius Flores Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent rein- forcement learning. In NIPS, 2017.
Adam Lerer and Alexander Peysakhovich. Learning social conventions in markov games. arXiv preprint 1806. 10071, 2018.
David K Lewis. Convention: Philosophical Study. Blackwell, 1969.
10

---

Under review as a conference paper at ICLR 2023
Ellen M Markman and Gwyn F Wachtel. Children's 's use of mutual exclusivity to constrain the meanings of words. Cognitive Psychology, pp. 121-157, 1988.
Daphne Maurer, Thanujeni Pathman, and Catherine Mondloch. The shape of boubas: found-shape correspondences in toddlers and adults. Developmental science, 9(3):316-322 2006.
Jennifer B Misyak, Tigran Melkonyan, Hossam Zeitoun, and Nick Chater. Unwritten rules: virtual bargaining underpins social interaction, culture, and society. Trends in cognitive sciences, 18(10): 512-519, 2014.
Nair, M Tambe, M Yokoo, D Pynadath, and S Marsella. Taming decentralized pomdps: Towards efficient policy computation for multiagent settings. International Joint Conferences on Artificial Intelligence. pp. 705-711, 2003.
Avik Pal, Jonah Philion, Yuan- Hoan-Hong aae Liao, and Sanja Fidler. Emergent road rules in multi-agent driving environments. In International Conference Learning Representations, 2020.
Michael Shum, Max Kleiman- Weiner, Michael L Littman, and Joshua B Tenenbaum. Theory of minds: Understanding behavior in groups through inverse planning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 6163-6170, 2019.
Dale O Stahl and Paul W Wilson On playersfoodels of other players: Theory and experimental evidence. Games and Economic Behavior, 10(1):218-254, 1995.
Peter Stone. Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Tweety-Fourth AAAI Conference on Artificial Intelligence, 2010.
DJ Strouse, Kevin R. McKee, Matthew Botvinick, Edward Hughes, and Richard Everett. Col- laborating with humans without human data. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL net Forum? 1doomm2111111ooomooooo
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. Proceedings of the tenth international conference machine learning, pp. 330-337, 1993.
Ming Tan, Cicero Dos Santos, Bing Xiang, and Bowen Zhou Improved representation learning for question answer matching. Annual Meeting of the Association for Computational Linguistics (ACL), pp. 464 473, 01 2016. doi: 10. 00.18633/v//166 1044.
Michael Tomasello, Malinda Carpenter, Josep Call, Tanya Behne, and Henrike Moll. Understanding and sharing intentions: The origins of cultural cognition. Behavioral and brain sciences, 28(5): 675-691, 2005.
Michael Tomasello, Malinda Carpenter, and Ulf Liszkowski. A new look at infant pointing. Child development, 78(3):705-722, 2007.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need. Advances Neural Information Processing Systems, 2017.
Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Too many cooks: Bayesian inference for coordinating multi-agent collabora- tion. Topics in Cognitive Science, 13(2)414-432, 2021.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. International Conference on Machine Learning, 2016.
Liu Yang, Qingyao Ai, J. Guo, and William Bruce Croft. ammm: Ranking short answer texts with baattdoo--ased neural matching model. Proceedings of the 25th ACM International on Confer- ence on Information and Knowledge Management, 2016.
Peyton Young. The evolution of conventions. Econometrica: Journal of the Econometric Society, pp. 57-84, 1993.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 TRAINING AND MODEL ARCHITECTURE DETAILS
Training Setup. By default we use standard experience-replay with replay memory of size 300K. For optimization, we use the mean squared error loss, stochastic gradient descent with learning rate set to 10-4 and minibatches of size 500 each. We train the agents using 4M episodes. To allow more data to be collected between training steps, we update the network only after we receive 500 new observations rather than after every observation. We use the standard exponential decay scheme with exploration rate Cm + (cu 6m) cxp(-n// K),K), where is the number of episodes, cm 0.01, 6 = 0.95, and K = 50, 000. All experiments were run on two computing nodes with 256GB of memory and a 28-Core Intel 2.4GHz CPU. A single training run takes roughly 8 hours for the SA21 model and hours for all other models. All hinters and guessers have the same model structure.
The model architecture details follows:
Feedforward Networks (MLP). The MLP agent is a feedforward ReLU neural network with hidden layers (width 128).
Attention Models. All three attention- models share the same single-head attention layer with size determined by the shapes of the input and output. By default do not add position encoding. In Attn and SA2I, we add a feedforward ReLU MLP with 3 hidden layers (width 128 each) after the attention layer. In CA21, add fully-connected layers with conformable dimensions but do add extra ReLU activation.
12

---

Under review as a conference paper at ICLR 2023
A.2 DESIGN OF HUMAN-AI AI EXPERIMENTS
In this section we provide details of the design of human-based experiments.
We recruited 10 individuals who are undergraduate and master students university. The instruc- tions they received include: (i) rules of hint guess; (ii) that they need to assume they are playing against a guesser who an ordinary human (they are told that they would play against Al bots); (ii) that the position of the cards are permuted so they cannot provide/interprel hints based on card position, and (iv) if cards have the same features they are effectively the same.
After they showed understanding of the instructions, the subjects were asked to act as hinters. Each subject was provided with 15 randomly generated games with F1 2, 3), F2 {A, B, C*} and N = 5. They were presented with both hands and the target card and were asked to write down their hints. sample question they received was:
Playable card is: 2B
Your (hinter) hand is: 1C, 1B, 3B, 2C, 1A Guesser hand is: 1B, 2C, 23, 20, 2B Please choose a hint card from YOUR OWN hand!
After the subjects provided hints, they given feedback ensure that they could not learn about the agent they are playing with; this ensures that the setting remains ad hoc.
Then used the hints provided by the subjects obtain human-human (hinter- (hinter-guesserr guesser) and human-AI scores. To obtain human/human scores, we randomly mix-matched the subjects so that each subject would now time, they were provided with both hands and the human hint, and were ask to choose one card to play. Same as before, they did not receive any feedback about their guesses ensure that the setting remained ad hoc.
the guesser for 15 games generated by another human subject. This
To obtain human-SA Sim and human- ML. scores, we reproduced the 150 games and fed human hints to randomly sampled SA2T Sim-MLP guesser agents.
13

---

Under review as a conference paper at ICLR 2023
A.3 DETAILS OF INTRA- AXP BASELINES
In this section we provide implementation details of other-play (Hu et al. 2020) and off-belief learning (Hu et al. 2021), two recent intra-algorithm cross play (intra-AXP) methods that we use as baselines.
Other-play (OP). The goal of OP is to find strategy that robust partners breaking symmetries in different ways. To achieve this, it uses reinforcement learning to maximize returns when each agent matched with agents playing the same policy, but under random relabeling of states and actions under known symmetries of the Dec-POMDP (Hu et al. (020). To apply OP, we change the objective function of the hinter from the standard self-play (SP) learning rule objective
arg max J (m², x²)
To the OP objective
n arg max E ( (T²))
(2)
where the expectation is taken with respect to a uniform distribution on Φ. where & describes the symmetries in the underlying Dec-POMDP (in the hint-and-play games, the symmetries are the two features).
As OP only changes the objective function, can be applied on top of any SP algorithm. We choose to apply OP on top of the MLP architecture, using the same training method as detailed in Appendix. A. and change the objective to the OP objective. In implementation, this means that the guesser will receive a feature-permuted version of the game, i.e. feature 1 and feature 2 (the letters and the numbers) of the hands and the hint that the guesser receives will be a permuted version of what the hinter originally receives.
Off-belief learning (OBL). OBL (Hu et al. 2021) regularizes agents' ability to make inferences based the behavior of others by forcing the agents optimize their policy Ti assuming past actions were taken by given fixed policy To. while in the same time assuming that future actions will be taken by गI In practice, OBL can be iterated in hierarchical order, where the optimal policy from the lower level becomes the input to the next higher level.
We apply OBL MLP agents and keep the training setup the same in Appendix A.1 At the lowest level (level 1), OBL agents assume To is the policies where actions are chosen uniformly at random, And OBL level 2 assumes the policy from OBL level I is the new To and forth.
14

---

Under review as a conference paper at ICLR 2023
A.4 MULTI-HEAD AND MULTI-LAYER ATTENTION
We show the XP results when using the SA2I architecture with different number of attention heads attention layers. We also show the same results for using either one-hot sinusoidal encoding. To enable using sinusoidal encoding, we fix the hand size 3 and only use one feature, F1 {0, I..,, 19), which equivalent fixing F2 {A].
For these experiments we use standard experience-replay with a replay memory of size IK. For optimization, we use the mean squared error loss, stochastic gradient descent with learning rate set to 10 and minibatches of size 200 each. We train the agents using 5M episodes. To allow more data to be collected between training steps, we update the network only after we receive 50 new observations rather than after every observation. We use the standard exponential decay scheme with exploration rate fm + (ep fm) exp((n) K), where n the number episodes, 6m 0.05, 6o 0.95, and K 15, 000. All experiments were on two computing nodes with 256GB of memory and 28-Core Intel 2.4GHz CPU, single training run takes roughly hours for 1-layer attention and 18 hours for 3-layer attention. All hinters and guessers have the same model structure.
Encoding
Layers
Heads
SP
XP
XP (Clus. 1)
XP (Clus. 2)
One-hot
1
1
0.81 + 0.02
One-hot
0.90 + 0.02
0.10
One-hot
3
1
0.40 # 0.04
0.38
One-hot
3
4
0.89
0.11
Sinusoidal
1
1
0.01
0.16
0.92
0.93
Sinusoidal
1
4
0.92
0.67
0.02
0..00
Sinusoidal
3
1
0.94
0.01
0.16
0.95
0.91
Sinusoidal 3
4
0.96 # 0.01
0.09
0.95
0.95 # 0.01
Table 4: Robustness results for using multi-head and multi-layer attention. We show SP and XP scores for SA2I agents with different number of attention modules (layers). attention heads, and different encoding of features. Each entry the average performance of 20 pairs of agents that are trained with different random seeds. Agents with sinusoidal encoding forms two clusters so we also show the within-cluste results.
15

---

Under review as a conference paper at ICLR 2023
A.5 EFFECT OF HAND SIZE ON PERFORMANCE
In this section we analyze the effect of hand size on agents' self-play and cross-play performance.
We fix the features be Fi {1, 2, and F2 {A, B, C} and run experiments with hand size N = {3, J. as opposed to N = in the main text. As shown in the following Table 5 SP and XP results largely confirm our findings for N
We also show the same results for a simplified version of the game, where both hinter and guesser have the same hand. In the simplified game, an optimal and intuitive strategy exists, which is to always hint the target card and then guess the card that is hinted. The training setup and model details are exactly the same as in A.1
Full Game (N = 3)
Same Hand Condition
Method
Cross-Play
Self-Play
Cross-Play
Self-Play
MLP
0.47 # 0.06
0.92 0.01
0.87 = 0.04
0.98 = 0.02
MLP Action In
0.47 + 0.07
0.92 = 0.01
0.70 0.04
0.99 = 0.01
Attn
0.47 + 0.06
0.92 = 0.01
0.47 + 0.05
0.95 = 0.01
CA21
0.46 + 0.05
0.81 = 0.01
0.46 0.06
0.88 = 0.04
SA2I
0.43 + 0.12
0.80 0.01
0.45
0.95 + 0.02
SA2T Sim
0.81 +: 0.01
0.81 + 0.01
1.00 + 0.00
1.00 + 0.00
SA21 Dissim
0.80 = 0.01
0.80 = 0.01
0.89 #
0.90 = 0.01
OP
0.55 + 0.03
0.54 0.04
0.90 + 0.01
0.98 0.01
OBL (level I)
0.47 0.07
0.07
0.44 = 0.06
0.47 + 0.03
0.48 = 0.03
OBL (level 2)
0.47 + 0.04
0.46 0.06
0.47 0.05
0.47 0.05
Full Game (N 7)
Same Hand Condition
Method
Cross-Play
Self-Play
Cross-Play
Self-Play
MLP
0.22 + 0.08
0.72 0.03
0.77 0.10
0.96 0.08
MLP Action In
0.25 + 0.09
0.76 = 0.01
0.62 + 0.14
0.95 = 0.05
Attn
0.24 +: 0.09
0.77 = 0.03
0.24 0.08
0.93 = 0.01
CA2I
0.22 0.11
0.77 = 0.02
0.22 0.13
0.80 = 0.09
SA2T
0.37 + 0.33
0.70 + 0.06
0.40
0.89 + 0.11
SA2T Sim
0.78 + 0.01
0.78 0.03
1.00 + 0.00
1.00 + 0.00
SA21 Dissim
0.67 + 0.02
0.67 - 0.02
0.67 +
0.67 = 0.02
OP
0.32 0.02
0.31 0.02
0.79
0.95 = 0.02
OBL (level 1)
0.21 + 0.04
0.22 = 0.03
0.22 - 0.03
0.22 = 0.04
OBL (level 2)
0.24 + 0.03
0.22 = 0.03
0.22 0.03
0.25 1 0.02
Table 5: Cross-play performance for card number N = and N = 7. We show results for both the full hint guess game and the simplified version where both agents have the same hand. Each entry is the average performance is of 20 pairs of agents that trained with different random seeds. Cross-play score is the non-diagonal mean of each grid. Self-play score is the diagonal mean, ie. the score attained when agents play with the peer they trained with.
16


Reviews:
[{"review_id": "YJlB9NXlMD", "reviewer": "Reviewer_Xa5L", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work investigates the learning of intuitive, interpretable policies that leverage features shared between observations and actions, by means of a novel two-player game (Hint-Guess) where such relationships play a pivotal role. Several different model architectures are evaluated, and they are found to demonstrate widely varying abilities. One particular attention architecture (SA2I) far outperforms the others on this task, and its policies are shown to take advantage of the task structure in ways that one would intuitively expect.", "summary_of_the_review": "The paper makes an interesting claim, but does not provide enough evidence to support it. \n\n==== After discussions with authors ====\n\nThe authors have supplied many important details that were missing from the original paper. But in my view, the most serious limitation of the work continues to be its reliance on a single, particularly simple task of no practical importance. This limitation was also called out by the other reviewers. \n\nIn addition, I still cannot see anything particularly surprising about the main results on this task, presented in Table 1. As noted by the paper, and confirmed by the authors in our extensive discussion, the poor performance of MLP and Attn is easily explained by the fact that they “do not consider action features at all”, which are crucial factors in this task by construction. The only remaining question is whether self-attention (SA2I) or cross-attention (CA2I) is more appropriate to the task. Since cross-attention is a less-costly special case of self-attention, it is no surprise that self-attention has an advantage. (The cross-play setting is the important one, as it factors out the possibility of learning a private language.)\n\nFor these reasons, and despite the commendable work that this paper represents, I have to leave my score unchanged. I look forward to extensions of this work on the important connection between semantic relationships and interpretable policies.", "strength_and_weaknesses": "**Strengths**\n\nAs motivation, the paper carefully argues for the importance of interpretable policies, in particular with respect to the exploitation of semantic relationships. As one example of such relationships, the Hint-Guess game (analogous to a type of card game) is cleverly designed to test an agent’s ability to take advantage of relationships between observations and actions. The experimental results show that SA2I is significantly more capable than the other models at this task. Additional studies and game variants characterize the gains in some detail, and even show that the best agents perform well when paired with human players.\n\n**Weaknesses**\n\nThere would be great value in an agent architecture that demonstrated it could reliably learn interpretable policies, and one can imagine a wide variety of tasks involving semantic relationships. But no single task (such as the Hint-Guess game) by itself can provide strong evidence that a particular model’s ability to learn interpretable policies would apply to a broader set of tasks. So one major weakness is the paper’s consideration of just a single task and modest variants of it.\n\nIn addition, given the nature of the Hint-Guess game, it is hard to imagine any successful policy that did not rely on the strategy that humans find obvious here. Despite this expectation (that only one particular policy could succeed), the winning agent manages to find two winning policies, one of which (Dissim) is far less intuitive than the other (Sim). This experimental outcome raises doubt that the SA2I architecture has any general ability to learn interpretable policies.\n\nIn fact, there is a simple, alternative explanation for why attention should be expected to outperform simple MLPs on this task. In the Hint-Guess game, each observation contains unordered sets of objects (the cards in each hand). Since attention-based models (like transformers) are set processors by design, it is no surprise that applying attention to the observation would outperform an MLP. This alternative explanation calls into question the paper’s claim that the SA2I model “has a strong inductive bias toward using the relationship between actions and observations in intuitive ways.”\n\nThe term “self-play” creates confusion. The term “self-play” is commonly understood (for games like Chess or Go) to mean that an agent is playing against a copy of itself, but that does not appear to be the case in this work. For instance, “we train agents in the standard self-play setting… where the hinter and guesser are jointly trained to maximize their score”. It appears that in this paper, self-play vs. cross-play simply indicates whether an agent was trained with the (non-self) agent it is evaluated with. But it’s hard for a reader to be sure without a clearer explanation. Similarly, in the intra-AXP setting, what is the algorithm being shared?\n\nSeveral crucial aspects of the Hint-Guess experiments are never adequately explained. For instance, the hinter and guesser are apparently two separate agents. But is one agent trained to always be the hinter, while the other agent is always the guesser? If so, do the hinter and guesser share any weights? Or is each agent trained in both roles, sometimes as the hinter and sometimes as the guesser? If so, then how does the agent know in any particular game whether it is playing as the hinter or as the guesser? Is this information included in the observation?\n\nImportant details of the observations are left out. For instance, “the observation input is a sequence of card representations for both hands $H_1$ and $H_2$, as well as the representation of the target card, $C_i^2$ (for the hinter) or the hinted card $C_j^1$ (for the guesser)”. Since the hand size is 5 for the main experiments, this could mean that each agent will see 11 cards at once:  2 hands plus a copy of either the target or the hinted card. Or are only 10 cards visible, and the target or hinted card is specified some other way? And how does the agent know which cards are its own? Perhaps the first 5 cards are always the agent’s own cards, or maybe some flag or embedding is applied to indicate ownership. Or is this information conveyed some other way? \n\nThe block diagrams in Fig. 3 are helpful, but they also raise important unanswered questions. When multiple Q values are output, how are those values mapped to specific cards, given that the order of the cards is randomly permuted on input, and the Reduce Mean operation loses order information? Does each Reduce Mean operation average a set of input vectors to produce a vector of the same dimensionality as the inputs? And exactly which vectors serve as the inputs? \n\nFinally, since this work proposes a novel task, the baseline results would be more meaningful if the baseline model hyperparameters had been tuned. But hyperparameter tuning is never mentioned.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UcxHMRqBp6", "reviewer": "Reviewer_w7Kr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates whether different model architectures leverage similarities between features of the observation and the action like humans do. It proposes a communication game where the agent can reference to observation by choosing actions whose features imply features of the observation. The authors found a self-attention architecture that produces a policy that is similar to that of humans. Experiments show that the policy can coordinate successfully with humans without ever being trained with them. \n", "summary_of_the_review": "I think the paper serves as a good starting point for exploring this topic. It would motivate subsequent studies on more complex tasks. I recommend acceptance.\n\n====After Rebuttal====\n\nI have read the authors' response. While I appreciate the explanations, my initial concerns cannot be resolved solely by the response and would require substantial revision of the paper. I thus retain my initial assessment. ", "strength_and_weaknesses": "Strengths:\n* This is a very interesting paper. The topic is well motivated by studies in linguistic and socio-cognitive science. I like that the authors explain intuitive concepts using well-illustrated examples. \n* The problem of learning to coordinate with humans is an important and challenging problem. This study may lead to novel ways of solving this problem. \n* The proposed task is easy to understand and useful in evaluating the capabilities of interest.\n* The experiments are well-designed. The empirical results support the claims made by the authors. \n\nWeaknesses:\n* The authors do not offer a satisfying explanation or motivation on why the SA2I architecture produces the observed results (whereas the CA2I does not). Given that they are searching for the architecture with the right inductive bias, it is unclear what that inductive bias is? What general principles can be drawn for designing more complex architecture to tackle more challenging problems?\n* The authors present the results as mere findings. I think a better way to structure the paper is to first clearly form expectations (or hypotheses) about the proposed architectures, explain why you have such expectations, and then present the results as verification of the expectations.\n* The proposed task remains simplistic. It is unclear whether the obtained findings will carry over to tasks with more complex observation and action spaces.  \n* The method requires a common representation for the observation and the action, which may not be trivial to construct in other scenarios.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1bgmIwy9Qs", "reviewer": "Reviewer_rXa8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper suggests an interesting idea of real-world actions often being grounded by observations and the shared action-observation features naturally emerging for coordination between agents. This paper hypothesizes that the policy architecture is crucial for emerging the shared action-observation features. The experiments are done using the simple hint-guess game and demonstrate that the sharable action-observation features emerge only when each action is fed into the Q-network.", "summary_of_the_review": "This paper proposes an interesting and potentially important problem for the community. However, the evaluation is done in too simplified tasks, which makes the paper not very convincing. I still value the novelty of the paper, so leaning toward weak acceptance. A few more tasks with sufficient complexity would make this paper very strong.", "strength_and_weaknesses": "\n### Strengths\n\n* This paper points out an interesting perspective of learning shared action-observation features for coordination that can be used commonly across independently trained agents.\n\n* The investigation of multiple different Q-network architectures shows clear evidence of the necessity for action representations in the cross-play experiments.\n\n* The evaluation of human subjects further proves that the learned strategies are compatible with humans.\n\n\n### Weaknesses\n\n* The problem formulation in the paper suggests a novel and interesting research direction. However, the example tasks seem to be too simple and not well connected to the realistic or practical use of the emerged (shared) action representations. Thus, one thing that can further strengthen the paper would be additional discussion or experiments with more realistic or practical scenarios.\n\n* The experiments are conducted on the one-feature scenarios and two-feature scenarios. It might be interesting to see whether the findings from these scenarios hold for a more number of features. It might be also interesting to discuss whether the same hypothesis holds for more than two agents.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject