On the Critical Role of Conventions in Adaptive Human-AI Collaboration
Authors: Andy Shih
Arjun Sawhney
Jovana Kondic
Stefano Ermon
Dorsa Sadigh
Venue: ICLR 2021 Poster (2021)
Keywords: Multi-agent games
emergent behavior
transfer learning
human-AI collaboration

Abstract: Humans can quickly adapt to new partners in collaborative tasks (e.g. playing basketball), because they understand which fundamental skills of the task (e.g. how to dribble, how to shoot) carry over across new partners. Humans can also quickly adapt to similar tasks with the same partners by carrying over conventions that they have developed (e.g. raising hand signals pass the ball), without learning to coordinate from scratch. To collaborate seamlessly with humans, AI agents should adapt quickly to new partners and new tasks as well. However, current approaches have not attempted to distinguish between the complexities intrinsic to a task and the conventions used by a partner, and more generally there has been little focus on leveraging conventions for adapting to new settings. In this work, we propose a learning framework that teases apart rule-dependent representation from convention-dependent representation in a principled way. We show that, under some assumptions, our rule-dependent representation is a sufficient statistic of the distribution over best-response strategies across partners. Using this separation of representations, our agents are able to adapt quickly to new partners, and to coordinate with old partners on new tasks in a zero-shot manner. We experimentally validate our approach on three collaborative tasks varying in complexity: a contextual multi-armed bandit, a block placing task, and the card game Hanabi.

Published as a conference paper at ICLR 2021
ON THE CRITICAL ROLE OF CONVENTIONS IN ADAPTIVE HUMAN- AI COLLABORATION
Andy Shih', Arjun Sawhney Jovana Kondic Stefano Ermon & Dorsa Sadigh Stanford University, 'Princeton University
f andyshih, ar arjunswwhne,, ermon, dorsa] ecteeeccsstttsnt.... stanford.edu jkondiceprinnettnnc edu
ABSTRACT
Humans can quickly adapt to new partners collaborative tasks (e.g. playing basketball), because they understand which fundamental skills of the task (e.g. how to dribble, how to shoot) carry over across new partners. Humans can also quickly adapt similar tasks with the same partners by carrying over conventions that they have developed (e.g. raising hand signals pass the ball), without learning coordinate from scratch. To collaborate seamlessly with humans, AI agents should adapt quickly to new partners and new tasks as well. However, current approaches have not attempted to distinguish between the complexities intrinsic a task and the conventions used by a partner, and more generally there has been little focus on leveraging conventions for adapting to new settings. In this work, we propose learning framework that teases apart vele-dependent representation from convention deppedden representation in a principled way. We show that, under some assumptions, our rule-dependent representation is a sufficient statistic of the distribution over best-response strategies across partners. Using this separation of representations, our agents are able to adapt quickly to new partners, and to coordinate with old partners on new tasks a zero-shot manner. We experimen- tally validate our approach three collaborative tasks varying in complexity: contextual multi-armed bandit, a block placing task, and the card game Hanabi.
INTRODUCTION
Humans collaborate well together in complex tasks by adapting to each other through repeated interactions. What emerges from these repeated interactions shared knowledge about the interaction history. We intuitively refer to this shared knowledge conventions. Convention formation helps explain why teammates collaborate better than groups of strangers, and why friends develop lingo incomprehensible to outsiders. The notion of conventions has been studied in language (Clark & Wilkes-Gibbs 1986; Clark, 1996; Hawkins et al., 2017; Khani et al., 2018) and also alluded to in more general multiagent collaborative tasks (Boutilier, 1996; Stone et al., 2010; Foerster et al., 2019; Carroll et al., 2019; Lerer & Peysakhovich. 2019; Hu et al., 2020). For example, Foerster et al. (2019) trained agents to play the card game Hanabi, and noted the emergent convention that "hinting for red or yellow indicates that the newest card of the other player is playable".
One established characterization of convention that is commonly used (Boutilier, 1996; Hawkins et al., 2017: Lerer & Peysakhovich, 2019) is an arbitrary solution to recurring coordination problem (Lewis, 1969). A convention thus one of many possible solutions that a group of partners happens to converge to. This is in contrast to problem solutions that are enforced by the rule constraints, and would have arisen no matter how the partners collaborated and what behavior they converged to. Success in collaborative task typically involves learning both types of knowledge, which we will refer to as convention-depender and rule-dependent behavior. The distinction between these types of behavior has been studied extensively in the linguistic literature (Franke al.; Brochhagen, 2020). In this work, we focus on the less-explored setting of implicit communication, and provide concrete approach to learn representations for these two different types of behavior.
In the context of multi-agent or human-AL AI collaboration, we would like our Al agents to adapt quickly to new partners. Al agents should be able to flexibly adjust their partner-specific convention-

---

Published as a conference paper at ICLR 2021
Rule representation
P
Convention representation
Repeated Interactions
player chess
time
Friendly Rock Paper Seissors
Convention Dependence
Hligh
Figure I: Partners form conventions in collaborative task through repeated interactions. An Al agent can adapt quickly to conventions with new partners by reusing shared rule representation g and learning new partner-specific convention representation g". Certain collaborative tasks, such as friendly Rock-Pape--Scisorss convention-dependenn than others, such 4-player chess.
dependent behavior while reusing the same rule-dependent behavior to simplify the learning problem just like how humans quickly adapt when playing basketball with new friend, without the need to re-learn the rules the game. We would also like our Al agents to coordinate well on similar tasks when paired with the same partners just like how humans coordinate better when playing new sport but with an old friend.
Although many existing works similarly recognize conventions as important factor of successful collaboration, they do not focus on separating conventions from rules of the task. This means that adapting
partner can be as difficult learning new task altogether. For example, existing techniques that emphasize modeling the partner's 's policy, such as theory of mind (Simon, 1995; Baker al., 2017: Brooks Szafir, 2019) or multi-agent reinforcement learning (Foerster al., 2018), attempt to model everything about the agent's ' belief of the partner's 's state and policies. Such belief modeling approaches very quickly become computationally intractable, as opposed to solely focusing on the relevant conventions developed with partner.
To address the challenges above, we propose framework that explicitly separates convention- dependent representations and rule-dependent representations through repeated interactions with multiple partners. After many rounds of solving a task (e.g. playing basketball) with different partners, an Al agent can learn to distinguish between conventions formed with specific partner (e.g. pointing down signals bounce pass) and intrinsic complexities of the task (c.g. dribbling). This enables us to leverage the representations separately for fast adaptation new interactive settings.
In the rest of this paper, we formalize the problem setting, and describe the underlying model of partners and tasks. Next, we present framework for learning separations between rule and convention representations. We show that, under some conditions, our rule representation learns a sufficient statistic of the distribution human interactions to test our hypothesis - that partners can carry over the same conventions across tasks indeed holds for human subjects. Finally, we show the merits of our method collaborative tasks: contextual bandit, block placing, and a small scale version of Hanabi (Bard et al., 2020).
best-response strategies. We then
study human-
RELATED WORK
Convention formation has been studied under the form of iterated reference games (Hawkins et al., 2017: 2019), and language emergence (Mordatch & Abbeel, 2018; Lazaridou al., 2017). these works, partners learn to reference objects more efficiently by forming conventions through repeated interactions. But in these tasks, there is little intrinsic task difficulty beyond breaking symmetries.
In multi-agent reinforcement learning, techniques such self-play, cross-training, or opponent learning (Foerster et al., 2018) have been used to observe emergent behavior in complex, physical set- tings (Nikolaidis & Shah, 2013; Liu et al., 2019; Baker et al., 2020). In addition, convention formation has been shown in tasks like Hanabi (Foerster et al., 2019; Hu & Foerster, 2019), Overcooked (Carroll al., 2019; Wang al.. 2020), and negotiation tasks (Cao al., 2018), where agents learn both how to solve a task and how to coordinate with partner. But, these works qualitatively analyze emergent conventions through post-hoc inspection of the learned policies, and do not learn representations
2

---

Published as a conference paper at ICLR 2021
Task 1
Conventions on Task 1
Conventions carried over to Task 2
11 02 03 04
A
A
B
B
B
Figure 2: Collaborative contextual bandit task. On the left we see a task with 2 contexts represented by rows and 4 arms represented by cells. At each context, the prize if both partners pull the same green arm is 1, and O otherwise. In the middle, we see a possible convention that two partners may converge to: pulling an in context A, and pulling 02 in context On the right, when the task changes in context A stays the same in context B, we expect the same two partners to continue to pull dg in context B.
that separate conventions from rule-dependent behavior. This makes it difficult to leverage learned conventions when adapting to new partners. More recently, an approach was proposed to design agents that avoid relying on conventions altogether; instead, the agents aim for unambiguous but potentially sub-optimal strategics (Hu et al., 2020).
Other approaches for studying conventions include game theory (Lerer & Peysakhovich, 2019) and theory of mind (Simon, 1995; Rabinowitz et al., 2018). Pragmatic reasoning (Goodman & Frank, 2016) is another framework that aims to estimate beliefs over the partner's states and policies, but all these frameworks that rely on modelling beliefs quickly become intractable. Instead of approximating these belief modelling frameworks (Monroe & Potts, 2015), we instead focus on learning representations for rules and conventions. Our work shares similarities with the paradigms of meta-lcarning (Schmidhuber. 1987; Finn et al., 2017) or multi-task learning (Caruana, 1997), if we view collaboration with different partners as new but related tasks. However, we are also interested in how conventions persist across tasks with similar symmetries.
More related to our work is perhaps modular policy networks (Devin et al., 2017), which learns policy modules and composes them together to transfer across robot arms with different degrees of freedom and across different tasks. However, their approach relies on input cleanly split between background observations of the task, and robot's internal observations. We do not assume such a split the input; our approach tries to learn the separation between rules and partners with one input channel.
3 PRELIMINARIES
We begin by formally defining our problem setting as two-player Markov Decision Process (MDP).
Two-Player MDP We consider a two-agent MDP with identical payoffs, which is a tuple {S, (Ae, Apl, P. R). S is set of states, Ag is set of actions for agent e, Ap is set of ac- tions for agent In general, e represents the ego agent (that we control), and p represents agents who partner with e. : the current state and actions of all agents, and R S x S > R is the real-valued reward function. Since we are interested in repeated interactions, we consider finite-horizon MDPs. A policy i is stochastic mapping from a state to an action. For our setting, our policy π = (Te, Tp) has two parts: Te : S + Ae and Tp* S > Ap. mapping the state into actions for agent and p respectively.
Ae Ap x S + [0, 1] is the probability of reaching state given
We also consider collaborative tasks that are partially observable in our experiments. In addition, some tasks are turn- based, which can be handled by including the turn information as part of the state, and ignoring the other player's 's actions. Here, we focus on tasks with discrete state and action spaces.
We describe a collaborative task to ground our discussion around adapting to new partners, and adapting to new tasks with old partners. Consider a contextual bandit setting with contexts, actions, and a prize for each action. The two players each independently pick an arm to pull, and they score prize(a) points if they both picked the same arm a; otherwise, they score 0 points. An example is depicted in Figure 2, where green boxes represent arms with prize 1, the rest have prize 0, and red and blue circles show the player's actions.
Running Example: Collaborative Contextual Bandit
In Task of Figure 2, context A only has one good arm 01 while the context B has two good arms 02 and aq. After repeated rounds of playing, two agents can eventually converge to coordinating convention that chooses one of the two good arms in context B (c.g. selecting the lefumost good arm a2). When the task shifts slightly as shown in Task 2 but context B remains the same across the two
3

---

Published as a conference paper at ICLR 2021
tasks, we reasonably expect the partners to adhere to the same convention they developed for context B of Task when playing context B of Task 2.
There are two axes of generalization at play. First, for fixed task we would like to learn the underlying structure of the task (c.g. green arm locations) to quickly adapt to a new partner when playing the same task. Second, for a fixed partner would like to keep track of developed conventions to quickly coordinate with this partner on a new task. In the next sections, we define the notion of a new task and new partner, and propose a framework effectively generalize across both dimensions.
MODEL OF PARTNERS AND TASKS
We consider a family of tasks that share the same state space, action space, and transition dynamics (all of which we refer to as the domain), but can differ in the reward function of the underlying MDP. We also consider family of partners that our ego agent would like to coordinate with. The underlying model governing the actions taken by partner at state of given task can be seen in Figure 3. For a new task, a new reward function is sampled from OR, while the domain of the MDP is fixed. For a new partner, we sample from 0n a new function which maps the Q-function We refer to p as the convention of partner. In other words, the action a of a partner at state of task depends on the partner's conventions and (indirectly through the Q f-function state s) the reward of the task. 9, the distribution over conventions and can correspond to, for example, the distribution over emergent conventions from Al-agents trained from self- play, or can be derived from human partners.
partner
p
OR
Rs
state of the task to an action.
task t
state 8
Figure Generating process of part- ner's actions for a two-player MDP. For each task we first sample reward func- tion R for the MDP. This determines the Q-function at each state. For each partner at each state s of task f, we sample function of that maps the Q- function at state to an action a!"
More formally, for a fixed domain let (ap) =
maxo, Q"(s, des ap). where Qt is the optimal Q-function for the two player MDP with reward R (Boutilier, 1996; Oliehock et al., 2008). In other words, Q is the Q-function from the partner's 's perspective state in the task with reward R assum- ing best response by the ego agent. Then, the convention of partner P S RlApl -> Ap determines
the partner's 's action at state s, given by p'(s, ).). For example, we might expect the distribution of actions across different partners at a state to follow Boltzmann rationality (Ziebart et al., 2008): Ep-e, [p(s,, (8, )+) (p]] x exp(Q (ap)). Lastly, we assume that behavior at different states are uncorrelated: a choice of action at one state tells us nothing about actions at other states.
Given this formulation, number of partners on the same task. For example, if many sampled partners all make the same action Op at state s, it is likely that (app) is much higher than the second best action, i.e., the optimal action is dependent on the rules of the task as opposed to the arbitrariness of partner strategies. Therefore
can hope to learn the underlying structure of task by playing with
partner will likely take action Op as well. Additionally, when coordinating with the same partner across different tasks, we can expect to see developed conventions persist across states with similar Q-functions. In particular, if Q Q for two different tasks with reward functions R1 and R2, then ps(s, ² = p'(s, ?2 so partner will take the same action at state across the two tasks (e.g. context B in Figure 2 across Task and 2).
We note that the roles of partners and tasks in our model are asymmetrical, assuming rational partners. task can completely determine the actions of all partners at states with one good action (e.g. context A in Figure 2), whereas partner cannot blindly pick the same action across all tasks. This asymmetry reflected our learning framework and our setup of adapting to new partners and new tasks.
5 LEARNING RULES AND CONVENTIONS
With the goal of adapting to new partners and tasks in mind, we propose a framework aimed at separating the rule representations associated with task from the convention representations
4

---

Published as a conference paper at ICLR 2021
8p
gf(a
nf(als)
g g'(a ()g)((ll))
2
g
Z
state
g'
partner module
5 () g((a (a
task module
Z,
g((als)
Figure 4: Policy network with separate task/partner modules to separate rule/convention representations. The task module g maps the state observations to action distribution g (a|s) and to latent variables z. Fach partner module maps another action distribution g (a|z). The policy of the ego agent when playing with partner i is set to be proportional to the product g (a and of (a|z).
associated with partner. At training time, we assume access to task and a set of partners sampled from the same partner distribution 8p. When adapting to a new partner sampled from 6p, we would like to learn a new convention representation but keep the learned rule representation. When adapting to new task with the original partners, we would like the opposite: learn new rule representation but keep the learned convention representation. In our setup, the ego agent knows the identity of the tasks and partners, and the goal to maximize the total reward with a new task and partner (the combination of which may be new the ego agent). Naturally, this points towards modular architecture that enables integrating new combinations of partner and task representations.
We propose the architecture shown in Figure 4, where learn to play with each of the training partners via reinforcement learning. Each rectangular module represents a multi-layer perceptron, and each box with grey bars represents distribution When training with multiple partners on the same task, the policy network uses one single shared task module g¹ and uses one partner module of for each partner i. The task module takes the state observations as input, and outputs latent variables z and an action distribution g' (alss) Then, each partner module takes as input, and outputs an action distribution gf (a]z). Here, of (a|z) does not represent partner i's action distribution, but rather represents our ego agent's 's action distribution in response partner The policy π; of the policy network for partner set to proportional the product of the action distributions.
actions in the action space of the MDP.
T (:(a))
Z: (a(ss) (al=))
As it stands, many possible combinations of task and partner module outputs could lead optimal action distributions Ty (a)s) vi. To encourage the task/partner modules to learn the rule/convention representations respectively, we include a regularization term that pushes g (aaas) to be the marginal best-response strategy over partners at state In practice, this amounts to minimizing the Wasserstein distance between the task module's 's output and the average best-response strategies n;.
D(s) = D ga(as)
Σ (als))
aEA
By pushing the task module 9 to learn the marginal best-response strategy. can cleanly capture the rule-dependent representations of the task, to used when adapting to a new partner. fact, under some assumptions, we can show that g² is learning the optimal representation, it is sufficient statistic of the distribution over best-response strategies across possible partners.
In partieular, for fixed task let [s, de: ap) represent the probability of the ego agent taking action de using its best-response strategy to partner action ap at state So, Σ., f(s, ae ap) = 1. We say the ego agent is deterministic if can only take on deterministic strategies: Vs, ap ae [(s,, Ge 0p) 1.
Lemma 1. For a deterministic ego agent, the marginal best-response strategy across partners is sufficient statistic of the distribution of best-response strategies across partners.
Next, we lay out the learning procedure in more detail in Algorithm 1. We use the task module g' and the partner module of when playing with partner i. We push the task module to learn the marginal best-response action by adding the loss term D top of standard policy gradient (PG) loss term.
5

---

Published as a conference paper at ICLR 2021
Algorithm 1: Learning Separate Representations for Partners and Tasks
Input: MDP M, n partners P1..-Pr Pn Op
Output: Policy network modules for adapting to new partners from 0p and new tasks Initialize the task module g" and n partner modules ,....,
2 G,T < Do.g..... ........hhh... Ih number of iterations
for j 1,
for + 1, do
Collect rollouts = (s, a. using {9',9;) with partner Pt M Update G with loss I (g , g. ,. [D]][((s)]
Return: G
partner, reuse the task module g along with a reinitialized partner module, and train both modules. At states with only one optimal action a" a*. As
Adapting to new partner When adapting
regardless of the partner's convention, the marginal best-response strategy will concentrate such, the task module g will immediately push the ego agent to choose a", improving adaptation time. In contrast, at states with many possible optimal actions depending on the partner's conventions, the marginal best-response strategy will spread the action distribution among the possible optimal actions, allowing the ego agent to explore amongst the optimal options with the new partner.
task When faced with new task. would like recognize the parts of the task where conventions with old partners on an old task carry over. In particular, in states for which the joint Q-function has not changed, our partners models will take the same actions before, and our ego agent can employ the same response as before. With this in mind, suppose we have trained task module g" and task. We can learn to coordinate paired with partner modules gf:-. ...% with frozen parameters, At states where the joint Q-function has not changed, the same latent variable outputs 2 will work well for all partners, so the task module will output the same and actions of the ego agent will not change. Then, we can coordinate with partners to 2n in zero-shot manner by pairing y with partner modules d 1 .9...........
Coordinating
2n partner modules of ........ P. with new task with partners 1 to n by fine-tuning the task module g",
of partners on an old
Score: 1st Try
Train
Test
A
A
0.5
B
B
Train-A
T
Train-B Test-B
Train-C T
C
Figure 5: User study on the collaborative contextual bandit task. Participants were asked to coordinate the Train task and, immediately afterwards, with the partner the Test task (shown the right). Some contexts have two equally optimal actions (green boxes), requiring symmetry breaking. On the left we plot the score, averaged across participants, of the first try at cach context. Our participants were able to coordinate well on Test-C on the first try, even though the context requires symmetry breaking.
5.1 UNDERSTANDING CONVENTIONS
When adapting to a new task with the same partner, we are relying on the hypothesis that our partners will carry over the same conventions developed on an earlier task, at states where the Q-function has not changed. Here, we would like to test this hypothesis and study and when conventions persist across tasks, through study of homann-human interactions in the collaborative contextual bandit task.
Conventions in Human- Human Interactions. We conducted a study with 23 pairs of human partners (46 subjects), using the collaborative contextual bandit task from Section 3. The study was conducted via an online interface, with partners in separate rooms. The actions of partners revealed each other the end of each try, and the partners had other form of communication.
User Study:
Independent Variables: We vary the prize of the arms at different contexts, i.e., which arms are good coordinate on. Pairs of partners were given tries coordinate 3 different contexts (as shown

---

Published as a conference paper at ICLR 2021
in Fig. 5). We then ask the same pair of partners coordinate the Test task with 3 new contexts. Dependent Measures: We measure the score of each pair's s first try at each context. The maximum score in each run for each context is I, corresponding to the two partners picking the same good arm. Hypothesis: Our hypothesis is that partners can coordinate well on the test task by carrying over over conventions developed
the train task.
Results; On the left of Figure 5 we plot the average score of the first try (zero-shot) of our participants solving each context in each task. In the Train task (grey bars), the partner pairs are coordinating for the first time. As expected, they generally had no trouble coordinating on Train-A (since there is only good option), but scored lower on Train-B B and Train-C (since there are 2 good options each).
Immediately following the 5 trials on the training task, the partner pairs coordinated on each context of the Test task (orange bars in Fig. 5), to see if they can leverage developed conventions on a new task. On Test-A, they did well since there is only one good option. However, they coordinated much better on Test-C compared to Test-B, even though both contexts have two good options. The main difference between the two contexts is that Test-C shares the same set of optimal actions (ag and a4) as Train-C, whereas Test-B does not share the same set of optimal actions as Train-B.
Overall, this study suggests that human partners can successfully carry over conventions developed through repeated interactions with their partners, and further coordinate better when carrying over conventions across tasks states where the set of optimal actions are similar (e.g. the higher gap for same context such as context C across the train and test tasks).
6 EXPERIMENTS
We experiment with our approach on three coordination tasks: collaborative contextual bandit, collaborative block placing, and Hanabi. We show results on adaptation to new partners for fixed task, and zero-shot coordination with same partners on new task. We compare with 1) a baseline (BaselineAgg) that learns a shared policy network to play with all training partners, using the average of the gradients with each partner and 2) a baseline (BasclineModular) that similarly uses a modular policy approach with separate modules for each partner, but does not explicitly represent the marginal best-response strategy, and 3) First-Order MAML (Nichol et al., 2018). 8). For our method, we vary À from 0.0 to 0.5: higher value pushes the task module output to more closely match the marginal action distribution. We also test the use of a low dimensional (the interface between the task and partner module). In general, it is unclear if using a low dimensional = is helpful. The partners in our experiments were either generated via self-play or were hand-designed to use varying conventions.
find that our method with regularization performs the best, and that
Oracle
A = 0.5
0.3
A = 0.0
low-dim
BaselineAgg
BaselineAgg EarlyStopping
Madellne
FOMAML
unoage ostuedo roward
30
10B
154
300
350
20
10C
150
203
25n
stages
(a) Arms 2
(b) Arms 3
MAN00A0 ostondo sowword
230
150
(c) Arms
(d) Arms User Study
Figure 6: Contextual bandit game: adapting single new partner. In Figures 6a-c, we train and on hand-designed partners. "Arms m²" refers to a task having m contexts with symmetries (exact task details in the Appendix). In Figure 6d, we train and test on partner policies derived from the data collected in our user study, and we use the Train task shown in Figure 5.
7

---

Published as a conference paper at ICLR 2021
0.5
0.3
0.0
BaselineAgg
BaselineAgg EarlyStopping
Distance
1.5
1.16
1:28
1.02
1
0.5
.46
.44 .46
.16
003 .01
Arms
Arms
Arms
Blocks
Figure 7: Wasserstein distance between task module output and marginals assuming uniform preference over optimal actions. Lower is better. We do not compare with BasclincModular since it does not have output that can be interpreted as marginals. The left three sets of bars are for the contextual bandit task, and the right set of bars are for the blocks task. We omit Hanabi since it is not easy to determine which actions are optimal. It is interesting that even without regularization À 0.0, the distance ground truth is still lower than the baselines, suggesting the model is learning some level of task speci-ie representation just due to the architecture.
Contextual Bandit We study the collaborative contextual bandit task described in Section 3, using 4 contexts and arms. We study variations of the task by altering the number of contexts with more than one good option (i.c. symmetries). We write "Arms m to refer to a task having m contexts with symmetries coordination should be easier for tasks having fewer contexts with symmetries.
In Figure 6a-c we show results on adaptation to new hand-designed partners (Figure 11 for self-play partners), varying the number of contexts with symmetries. We also plot the green oracle curve, by using our learning framework along with oracle best-response marginals (assuming uniform distribution over optimal actions across partners). To see if our method learns the correct marginals, we measure the Wasserstein distance between the learned marginals and the oracle marginals, in Figure 7. Using À = 0.5 produces task module outputs that closely match the oracle marginals. To test zero-shot coordination with the same partners on new tasks, we tweak the task by altering contexts with only one good action, keeping the contexts with symmetries the same (similar to the change in Figure 2). We use hand-designed partners to make sure that partners use the same conventions unchanged contexts. In Figure 10 we see that our method outperforms MoselineModular.
We include an additional plot in Figure 6d, where we instead use the Train task from the user study as shown in Figure 5. We use the data collected in our user study to create the partner policies, in lieu of the hand-designed partners. We observe similar trends that the modular learning framework with marginal regularization performs the best.
Block Placing Next, we study a collaborative block placing task which involves 2 goal-grid, with a single red block and a single blue block on the grid. The goal is to reconstruct the goal-grid from blank working- gordi, with partner 1 (P1) only controlling a red block and partner 2 (P2) only controlling blue block. The partners take turns moving their blocks on the working trrkking-gid,,, and can see each other's actions on the working-grid However, only P1 can see the goal-grid configuration; P2 only receives information from the working-grid Each game lasts 6 turns, with P1 taking the first turn. Specifically, on Pl's 's turn, can move the red block to one of the four cells, remove it from the grid, or pass the turn (similarly for P2 blue block). Blocks cannot be placed on top of each other. The partners share rewards, and earn 10 points for each correctly placed block, for a maximum of 20 points per game. P1 needs to both place its red block correctly, and communicate the position of the blue block to P2. As such, success in this task requires a mixture of rule-dependent and convention-depeedent skills.
In Figures 9a and 9h we plot the results on adaptation to new hand-designed and self-play partners, where our method outperforms the baselines for non-zero values of X. To verify that our task module is learning good task representation, we compute the distance between the task module output and the oracle marginal (Figure 7). We see that our approach with À 0.5 leads to a smallest gap from the oracle marginals. Finally, similar to the contextual bandit task, we test zero-shot coordination with the same partners on new tasks in Figure 10. We tweak the task by changing the colors of target blocks, and use hand-designed partners to ensure the developed conventions persist. Again, our method performs better on the new task/partner combination than Baseline-Modulrr We provide more details of the experimental setup in the Appendix.
Hanabi Finally, we experiment on a light version of the collaborative card game Hanabi, which has been studied in many related works in MARL. We focus on the two-player version of Hanabi, with color and 5 ranks (for a maximum score of 5). Since it is not straightforward how to create a
8

---

Published as a conference paper at ICLR 2021
Goal
Visibility
Working Grid
Pl
P2
Start
Figure 8: Block placing task: each row displays new round the task. the left, we see the goal-grid and how it appears to each player. Since P2 cannot see the goal-grid, we show a fully grey grid. On the right-side we see the working grid evolve over the course of 6 turns. P1 edits the red block on turns 1, 3, 5 and P2 edits the blue block turns 2, 4, 6.
variety of hand-designed partners, or tweak the task while maintaining the same symmetries, we use self-play partners only and omit experiments on coordinating on new task with the same partners. The results on adaptation to new self- eelf partners are shown in Figure 9c.
À 0.5
À 0.0
low-dim z FOMAML
BaselineAgg
BaselineAgg EarlyStopping
Baseline
sterage opioode reward
apormit opptooti reward
3:75
2:50
3.25
(a) Blocks: hand-designed partners.
(b) Blocks: self-play partners.
(c) Hanabi: self-play partners.
Figure 9: Adapting to a single new partner for block placing task and Hanabi.
À = 0.5
BasulineModular
20
Score
Score
16
0.5
12
Arms
Arms
Arms 3
Blocks
Figure 10: Zero shot performance on new task/partner combination. Higher is better. Orange refers to our method with À = 0.5 and grey refers to BaselineModular. Arms means the task has contexts with symmetries. We create the task by altering contexts where there is only good action (as result, omit Arms 4 since contexts with only one good action). We do not compare with BaselineAgg since non-mon non-modular and cannot be composed for new task/partner combinations.
7 CONCLUSION
study the problem of adapting to new settings in multi-agent interactions. To develop Al-agents that quickly adapt
partners and tasks, introduced framework that lears rule-dependent and convention -enetddenddeeedeent representations. Our AI agents can adapt to conventions from new partners without re-learning the full complexities of task, and can carry existing conventions with same partners on new tasks. We run a study of human-human interactions that suggests human conventions persist across tasks with similar symmetries. Our experiments show improvements in adapting to new settings on a contextual bandit task, a block placing task, and Hanabi.
ACKNOWLEDGMENTS
This work supported by NSF (#1941722, #2006388), and the DARPA HICON-LEARN project.

---

Published as a conference paper at ICLR 2021
REFERENCES
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In 8th International Conference on Learning Representations, ICLR, 2020.
Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4): 1-10, 2017.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et The Hanabi challenge: new frontier for AI research. Artificial Intelligence, 280: 103216, 2020.
Craig Boutilier. Planning, learning and coordination in multiagent decision processes. In Proceedings of the 6th conference on Theoretical aspects of rationality and knowledge, pp. 195-210, 1996.
Thomas Brochhagen. Signalling under uncertainty: interpretative alignment without a common prior. The British Journal for the Philosophy of Science, 71(2):471-4)) -466, 2020.
Connor Brooks and Daniel Szafir. Building second-order mental models for human-robot interaction. arXiv preprint arXiv: (ee.//999900005 2019.
Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z. Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. In 6th International Conference on Learning Representations, ICLR, 2018.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems, pp. 5175-5186. 2019.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Herbert H Clark. Using language. Cambridge university press, 1996.
Herbert H. Clark and Deanna Wilkes-Gibbs. Referring as collaborative process. Cognition, 22: 1-39, 1986.
Coline Devin, Abhishek Gupta, Trevor Darrell, Picter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2169-2176 IEEE, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ArXiv, 703.03400, 2017.
Jakob Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-leaming awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, 2018.
Jakob N. Foerster, H. Francis Song, Edward Hughes, Neil Burch, Iain Dunning, Shimon Whiteson, Matthew M Botvinick, and Michael H. Bowling, Bayesian action decoder for deep multi-agent reinforcement learning. Proceedings of the 36th International Conference on Machine Learning. ICMI., 2019.
Michael Franke et al. Signal to act: Game theory in pragmatics.
Noah D Goodman and Michael C Frank. Pragmatic language interpretation probabilistic inference.
Trends in cognitive sciences, 20111):188889, 2016.
Robert X. D. Hawkins, Megumi Sano, Noah D. Goodman, and Judith E, Fan. Graphical convention formation during visual communication. In Proceedings of the 41st Annual Meeting of the Cognitive Science Sociery, 2019.
Robert XD Hawkins, Mike Frank, and Noah D Goodman. Convention Convention-formation formation in iterated reference games. In Proceedings the 39th Annual Meeting of the Cognitive Science Society, 2017.
10

---

Published as a conference paper at ICLR 2021
Hengyuan Hu and Jakob N Foerster. Simplified action decoder for deep multi-agent reinforcement learning. International Conference on Learning Representations, 2019.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-play" for zero-shot coordination. In Proceedings of the 37th International Conference on Machine Learning, ICML, 2020.
Fereshte Khani, Noah D Goodman, and Percy Liang. Planning, inference and pragmatics in sequential language games. Transactions of the Association for Computational Linguistics, 6:543-555, 2018.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In 5th International Conference on Learning Representations, ICLR, 2017.
Adam Lerer and Alexander Peysakhovich. Learning existing social conventions via observationally augmented self- In Proceedings of the 2019 AAAI/ACM Conference on Al, Ethics, and Society, pp. 107-114, 2019.
David Lewis. Convention: A philosophical study. 1969.
Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. Emergent coordination through competition. In 7th International Conference on Learning Repre- sentations, ICLR, 2019.
Will Monroe and Christopher Potts. Learning in the Rational Speech Acts model. In Proceedings of 20th Amsterdam Colloquium, Amsterdam, December 2015. ILL.C
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Thirty- -Scron- AAAI Conference on Artificial Intelligence, 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02500 029992 2018
Stefanos Nikolaidis and Julie Shah. Human-robot cross-training: computational formulation, mod- eling and evaluation of a human team training strategy. In 2013 8th ACM/IEEE International Conference on Human-Robi Interaction (HRI), pp. 33-40. IEEE, 2013.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353. 2008
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Eslami, and Matthew Botvinick. Machine theory of mind. In Proceedings of the 35th International Conference on Machine Learning, JCML., 2018
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3. https: https://githtb. http:///gt........ottteeeeet wtt////o-////////////// baselines3, 2019.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning. 1987.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06444 2017.
Herbert A Simon. The information- information-proceesinng theory of mind. American Psychologist, 1995.
Peter Stone, Gal A Kaminka, Sarit Kraus, Jeffrey S Rosenschein, et al. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Proceedings of the Twenty-Fourt AAAI Conference on Artificial Intelligence, AAAI, 2010.
Rose E Wang, Sarah A Wu, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman- Weiner. Too many cooks: Coordinating multi- agett collaboration through inverse planning. In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2020.
Brian D. Ziebart, Andrew L. Maas, J. Bagnell, and Anind K. Dey. Maximum entropy inverse rein- forcement learning. In Proceedings of the Twenty-Thine AAAI Conference on Artificial Intelligence, AAAI, 2008
11

---

Published as a conference paper at ICLR 2021
A PROOF OF LEMMA
Proof. The marginal of the best-response strategy across partners at state of a task with reward R can be written as:
p(ae) E, [f(s, de, p(s, Q!!)]
When the ego agent is deterministic, its best-response strategy [s,, de. ap) can be rewritten function I' where ["(s, ap) = de [(s, de; Up) = 1. Then, we can write likelihood best-response strategy from newly sampled partner function of the marginal:
de) =((ae)
Loosely speaking, when the best-response strategies not deterministic, storing only their marginals throws away information. But in the case when the strategies are deterministic, each strategy can be represented single action, simply showing that storing their marginals does not throw away information.
B ARCHITECTURE DETAILS AND HYPERPARAMETERS
The specific size of each module in Figure 4 is shown in Table 1. We train the policy network using Proximal Policy Optimization (PPO) (Schulman al., 2017) with the Stable Baselines software package (Raffin et al., 2019). For the state-value function, we use a network of the same size, and combine the value predictions from the task module and the partner module by adding them together.
Bandit
Blocks
Hanabi
Task Module Layer Size
30
80
500
Partner Module Layer Size
30
80
500
Low Dimensional 2
5
20
50
Table Architecture: each module is 2-layer fully connected MLP with ReLU activation. Values the table denote the size of each layer. We set the number of latent variables z outputted by the task module to be the same as the layer size in all cases except for the low dimensional setting, where we use the dimensions reported here.
Bandit
Blocks
Hanabi
Timesteps
10000
lc6
5e5
Minibatch size
16
40
160
Num. epochs
20
10
Learning Rate
3e-4
3e-4
3e-4
Table Hyperparameters for training self-play partners.
Bandit
Blocks
Hanabi
Minibatch size
16
160
160
Num. epochs
50
5
Table 3: Hyperparameters for adapting to new partners.
C EXPERIMENTAL SETUP
Code for the experiments in our paper is available at https://github.com Stanford-
C.I CONTEXTUAL BANDIT
Task setup We use a contextual bandit with 4 contexts and 8 arms. We label the contexts 1,.... .4
and the arms 1, 1, 8. In the initial setup, context the optimal actions (green boxes) are arms
12

---

Published as a conference paper at ICLR 2021
Bandit
Blocks
Hanabi
Inner iterations
4
4
4
Inner learning rate
3e-4
3e-4
3e-4
Outer step size
0.25
0.25
0.25
Table 4: Hyperparameters for First-Order MAML.
and 4. So, all contexts have symmetries, since there are multiple equally optimal actions. We experiment with different versions of the task by changing the number of contexts with symmetries, from 4 down to 1. To get m contexts with symmetries, we set the optimal action context to only So, contexts : have two equally optimal actions, and contexts fm++1, mn+1.....
arm i, if have only one optimal action. Naturally, when there are fewer contexts with symmetries, coordination easier.
Partner setup We use both partners obtained by self-play and hand uand-desiged partners. For the self- play partners, train using PPO with the hyperparameters shown in Table 2. For the hand-designed partners, the partners were designed to pick any one of the equally optimal actions and to stick to the same choice. The distribution over choices across the hand-designed partners made to be uniform over the optimal actions. We used set of 10 self-play partners and hand-designed partners for training, and set of the same size of testing.
Tweaked task For coordinating on new task (Figure 10), we change the optimal action for contexts where there no symmetries. As such, this is only possible when the number of symmetries m is less than 4. For the new task, a context where i > m, we modify the optimal action from to i 1 The contexts with symmetries remain unchanged. For the hand-designed partners, we ensured that they stuck to the same choices (conventions) as the initial task in the contexts with symmetries.
C.2 BLOCK PLACING
Task setup We used the task setup described in Section 6. In our experiments, the ego agent is P1 and the partner agents P2.
Partner setup We use both partners obtained by self-play and hand-designed partners. For the self-play partners, we train using PPO with the hyperparameters shown in Table 2. For the hand- designed partners, we design the partners to interpret the first action (turn 1) of the ego agent based on fixed permutation of the block positions. For example, a possible partner may always place the blue block top-right whenever the ego agent places the red block top-left on turn 1. We used a set of self-play partners and 3 hand-designed partners for training, and set of self-play partners and hand-de Rend-designed partners for testing.
Tweaked task For coordinating on a new task (Figure 10), we change the reward function by reversing the interpretation of the red and white cells in the goal grid. In other words, P1 has to place the red block on the working grid one of the two positions that is a white cell on the goal grid. P2's goal is unchanged; P2 still has place the blue block where the blue block appears in the goal grid, so the relevant symmetries remain unchanged. For the send-designed partners, we ensured that they stuck the same permutations (conventions) the initial task when interpreting the ego agent's action on turn
C.3 HANABI
Task setup We used the Hanabi Learning Environment package (Bard et al., 2020), with the following configuration: color, ranks, 2 players, hand size 2, information tokens, and 3 life tokens. The maximum score is points.
Partner setup We only partners obtained by self-play, For the self-play partners, train using PPO with the hyperparameters shown in Table 2. We used a set of 4 self-play partners for training, and set of the same size for testing.
13

---

Published as a conference paper at ICLR 2021
C.4 ADDITIONAL PLOTS
Oracle
A - 0.5
À - 0.3
À - 0.0 FOMAML
low-dim Z
BaselineAgg
BaselineAgg EarlyStopping
BaselineModular
opicodo reward
average episnde reward
average apioode reward
03b
0.90
9.7
0.00
9.1-
0.5
0.8
stops
stops
steps
(a) Arms 2
(b) Arms 3
(c) Arms 4
Figure 11: Contextual bandit game: adapting to single new partner. Similar to Figure 6, but instead we train and test self-play partners.
C.5 USER STUDY INDIVIDUAL RESULTS
Here we provide more detailed results from our user study. For each context and task we report joint actions made by each pair of users on their first and on their fifth try.
Train-A
Test-A
Train-B
Test-B B
Train-C
Test-C
ID
Try I
Try 5
Try
Try 5
Try
Try 5
Try
Try 5
Try
Try 5
Try
Try 5
1
1/1
1/1
3/3
3/3
3/3
3/3
2/1
1/1
2/2
2/2
2/2
2/2
2
I/1
1/1
3/3
3/3
3/3
3/3
1/2
1/1
2/2
2/2
2/2
2/2
3
1/1
1/1
3/3
3/3
4/3
3/3
2/1
1/1
4/2
2/2
2/2
2/2
4
1/1
1/1
3/3
3/3
3/3
3/3
2/2
1/1
2/2
2/2
2/2
2/2
5
1/1
1/1
3/3
3/3
4/4
4/4
1/2
2/2
4/4
4/4
4/4
4/4
6
I/1
1/1
3/3
3/3
3/3
3/3
2/2
1/1
2/2
2/2
2/2
2/2
7
I/1
1/1
1/1
3/3
3/4
3/3
1/2
1/1
2/2
2/2
2/2
2/2
8
1/1
1/1
3/3
2/2
1/1
2/2
2/2
9
1/1
1/1
3/3
3/3
3/3
3/3
2/2
1/1
2/2
2/2
2/2
2/2
10
1/2
1/I
1/3
3/3
2/2
4/4
4/2
2/1
3/3
2/4
2/2
2/4
11
I/I
I/I
3/3
3/3
3/3
3/3
I/I
1/1
2/2
2/2
2/2
2/2
12
I/1
1/1
3/3
3/3
4/3
3/3
1/1
1/1
2/4
2/2
2/2
2/2
13
I/1
I/1
3/3
3/3
4/3
3/3
2/1
1/1
4/2
2/2
2/2
2/2
14
1/1
1/1
3/3
3/3
4/3
4/4
2/2
2/2
4/2
4/4
4/4
4/4
15
1/1
1/1
3/3
3/3
4/3
3/3
1/2
2/2
2/2
2/2
2/2
2/2
16
1/1
1/1
3/3
3/3
4/3
4/4
2/2
2/2
4/3
4/4
4/4
4/4
17
I/1
I/1
3/3
3/3
4/4
4/4
2/2
2/2
4/2
2/2
2/2
2/2
18
I/1
1/1
3/3
3/3
3/4
3/3
1/2
1/1
2/2
2/2
2/4
2/2
19
1/1
1/1
3/3
4/4
2/2
2/2
4/2
4/4
4/4
4/4
20
1/1
1/1
3/3
3/3
4/3
3/3
2/3
2/2
2/2
2/2
2/2
2/2
21
1/1
1/1
3/3
2/3
2/2
4/2
22
1/1
1/1
3/3
3/3
3/4
4/4
1/2
2/2
2/2
2/2
2/2
2/2
23
1/1
1/1
3/3
3/3
1/1
1/1
4/2
2/2
2/2
Table 5: Detailed individual results from contextual bandit user study. representing the actions made by the two partners. The task is shown in Figure 5.
Each cell form 01/02
14


Reviews:
[{"review_id": "zM-Um462ct", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "==== On the Critical Role of Conventions in Adaptive Human-AI Collaboration ====\n##### Summary #####\nThis paper studies how artificial agents can be endowed with the human-like capability to, one the one hand, retain behaviors that best fit the task environment(s) when no equally good alternatives exist and, on the other, transfer arbitrary partner-specific conventions to other tasks. Addressing this challenge is important. Chiefly, it promises to improve the number of iterations needed to converge on optimal behavior for cases where analogous strategies were already converged on in a different task / with the same partner. This work proposes to achieve this by combining two separate learned representations. One for partner-specific behaviors. The other for the task itself. The latter module, reused across partners, ends up regulating behavior for cases where only one optimal actions exists. When multiple optimal actions exist there is some slack for players to explore and converge on an arbitrary optimal action. In this sense, agents are endowed with the ability to reuse optimal action policies that will work across agents as well as to reuse optimal partner-specific policies when faced with a context with multiple solutions.\n\n##### Reasons for score #####\n\nI ultimately decided for rejection. This work has many merits: the topic is very important; it is of relevance to many fields; the approach is sound and the experiments interesting. However, I fail to see how much this work advances our understanding of the interplay between partner-specific and task general behavior. The main reason is that, while sound and straightforward to follow, it leaves open a lot of crucial questions (e.g., How do we recognize what a task/context is? Is the separation of the modules motivated? If so, how? What do we learn from the human experiment that we didn't before? See \"Cons\" below for details). I'm very happy to be convinced otherwise but I don't think that these concerns can be addressed in the present submission.\n\n##### Pros #####\n+ Interesting and important topic that is relevant to many fields\n+ Technically sound approach and clear exposition. \n+ Comprehensive experimentation. I really liked that the approach was put to the test across agents, human and artificial, and tasks (Contexual Bandit, Block Placing; Hanabi)\n\n\n##### Cons #####\n\n- Goals: My main issue is that the goals of this study are unclear. This work would be greatly enhanced by clarifying what they are and what is ultimately achieved. I do not think its true or fair to state that, as the authors put it in the abstract, \"current approaches have not attempted to distinguish a task and the conventions used by a partner, and [that] more generally there has been little focus on leveraging conventions for adapting to new settings\". To name just one of many examples, in the linguistics literature this distinction is standardly made and has long been studied (e.g., by Clark & Wilkes-Gibbs, 1986; Clark 1996; Hawkins et al. 2017, all cited in this work). This is also how semantics vs. pragmatics is defined in game-theoretic approaches to language use & dialog (e.g., Franke 2009, \"Signal to Act\"; or Brochhagen 2017, \" Signalling under Uncertainty: Interpretative Alignment without a Common Prior\"), as well as how it is generally understood within Gricean pragmatics. In other words, the distinction has been made and rests on a long philosophical tradition. I therefore don't think that taking this separation seriously alone is enough to motivate this investigation.\n\n- Further motivations: In a similar vein to the point above, there's a lack of detail on what sets this work apart from previous investigations. For instance, the critique that \"[...]\" all these frameworks [...] quickly become intractable\" is not very strong in light of the existence of approximations and solutions for the models mentioned (e.g., Monroe 2018's \"Learning in the Rational Speech Acts Model\", which also uses neural networks to model rule-dependent behavior).\n\n- Clarification: In what sense are Train and Test different tasks? (Section 5)\n\n- Analysis: The experiment in Section 5.1 averages across participants. I'd suggest looking at / reporting individual-level variation. These kinds of experiments usually vary a lot from individual to individual (see, e.g., Kanwal et al. 2017, \"Zipf’s Law of Abbreviation and the Principle of Least Effort\"). Population-level averages can therefore inadvertently hide or misrepresent what subjects are actually doing.\n\n- On page 4, the authors state: \"we assume that behavior at different states are uncorrelated: a choice of action at one state tells us nothing about actions at other states.\" I take it that this assumption is made for simplicity's sake. However, isn't this also an integral part of human-like abilities? If I realize my partner's behavior accords --or does not accord-- to some conventions I had already established, I might as well behave accordingly. For instance, if I'm playing a game of chess with someone I might match their level of expertise (e.g., avoid castling with a complete beginner); and if I'm speaking with someone at a conference, I might change my phrasing based on theirs based on what I believe to be their background to be. \n\n##### Questions during rebuttal period #####\nSee cons above\n\n##### Updated reviewer #####\nThe authors have addressed my main concerns satisfactorily, in a clear and concise matter. I have updated my recommendation to reflect this.\n\n##### Minor comments #####\n- Figure 6 has a different y-axis across plots. This is confusing at a fast glance, and makes the subplots' comparison quite hard.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IzjP4DTMtcL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes that in human-AI collaboration using deep neural nets, the AI agents we train should separate learning the _rules_ of the environment from the _conventions_ used to coordinate with humans in that environment. It proposes a simple method to do so: learn a single task-specific module that is always used, as well as many partner-specific modules that are used with specific partners. Intuitively, when trained with multiple partners, the task-specific module should learn heuristics that work across partners (the environment-specific rules) while the partner-specific model should learn personalized heuristics for each partner (the conventions). They further incentivize this by regularizing the task-specific module towards the average of the partner policies.\n\nQuality: The one qualm I have is that the environments studied are relatively simple (though even 1-color Hanabi is a fairly challenging coordination problem). Other than that the paper is high quality. I especially appreciated the user study.\n\nClarity: I found the paper reasonably clear, though some parts took some time to understand.\n\nOriginality: To my knowledge, this is the first paper exploring the distinction between rules and conventions within the deep RL paradigm, and it shows good results both in simulation and with real humans in a user study (albeit in a very simple toy domain).\n\nSignificance: Human-AI collaboration is clearly important and significant, and the application of deep RL to human-AI collaboration has grown in the last 2-3 years. This paper extends this field with an important contribution.\n\nThe main weakness of the approach I see is that it doesn’t have an obvious way to handle the fact that humans will typically _adapt_ to whatever policy the robot plays. This may not happen in the simple environments considered in this paper, but definitely does happen in larger environments. Perhaps this technique would work anyway: arguably, an adaptive human just means that the convention changes, and simply continuing to train the partner module could be enough for the robot to adapt to this change in the human’s convention.\n\nRegardless, I think even the contribution of how to deal with multiple different non-adaptive humans is significant and relevant to ICLR.\n\nQuestions for the authors:\n\n1. Why does the partner module operate “on top of” the task module? Why not instead have both modules take in the state as an input and produce a probability distribution over actions, that are then multiplied together?\n2. How might this extend to collaboration with adaptive humans?\n\nTypos / nitpicks:\n\nThe phrase “ego agent” was confusing to me, and I think it wasn’t explained anywhere? I did eventually figure it out though.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GbUX4VGLOgn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper makes the observation that when performing cooperative tasks with partners, there are two components to learn: how to perform the task, and how to coordinate with the partner according to conventions. Therefore, it proposes to separate these two components via a modular architecture, which learns a task-specific action distribution that attempts to marginalize out all possible partner conventions, and a series of partner-specific action distributions. The agent's own policy is the product of these two distributions. When coordinating with a new partner, the partner-specific component is learned from scratch using a pre-trained task-specific component, and vice versa. \n\nThe paper goes after an ambitious and useful problem (rapid adaptation to coordinating with novel partners on new tasks), and proposes a novel technique for doing so. A weakness is that the paper does not use reasonable baselines, and effectively compares only to ablations of their own model. Why not compare to a meta-learning technique? Or compare to some of the existing SOTA methods for Hanabi? \n\nIn general the paper is well written, but it could be made significantly clearer by providing further details on how the partner action distributions g^p_i(a|s) are obtained. Given the explanation in the beginning of Section 4, I was initially under the impression that these represented the partner's policy distribution produced by its Q-values, or perhaps the partner's actual action frequencies obtained from observing trajectories. However, it seems that the model is learned entirely end-to-end, and so these distributions actually represent how the agent's own policy should be modified according to which partner it is playing with. Is this correct? If so, this explanation should be added to the paper to make it more clear how the technique can apply beyond simple domains like the contextual bandit, in which agents must choose the *same* actions as the partner.\n\nThe fact that the partner module must be re-initialized and learned from scratch for each new partner is a weakness of the method.  Why not learn some type of partner embedding that would enable generalization to new partners at test time that use similar conventions to training partners?\n\nThe experiments section of the paper felt rushed and lacking in explanation compared to the first 6 pages. The clarity/impact could be enhanced by explaining the experiments in more detail. In particular, the block placing task is not explained (do agents place blocks separately? do they have to place a block together at the same time?). Also, the need for \"hand-designed\" partners is not explained, nor is what they are hand-designed to do. \n\nSince the paper collects a human user study on conventions, why not test how well the trained models are able to coordinate with humans? This would significantly enhance the impact of the paper. \n\nOther suggestions:\n- Figure 2 caption does not include the explanation that agents must choose the same action to get a reward\n- A legend should be added to Figures 7, 8, and 9. \n- Figure 7 is interesting in that even without the Wasserstein distance penalty (when lambda=0), the Wasserstein distance to the ground truth marginal best response is still low, suggesting the model is learning some level of task-specific representation just due to the architecture. This could be explained further in the text. \n\nEdit: I have updated my score based on the new experiments added during the rebuttal process. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "6ODjbVUx3M1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an interesting model to study multi-agent interactions, in uncertain environments. In s nutshell, the model proposed consists of a MDP (Finite state, action, horizon) and two players playing simultaneously (the turn-by-turn model can be subsumed by the simultaneous move model as stated in the paper). In the absence of learning, the finite MDP has an optimal solution. The key contribution of the paper is to focus on instances when the optimal solution is not unique. In a two-player model, this requires symmetry breaking in order for a sample path of the MDP to track the optimal trajectory. \n\nThe above is when the MDP and rewards are all well known. The setting in the paper concerns a learning situation where some or all of the components of the MDP is unknown. In this case, the agents must learn the MDP, while breaking symmetry (coordinate) in converging on a sample path closest to an optimal one. The problem, is very interesting and the authors propose a nice formulation to study these questions.\n\nI have a few high-level suggestions to the authors.\n\n1. The model description is mathematically imprecise. Are the agents aiming to optimize the total reward collected, in presence of unknown model and partner ? What information about the partners are known to the agent ? (Is the agent distribution common information ? ) Are the partners assumed to know the underlying MDP, or they are also simultaneously learning ? If the partner are also simultaneously learning, is their \"state of knowledge\" at the beginning known to the ego agent ?\n\nI understand that having a robust solution to all of the above problems is perhaps too hard. Nevertheless, clarifying the precise setup mathematically will greatly aid the reader. \n\n2. The human experiment, were the users able to communicate to each other in any way ? Were they total strangers to each other, or known acquaintances ? Clearly specifying the \"conditions of the experiment\", can help take the results in context.\n\nOverall, I believe the paper is attempting to study an interesting (and hard) problem. But my (low) rating is based on the clarity of the presentation.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new paradigm for learning to perform cooperative tasks with partners, which factors the problem into two components: how to perform the task and how to coordinate with the partner according to conventions. The setting is new and the reviewers are excited about the paper. A clear accept.
Decision: Accept (Poster)