Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an interesting model to study multi-agent interactions, in uncertain environments. In s nutshell, the model proposed consists of a MDP (Finite state, action, horizon) and two players playing simultaneously (the turn-by-turn model can be subsumed by the simultaneous move model as stated in the paper). In the absence of learning, the finite MDP has an optimal solution. The key contribution of the paper is to focus on instances when the optimal solution is not unique. In a two-player model, this requires symmetry breaking in order for a sample path of the MDP to track the optimal trajectory. 

The above is when the MDP and rewards are all well known. The setting in the paper concerns a learning situation where some or all of the components of the MDP is unknown. In this case, the agents must learn the MDP, while breaking symmetry (coordinate) in converging on a sample path closest to an optimal one. The problem, is very interesting and the authors propose a nice formulation to study these questions.

I have a few high-level suggestions to the authors.

1. The model description is mathematically imprecise. Are the agents aiming to optimize the total reward collected, in presence of unknown model and partner ? What information about the partners are known to the agent ? (Is the agent distribution common information ? ) Are the partners assumed to know the underlying MDP, or they are also simultaneously learning ? If the partner are also simultaneously learning, is their "state of knowledge" at the beginning known to the ego agent ?

I understand that having a robust solution to all of the above problems is perhaps too hard. Nevertheless, clarifying the precise setup mathematically will greatly aid the reader. 

2. The human experiment, were the users able to communicate to each other in any way ? Were they total strangers to each other, or known acquaintances ? Clearly specifying the "conditions of the experiment", can help take the results in context.

Overall, I believe the paper is attempting to study an interesting (and hard) problem. But my (low) rating is based on the clarity of the presentation.
review_title: Interesting Problem - Can benefit from a more clear writing

final_score: rating: 7: Good paper, accept
review_content: This paper makes the observation that when performing cooperative tasks with partners, there are two components to learn: how to perform the task, and how to coordinate with the partner according to conventions. Therefore, it proposes to separate these two components via a modular architecture, which learns a task-specific action distribution that attempts to marginalize out all possible partner conventions, and a series of partner-specific action distributions. The agent's own policy is the product of these two distributions. When coordinating with a new partner, the partner-specific component is learned from scratch using a pre-trained task-specific component, and vice versa. 

The paper goes after an ambitious and useful problem (rapid adaptation to coordinating with novel partners on new tasks), and proposes a novel technique for doing so. A weakness is that the paper does not use reasonable baselines, and effectively compares only to ablations of their own model. Why not compare to a meta-learning technique? Or compare to some of the existing SOTA methods for Hanabi? 

In general the paper is well written, but it could be made significantly clearer by providing further details on how the partner action distributions g^p_i(a|s) are obtained. Given the explanation in the beginning of Section 4, I was initially under the impression that these represented the partner's policy distribution produced by its Q-values, or perhaps the partner's actual action frequencies obtained from observing trajectories. However, it seems that the model is learned entirely end-to-end, and so these distributions actually represent how the agent's own policy should be modified according to which partner it is playing with. Is this correct? If so, this explanation should be added to the paper to make it more clear how the technique can apply beyond simple domains like the contextual bandit, in which agents must choose the *same* actions as the partner.

The fact that the partner module must be re-initialized and learned from scratch for each new partner is a weakness of the method.  Why not learn some type of partner embedding that would enable generalization to new partners at test time that use similar conventions to training partners?

The experiments section of the paper felt rushed and lacking in explanation compared to the first 6 pages. The clarity/impact could be enhanced by explaining the experiments in more detail. In particular, the block placing task is not explained (do agents place blocks separately? do they have to place a block together at the same time?). Also, the need for "hand-designed" partners is not explained, nor is what they are hand-designed to do. 

Since the paper collects a human user study on conventions, why not test how well the trained models are able to coordinate with humans? This would significantly enhance the impact of the paper. 

Other suggestions:
- Figure 2 caption does not include the explanation that agents must choose the same action to get a reward
- A legend should be added to Figures 7, 8, and 9. 
- Figure 7 is interesting in that even without the Wasserstein distance penalty (when lambda=0), the Wasserstein distance to the ground truth marginal best response is still low, suggesting the model is learning some level of task-specific representation just due to the architecture. This could be explained further in the text. 

Edit: I have updated my score based on the new experiments added during the rebuttal process. 
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper proposes that in human-AI collaboration using deep neural nets, the AI agents we train should separate learning the _rules_ of the environment from the _conventions_ used to coordinate with humans in that environment. It proposes a simple method to do so: learn a single task-specific module that is always used, as well as many partner-specific modules that are used with specific partners. Intuitively, when trained with multiple partners, the task-specific module should learn heuristics that work across partners (the environment-specific rules) while the partner-specific model should learn personalized heuristics for each partner (the conventions). They further incentivize this by regularizing the task-specific module towards the average of the partner policies.

Quality: The one qualm I have is that the environments studied are relatively simple (though even 1-color Hanabi is a fairly challenging coordination problem). Other than that the paper is high quality. I especially appreciated the user study.

Clarity: I found the paper reasonably clear, though some parts took some time to understand.

Originality: To my knowledge, this is the first paper exploring the distinction between rules and conventions within the deep RL paradigm, and it shows good results both in simulation and with real humans in a user study (albeit in a very simple toy domain).

Significance: Human-AI collaboration is clearly important and significant, and the application of deep RL to human-AI collaboration has grown in the last 2-3 years. This paper extends this field with an important contribution.

The main weakness of the approach I see is that it doesn’t have an obvious way to handle the fact that humans will typically _adapt_ to whatever policy the robot plays. This may not happen in the simple environments considered in this paper, but definitely does happen in larger environments. Perhaps this technique would work anyway: arguably, an adaptive human just means that the convention changes, and simply continuing to train the partner module could be enough for the robot to adapt to this change in the human’s convention.

Regardless, I think even the contribution of how to deal with multiple different non-adaptive humans is significant and relevant to ICLR.

Questions for the authors:

1. Why does the partner module operate “on top of” the task module? Why not instead have both modules take in the state as an input and produce a probability distribution over actions, that are then multiplied together?
2. How might this extend to collaboration with adaptive humans?

Typos / nitpicks:

The phrase “ego agent” was confusing to me, and I think it wasn’t explained anywhere? I did eventually figure it out though.
review_title: Good paper on an important topic

final_score: rating: 7: Good paper, accept
review_content: ==== On the Critical Role of Conventions in Adaptive Human-AI Collaboration ====
##### Summary #####
This paper studies how artificial agents can be endowed with the human-like capability to, one the one hand, retain behaviors that best fit the task environment(s) when no equally good alternatives exist and, on the other, transfer arbitrary partner-specific conventions to other tasks. Addressing this challenge is important. Chiefly, it promises to improve the number of iterations needed to converge on optimal behavior for cases where analogous strategies were already converged on in a different task / with the same partner. This work proposes to achieve this by combining two separate learned representations. One for partner-specific behaviors. The other for the task itself. The latter module, reused across partners, ends up regulating behavior for cases where only one optimal actions exists. When multiple optimal actions exist there is some slack for players to explore and converge on an arbitrary optimal action. In this sense, agents are endowed with the ability to reuse optimal action policies that will work across agents as well as to reuse optimal partner-specific policies when faced with a context with multiple solutions.

##### Reasons for score #####

I ultimately decided for rejection. This work has many merits: the topic is very important; it is of relevance to many fields; the approach is sound and the experiments interesting. However, I fail to see how much this work advances our understanding of the interplay between partner-specific and task general behavior. The main reason is that, while sound and straightforward to follow, it leaves open a lot of crucial questions (e.g., How do we recognize what a task/context is? Is the separation of the modules motivated? If so, how? What do we learn from the human experiment that we didn't before? See "Cons" below for details). I'm very happy to be convinced otherwise but I don't think that these concerns can be addressed in the present submission.

##### Pros #####
+ Interesting and important topic that is relevant to many fields
+ Technically sound approach and clear exposition. 
+ Comprehensive experimentation. I really liked that the approach was put to the test across agents, human and artificial, and tasks (Contexual Bandit, Block Placing; Hanabi)


##### Cons #####

- Goals: My main issue is that the goals of this study are unclear. This work would be greatly enhanced by clarifying what they are and what is ultimately achieved. I do not think its true or fair to state that, as the authors put it in the abstract, "current approaches have not attempted to distinguish a task and the conventions used by a partner, and [that] more generally there has been little focus on leveraging conventions for adapting to new settings". To name just one of many examples, in the linguistics literature this distinction is standardly made and has long been studied (e.g., by Clark & Wilkes-Gibbs, 1986; Clark 1996; Hawkins et al. 2017, all cited in this work). This is also how semantics vs. pragmatics is defined in game-theoretic approaches to language use & dialog (e.g., Franke 2009, "Signal to Act"; or Brochhagen 2017, " Signalling under Uncertainty: Interpretative Alignment without a Common Prior"), as well as how it is generally understood within Gricean pragmatics. In other words, the distinction has been made and rests on a long philosophical tradition. I therefore don't think that taking this separation seriously alone is enough to motivate this investigation.

- Further motivations: In a similar vein to the point above, there's a lack of detail on what sets this work apart from previous investigations. For instance, the critique that "[...]" all these frameworks [...] quickly become intractable" is not very strong in light of the existence of approximations and solutions for the models mentioned (e.g., Monroe 2018's "Learning in the Rational Speech Acts Model", which also uses neural networks to model rule-dependent behavior).

- Clarification: In what sense are Train and Test different tasks? (Section 5)

- Analysis: The experiment in Section 5.1 averages across participants. I'd suggest looking at / reporting individual-level variation. These kinds of experiments usually vary a lot from individual to individual (see, e.g., Kanwal et al. 2017, "Zipf’s Law of Abbreviation and the Principle of Least Effort"). Population-level averages can therefore inadvertently hide or misrepresent what subjects are actually doing.

- On page 4, the authors state: "we assume that behavior at different states are uncorrelated: a choice of action at one state tells us nothing about actions at other states." I take it that this assumption is made for simplicity's sake. However, isn't this also an integral part of human-like abilities? If I realize my partner's behavior accords --or does not accord-- to some conventions I had already established, I might as well behave accordingly. For instance, if I'm playing a game of chess with someone I might match their level of expertise (e.g., avoid castling with a complete beginner); and if I'm speaking with someone at a conference, I might change my phrasing based on theirs based on what I believe to be their background to be. 

##### Questions during rebuttal period #####
See cons above

##### Updated reviewer #####
The authors have addressed my main concerns satisfactorily, in a clear and concise matter. I have updated my recommendation to reflect this.

##### Minor comments #####
- Figure 6 has a different y-axis across plots. This is confusing at a fast glance, and makes the subplots' comparison quite hard.
review_title: Interesting and sound approach to adaptive behavior in agents (task- and partner-specific). However, it's not clear what the studies' goals are and, as a consequence, what we learn about the object of study.

Meta-review: This paper proposes a new paradigm for learning to perform cooperative tasks with partners, which factors the problem into two components: how to perform the task and how to coordinate with the partner according to conventions. The setting is new and the reviewers are excited about the paper. A clear accept.
Decision: Accept (Poster)