Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper suggests an interesting idea of real-world actions often being grounded by observations and the shared action-observation features naturally emerging for coordination between agents. This paper hypothesizes that the policy architecture is crucial for emerging the shared action-observation features. The experiments are done using the simple hint-guess game and demonstrate that the sharable action-observation features emerge only when each action is fed into the Q-network.
strength_and_weaknesses: 
### Strengths

* This paper points out an interesting perspective of learning shared action-observation features for coordination that can be used commonly across independently trained agents.

* The investigation of multiple different Q-network architectures shows clear evidence of the necessity for action representations in the cross-play experiments.

* The evaluation of human subjects further proves that the learned strategies are compatible with humans.


### Weaknesses

* The problem formulation in the paper suggests a novel and interesting research direction. However, the example tasks seem to be too simple and not well connected to the realistic or practical use of the emerged (shared) action representations. Thus, one thing that can further strengthen the paper would be additional discussion or experiments with more realistic or practical scenarios.

* The experiments are conducted on the one-feature scenarios and two-feature scenarios. It might be interesting to see whether the findings from these scenarios hold for a more number of features. It might be also interesting to discuss whether the same hypothesis holds for more than two agents.
clarity,_quality,_novelty_and_reproducibility: * The paper is well-motivated and clearly written.

* In Figure 4 and Figure 6, "A2I" needs to be "SA2I".

* The proposed problem and experimental setup are novel.

* The experiments are well-organized and thoroughly analyzed. But, the experiments are done in a simplified task, which makes the conclusion weaker.

summary_of_the_review: This paper proposes an interesting and potentially important problem for the community. However, the evaluation is done in too simplified tasks, which makes the paper not very convincing. I still value the novelty of the paper, so leaning toward weak acceptance. A few more tasks with sufficient complexity would make this paper very strong.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work investigates the learning of intuitive, interpretable policies that leverage features shared between observations and actions, by means of a novel two-player game (Hint-Guess) where such relationships play a pivotal role. Several different model architectures are evaluated, and they are found to demonstrate widely varying abilities. One particular attention architecture (SA2I) far outperforms the others on this task, and its policies are shown to take advantage of the task structure in ways that one would intuitively expect.
strength_and_weaknesses: **Strengths**

As motivation, the paper carefully argues for the importance of interpretable policies, in particular with respect to the exploitation of semantic relationships. As one example of such relationships, the Hint-Guess game (analogous to a type of card game) is cleverly designed to test an agent’s ability to take advantage of relationships between observations and actions. The experimental results show that SA2I is significantly more capable than the other models at this task. Additional studies and game variants characterize the gains in some detail, and even show that the best agents perform well when paired with human players.

**Weaknesses**

There would be great value in an agent architecture that demonstrated it could reliably learn interpretable policies, and one can imagine a wide variety of tasks involving semantic relationships. But no single task (such as the Hint-Guess game) by itself can provide strong evidence that a particular model’s ability to learn interpretable policies would apply to a broader set of tasks. So one major weakness is the paper’s consideration of just a single task and modest variants of it.

In addition, given the nature of the Hint-Guess game, it is hard to imagine any successful policy that did not rely on the strategy that humans find obvious here. Despite this expectation (that only one particular policy could succeed), the winning agent manages to find two winning policies, one of which (Dissim) is far less intuitive than the other (Sim). This experimental outcome raises doubt that the SA2I architecture has any general ability to learn interpretable policies.

In fact, there is a simple, alternative explanation for why attention should be expected to outperform simple MLPs on this task. In the Hint-Guess game, each observation contains unordered sets of objects (the cards in each hand). Since attention-based models (like transformers) are set processors by design, it is no surprise that applying attention to the observation would outperform an MLP. This alternative explanation calls into question the paper’s claim that the SA2I model “has a strong inductive bias toward using the relationship between actions and observations in intuitive ways.”

The term “self-play” creates confusion. The term “self-play” is commonly understood (for games like Chess or Go) to mean that an agent is playing against a copy of itself, but that does not appear to be the case in this work. For instance, “we train agents in the standard self-play setting… where the hinter and guesser are jointly trained to maximize their score”. It appears that in this paper, self-play vs. cross-play simply indicates whether an agent was trained with the (non-self) agent it is evaluated with. But it’s hard for a reader to be sure without a clearer explanation. Similarly, in the intra-AXP setting, what is the algorithm being shared?

Several crucial aspects of the Hint-Guess experiments are never adequately explained. For instance, the hinter and guesser are apparently two separate agents. But is one agent trained to always be the hinter, while the other agent is always the guesser? If so, do the hinter and guesser share any weights? Or is each agent trained in both roles, sometimes as the hinter and sometimes as the guesser? If so, then how does the agent know in any particular game whether it is playing as the hinter or as the guesser? Is this information included in the observation?

Important details of the observations are left out. For instance, “the observation input is a sequence of card representations for both hands $H_1$ and $H_2$, as well as the representation of the target card, $C_i^2$ (for the hinter) or the hinted card $C_j^1$ (for the guesser)”. Since the hand size is 5 for the main experiments, this could mean that each agent will see 11 cards at once:  2 hands plus a copy of either the target or the hinted card. Or are only 10 cards visible, and the target or hinted card is specified some other way? And how does the agent know which cards are its own? Perhaps the first 5 cards are always the agent’s own cards, or maybe some flag or embedding is applied to indicate ownership. Or is this information conveyed some other way? 

The block diagrams in Fig. 3 are helpful, but they also raise important unanswered questions. When multiple Q values are output, how are those values mapped to specific cards, given that the order of the cards is randomly permuted on input, and the Reduce Mean operation loses order information? Does each Reduce Mean operation average a set of input vectors to produce a vector of the same dimensionality as the inputs? And exactly which vectors serve as the inputs? 

Finally, since this work proposes a novel task, the baseline results would be more meaningful if the baseline model hyperparameters had been tuned. But hyperparameter tuning is never mentioned.

clarity,_quality,_novelty_and_reproducibility: **Clarity**

As explained above, critical aspects of the work are never clearly explained. Apart from this, much of the paper is clear and well-written.

In Fig. 4, it appears that the first 6 agents chose the Sim strategy, and the last 7 agents chose the Dissim strategy. Apparently the agents were intentionally grouped together according to the strategy they chose, but I don’t see where the paper says this. Also in Fig. 4, shouldn’t the final plot be labeled SA2I instead of A2I?

**Quality**

Much of the work is carefully executed, but I cannot fairly assess the significance of the Hint-Guess results without clarifications of the points mentioned above.

**Novelty**

The Hint-Guess game appears to be new and distinctive. The architectures tested are simple and composed of very standard components.

**Reproducibility**

The eventual release of the source code will help, but won’t make up for the paper’s current lack of critical technical details.

summary_of_the_review: The paper makes an interesting claim, but does not provide enough evidence to support it. 

==== After discussions with authors ====

The authors have supplied many important details that were missing from the original paper. But in my view, the most serious limitation of the work continues to be its reliance on a single, particularly simple task of no practical importance. This limitation was also called out by the other reviewers. 

In addition, I still cannot see anything particularly surprising about the main results on this task, presented in Table 1. As noted by the paper, and confirmed by the authors in our extensive discussion, the poor performance of MLP and Attn is easily explained by the fact that they “do not consider action features at all”, which are crucial factors in this task by construction. The only remaining question is whether self-attention (SA2I) or cross-attention (CA2I) is more appropriate to the task. Since cross-attention is a less-costly special case of self-attention, it is no surprise that self-attention has an advantage. (The cross-play setting is the important one, as it factors out the possibility of learning a private language.)

For these reasons, and despite the commendable work that this paper represents, I have to leave my score unchanged. I look forward to extensions of this work on the important connection between semantic relationships and interpretable policies.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigates whether different model architectures leverage similarities between features of the observation and the action like humans do. It proposes a communication game where the agent can reference to observation by choosing actions whose features imply features of the observation. The authors found a self-attention architecture that produces a policy that is similar to that of humans. Experiments show that the policy can coordinate successfully with humans without ever being trained with them. 

strength_and_weaknesses: Strengths:
* This is a very interesting paper. The topic is well motivated by studies in linguistic and socio-cognitive science. I like that the authors explain intuitive concepts using well-illustrated examples. 
* The problem of learning to coordinate with humans is an important and challenging problem. This study may lead to novel ways of solving this problem. 
* The proposed task is easy to understand and useful in evaluating the capabilities of interest.
* The experiments are well-designed. The empirical results support the claims made by the authors. 

Weaknesses:
* The authors do not offer a satisfying explanation or motivation on why the SA2I architecture produces the observed results (whereas the CA2I does not). Given that they are searching for the architecture with the right inductive bias, it is unclear what that inductive bias is? What general principles can be drawn for designing more complex architecture to tackle more challenging problems?
* The authors present the results as mere findings. I think a better way to structure the paper is to first clearly form expectations (or hypotheses) about the proposed architectures, explain why you have such expectations, and then present the results as verification of the expectations.
* The proposed task remains simplistic. It is unclear whether the obtained findings will carry over to tasks with more complex observation and action spaces.  
* The method requires a common representation for the observation and the action, which may not be trivial to construct in other scenarios.  

clarity,_quality,_novelty_and_reproducibility: The paper is quite clearly written. I would swap section 6.1 with section 5. The paper would benefit from adding motivation for the successful architecture (SA2I), and an explanation of its behavior. It took some time for me to realize that the architecture represents the hinter OR the guesser, rather the hinter AND the guesser. 

The empirical study is novel and is rigorously conducted. The findings are also novel and significant. 
summary_of_the_review: I think the paper serves as a good starting point for exploring this topic. It would motivate subsequent studies on more complex tasks. I recommend acceptance.

====After Rebuttal====

I have read the authors' response. While I appreciate the explanations, my initial concerns cannot be resolved solely by the response and would require substantial revision of the paper. I thus retain my initial assessment.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
The paper explores the emergence of shared features for observations and actions in a multi-agent coordination setting. That this is indeed the case is demonstrated on a single task where multiple variants of agents are explored.

Strengths:
- A interesting new perspective on multi-agent learning
- Exploration of architectures
- Experiments with humans

Weaknesses:
- The paper only focuses on a single task and it remains unclear how general the method is
- There are still doubts about how fair the comparison is, given that the baselines don't have access to features that are crucial for solving the task
- Requires constructing a common representation
justification_for_why_not_higher_score: All reviewers point our the same limitation of having a single the very simplified task and the arising question if the findings will generalize, which cannot be addressed without a substantive re-write.
Having a single task is understandable as it allows exploring the method in detail, but combined with the 'non-surprising' outcomes the paper is a bit too limited in its current form.
Decision: Reject