Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach for zero-shot coordination that learns to assign actions to observations in a semantically meaningful way. The authors show that semantic actions have a better inductive bias leading to increased consistency in cross-play. Their proposed method is a learning based approach where an attention model jointly processes featurized representation of the observation and the action. The evaluations show that the agents produce human-compatible policies on a simplified Hanabi card game play. 

main_review: The paper presents a nice conceptual overview about the need for human compatibility of actions associated with observations in zero shot coordination scenarios. They list the problems that arise when statistically independent actions are assigned to observations. The resulting disconnect not only affects human interpretability, but also lacks consistency during cross-play in cooperative games such Hanabi.

The proposed approach for zero-shot coordination learns to assign actions to observations in a semantically meaningful way. To produce semantic actions, the action feature is embedded along with the observation to predict contextualized action. The method itself is not new, and is a standard technique employed in many language and vision tasks such as matching models, where the query is concatenated with the featurized responses before passing them through a classification network that predicts the responses. It is surprising that the idea has not been explored in POMDP networks before, and a more comprehensive related work section could help in understanding why this is the case.

The paper points to several logical primitives used in the game play by humans, such as exact match, similarity, implicatures such as mutual exclusivity and similarity + exclusivity. There could be several other strategies used during game-play. How many of these strategies are actually learnt by the algorithm? How did the authors find one-to-one correspondence between the human designed strategies, and the statistically learnt strategies? 
A related question is how does the model choose diverse strategies at different times during the game-play. There are no regularizations on policy selection, so one can assume that the model might collapse to learn a single simple strategy e.g., exact match instead of the diverse logic plays. 

Given the the action featurization is weak (just a one-hot vector), how does the method ascertain that the action features are not leaking into the contextualized policy prediction? This looks likely when we look at the clustered results from Action-In experiments, where the model shows two modes - exact similarity and exact dissimilarity. Would the model still work (or probably work better) if higher order embeddings are added to reduce the actual feature distance between input action and output?

The action-observation relationship is achieved by concatenating the features before passing them through an attention model. The paper does not explain why concatenation + attention module is an ideal choice for modeling the logic, or if any competing designs are worth exploring. Could the features be modeled differently such as within a Bayesian framework, or through deep generative models?  A few methods are mentioned in the related work, but none of them are used as a baseline technique.



summary_of_the_review: Strengths:
1. Topical: Looks into a relatively new problem of generalization during cross play after training using self-play.
2. Strong conceptual overview and motivation with examples.
3. Use of attention based embedding for contextualized policy prediction seems to be new in POMDPs, although it has been used in vision and NLP applications before.

Weaknesses:
1. Weak literature review with very general coverage of the problem space.
2. Lack of strong baselines - The related work mentions Bayesian/causal reasoning based approaches that are considered too expensive. It might help to show the performance-accuracy tradeoff of those methods vis-a-vis currently proposed approach.
3. The quantitative evaluations are not well explained / justified. E.g., 
  (a) why does the baseline DQN (simplest model) perform remarkably well for the same hand condition, while the proposed approach has significantly worse accuracy? 
  (b) Why does the proposed approach always converge into two clusters of related cross-plays?
  (c) How is the diversity of the learnt policies measured? 

Overall, although the problem itself is novel, the treatment is not convincing enough and does not meet the bar for an ICLR acceptance yet.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper exploits semantic relationships between features of observations and an action for zero-shot coordination in multi-agent reinforcement learning. It extends decentralized partially observable Markov decision process (Dec-POMDP) by representing observations and actions with shared features. Technically it uses attention mechanism of deep learning to exploit the semantic relationships between observations and actions. Besides, the paper develops a novel human interpretable environment for analysis.
main_review: Strengths:
1) The paper is well organized. The idea and method are clearly presented. The experiments are conducted in a novel environment (proposed Hint-and-Guess game).
2) The paper proposes a novel method to extend Dec-POMDP with shared feature representations for both observations and actions.
3) The paper develops a novel Hint-and-Guess game for evaluating effects of different deep learning architectures.

Weaknesses:
1) Some details of the experiment setting may need further explanation. It is not clear why the paper fixes hand size N=5 (and feature size) for the game, and how different hand sizes may affect the performance.
2) More analysis of the results is needed. Why the 13 agents tend to make two clusters with two groups of coordination types? Why these agents form different clusters for the two versions of full game and same hand condition?
3) What would your method work in other zero-shot coordination games other than your proposed game?
summary_of_the_review: In summary, the problem tackled in the paper sounds interesting and the idea and method are reasonable and easy to follow. While more explanations and analysis are preferred for the experiments, my current rating leans to acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a new Zero-shot Coordination (ZSC) task which involves a Hinter and a Guesser and a set of explicit features shared between the two. The paper evaluates several methods and conclude that an attention mechanism that put attention on both observation and action features is helpful to produce agents that are successful in cross play.
main_review: Strengths:
* The introduction is nicely written and introduces ample motivation from multiple backgrounds.
* The proposed task is very clearly described, as well as scenarios of interests / challenges.
* Experiment visualization is very informative, showcasing interesting dichotomy of self-play vs cross-play performances.
* The experiments seem to have enough details and reproducible.

Weakness:
* I feel like section 3 and 4 a bit verbose. Even for me, someone not in the specific field of multi-agent RL, the text seems a bit redundant.
* Notation in section 4 is not really clear, with some notation overloading / ambiguity. For example, the number of players n is overloaded with each object's number of feature n. 
* No baseline in ZSC is considered. In related work, it's mentioned that prior work needs some experimenter-coded input. But I don't see why similar baseline can't be examined here, either as a baseline or an upper-bound.
* The motivation behind having same-hand setup is unclear to me.
summary_of_the_review: Though I'm not an expert in multi-agent RL, I think the work will be interesting to the field. 
The paper offers an interesting study showing that action and observation information should both be leveraged to form interesting communication / coordination strategy.
However, the paper is limited by the features are explicitly available, making strategy generation less impressive.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores several architectures for reinforcement-learning in the
context of a newly proposed card game (Hinter-Guesser).
A number of the architectures seem to have poor performance on the game.
Moreover, whatever they learn is not compatible with what other learners learn,
so that they cannot mutually understand Hints & Guesses. The stated aim is:
"to build an agent with the capacity to develop these kinds of abstract correspondences during
self-play such that they can robustly succeed during cross-play or during play with people."
The architectures generally succeed (70-90%) at playing against themselves (with the shared learned
policy) but perform badly when playing with independently trained agents,
except when trained with the desired action as part of the input (unclear how
this can be tested on unknown cases when 'answer' is given as part of the
inputs).


main_review: 
The paper shows that in the case of the proposed new game, that
"attention-based architectures that jointly process a
featurized representation of the observation and the action" performed
better. The evidence shown supports this claim.

This paper is quite far from my expertise, and there are a number of
confusions I had about the paper:
- What is special about the new hint-guess game? Does it feature some aspect that
  the games used in current experiments ignore?
- Why is it called zero-shot coordination, when there is considerable training
  using training data that seems virtually identical to the proposed task?
- What can we generalize from the game/experiments that is valuable for other
  tasks? In particular, given the goal of having 'human-compatible' learning,
  how is this addressed?
- Figure 5 shows the  probability of proposed 'hints' given
all possible target 'answers'. Is is surprising that they are largely uniform, given that any
possible card could be the appropriate answer depending on the configuration of the other
cards?

- In Fig 5, I don't see the connection between the card-selection probabilities
and the conclusion of 'private languages'. They seem to be 2 different questions and
I don't see how the results in Fig 5 connects them.
- Section 8.3 says: "We find that the agents in cluster 1 of Action-In demonstrate
coordination patterns that are nearly identical to the human-interpretable response."
There are no human-based experiments, and I'm unclear what the evidence is that supports
this claim even if there was some human data.
- The final discussion says "relies on a majority vote". I did not see any mention of
a voting mechanism previously.
- Should the first formula on page 6 be S_k = instead S_i =
- Do any of the observations from the analysis of the Hint-Guess game apply to other
collaboration games?
- Why were 5 cards chosen as the configuration? Do results vary with other numbers?
- What is the intuition behind each proposed attention mechanism?

summary_of_the_review: The proposed approach works for this bespoke game. It would be good to have more evidence that the approach is generally useful.
review_title: null

Meta-review: The author response addressed some reviewer concerns, and generally reviewers increased their scores. However, there are important, and unanswered concerns about the generalization of the model. The discussion raised the concerns that despite the paper claim of "a specific class of higher order reasoning" emerging, the result suggests relatively simple strategies. This might not be a limitation of the approach, but of the evaluation scenario. So, this either requires a more nuanced view of the findings, and further empirical evidence to support the claim.
Decision: Reject