Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper identifies weaknesses in Off-belief Learning [ICML 2021] that lead to large distribution shifts at test-time evaluation in zero-shot coordination (particularly when evaluated with human players.) The paper contributes two extensions (off-team belief learning and off-team reinforcement learning) which together improve cross-play evaluation and zero-shot coordination with another agent (rank bot) and human proxy (clone bot).
strengths_and_weaknesses: The proposed methods are original extensions to an increasingly specific approach to zero-shot coordination in Hanabi. However, the diminishing returns in improvements to this system and limited scope of its evaluation weaken the significance of this finding. Most notably the improvement in ad-hoc teamplay evaluation with rank-bot (line 326) is insignificant when considering the variance of both methods evaluated and the effect size of the improvement in proxy human-AI coordination (line 327) is tiny. Furthermore, no evidence is given in this paper that the clone bot used for proxy human-AI coordination is a good human proxy.

The overall clarity of the contribution made is significantly weakened by the paper's over reliance on references to "Off-belief Learning" [Hu et al. ICML 2021] causing the paper to not be a standalone reference to reproduce the work. For example (but not limited to) the zero-shot coordination task or cross-play metric used throughout the paper are not formally defined. For details on the critical process of training the policy and belief model readers are deferred in line 209 to "follow practices in the original OBL paper [Hu et al. ICML 2021]" and the core contributions of this paper (OT-BL and OT-RL) are then defined just by how they differ to this past work. 

The specific details of how the proposed extensions differ from the past work are also incomplete and potentially incorrect in parts:

a) In Figure 1 left should the arrows into \pi from \tau_t and \tau_t+1 be reversed or the same as OT-BL (Figure 1 Right)? 

b) In Figure 1 Right, the blue fictitious transitions use \pi in the illustration and \pi_L in the caption? Algorithm 1 also uses \pi and not \pi_L. Is it just the caption on Figure 1 that is incorrect?

c) The loss in Equation 1 is not defined.

d) Is it intentional that the only difference in Figure 2 left (OBL) and Figure 2 right (OT-RL) is the policy used to sample real actions?

e) On line 273, does P_{pi_L} mean both players play using policy \pi_L?

questions: During the discussion period, I'm particularly keen to hear from the authors on the following prioritized list of questions that could improve my current rating of this paper:

1) What additional evidence can the authors provide that clone-bot is a good human proxy?

2) Assuming clone-bot is a good human proxy, what evidence is there that the small improvement in proxy human-AI coordination (line 327) is a significant advancement?

3) Please clarify the issues (a)-(e) regarding specific details raised in the section on strengths and weaknesses above.

4) Please justify the very strong claim in the abstract that "Zero-shot coordination is a ... pre-requisite for human-AI coordination".
limitations: The paper includes a sufficient broader impact section. However, the closing sentence regarding "working in purely cooperative settings" is a short sighted conclusion. I encourage the authors to consider the issues raised in Section 5 of "Open Problems in Cooperative AI" Dafoe et al. Arxiv 2020.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces the concept of off-team learning, where a target joint policy is learned using a behavior joint policy to address covariate shift in off-belief learning (OBL). First off-team belief learning (OT-BL) is proposed to address covariate shift, where the belief model is iteratively updated on fictitious transitions. Second, off-team reinforcement learning (OT-RL) is proposed to address test time covariate shift with respect to unseen agents, where Q-values can be learned with belief models of any distribution. Both approaches are evaluated in the Hanabi challenge. The belief model of OT-BL is shown to produce more valid fictitious samples than OBL, while OT-RL is shown to outperform OBL in various settings.
strengths_and_weaknesses: - **Originality**: The idea and concepts presented in the paper are novel, interesting, and highly relevant to current applications and challenges of multi-agent systems.
- **Quality**: The paper is technically sound. The explanations and experiments are presented well and the results clearly support the claims. 
- **Clarity**: The paper is well-written, clearly structured, and easy to follow. 
- **Significance**: The experiment design and results indicate a clear improvement over OBL.

The authors did a great job on motivating and presenting their approach. Since a fundamental issue is addressed, a smaller toy problem like the cooperative communication game from the OBL paper would have been helpful.

Minor comments:
- The tables should highlight the best performing result in boldface for better readability.
- Text and labels in Fig. 1 and 2 are very small and hard to read when printed.
questions: None
limitations: Limitations and potential negative societal impact are adequately addressed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper investigates the problem of off-policy learning in cooperative games. Within this area, they address a problem they call "off-team (OT) learning", where the teammates that generated the experience differ from the target teammates. They accomplish this by disentangling the belief in the trajectory, specifically the previous joint actions, from action evaluation. As a result, a policy can be learned that may adapt to a target team through its belief maintenance. Their algorithm is evaluated in a variety of Hanabi settings and shows some empirical improvements.
strengths_and_weaknesses: **Strengths**
 - Their algorithms are described in the text, graphics, and pseudocode allowing for diverse learning styles to approach the material.
 - Evaluations include some amount of ablations/sensitivity-analysis to better understand the merits of the proposed algorithm.

**Weaknesses**
 - The paper is dense with jargon and heavily marked symbols with detailed meanings making it more challenging to read than I believe is necessary. 
 - Maintaining and correcting for beliefs as a way to make strategic adjustments has been in many flavors, making this work's originality small. 

questions: 1. "Off-team learning" relies on the learned belief model bein accurate up to time t-1. This seems unrealistic in practice to rely on as an assumption when the algorithm is jointly learning it alongside the policy. Can the authors comment on the need for accuracy in the belief model both in terms of the theoretical guarantees of the algorithm and the empirical performance? 

2. Coordination games may exhibit many diverse equilibria. How does this algorithm cope with target teammates that play equilibrium vastly different than that expressed by the behavioural teammates?

3. The authors claim L224: "The goal of ZSC is to evaluate the consistency of an algorithm across different seeds." This is not true. The goal of ZSC is to coordinate productively with unseen opponents. If the authors' statement was true any deterministic algorithm solves this problem.  

4. In L236 the authors discuss the robustness of their algorithm given covariate shift with respect to the environment dynamics. However, it seems to me, that is mostly orthogonal to the work. In the abstract it seems the focus is on only differing teammates. Shouldn't the emphasis be on covariate shift to radically different teammates? And/or the combination of the two changes? Could the authors explain this focus and why they de-emphasized these other facets? 

5. L283, could the authors explain "only a small number, o=3200, of trajectories"? How many trajectories are 3200 observations? Why is it measured in observations? 
 
limitations: The broader impact section could be improved to reflect the impact belief-modelling style algorithms can have on biases.
review_title: null

Meta-review: metareview: The authors propose an improvement to off-belief learning, Off-Team learning, which closes the gap between belief models trained on fixed policies, and evaluation on learned policies for ZSC coordination problems. All reviewers have voted to weak/borderline accept - since I see no conceptual issues with the proposed framework and the evaluation seems sound, I will also vote to accept. The major area of constructive criticism, is that the work seems to be somewhat incremental with respect to Hu et al. (ICML 21). 

confidence: Less certain
Decision: Accept