Off-Team Learning
Authors: Brandon Cui
Hengyuan Hu
Andrei Lupu
Samuel Sokota
Jakob Nicolaus Foerster
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-Agent Reinforcement Learning
Reinforcement Learning
Cooperative Multi-Agent Reinforcement Learning
Deep Reinforcement Learning

Abstract: Zero-shot coordination (ZSC) evaluates an algorithm by the performance of a team of agents that were trained independently under that algorithm. Off-belief learning (OBL) is a recent method that achieves state-of-the-art results in ZSC in the game Hanabi. However, the implementation of OBL relies on a belief model that experiences covariate shift. Moreover, during ad-hoc coordination, OBL or any other neural policy may experience test-time covariate shift. We present two methods addressing these issues. The first method, off-team belief learning (OTBL), attempts to improve the accuracy of the belief model of a target policy πT on a broader range of inputs by weighting trajectories approximately according to the distribution induced by a different policy πb. The second, off-team off-belief learning (OT-OBL), attempts to compute an OBL equilibrium, where fixed point error is weighted according to the distribution induced by cross-play between the training policy π and a different fixed policy πb instead of self-play of π. We investigate these methods in variants of Hanabi.

Off-Team Learning
Brandon Cui MosaicML
Hengyuan Hu Stanford University hengyuan hatpguaaness..tttorddd.o stanford.edu
grandon@mosaicom cmm.com com
Andrei Lupu
Samuel Sokota Carnegie Mellon University ssokota @ andrew.cmu.ed.
Meta Al & FLAIR, University of Oxford alupu@meta.com
Jakob N. Foerster
FLAIR,, University of Oxford
jakob.foerster wwn.eeeeng.... ox.aouk.
Abstract
Zero-shot coordination (ZSC) evaluates an algorithm by the performance of a team of agents that were trained independently under that algorithm. Off-belief learning (OBL) is a recent method that achieves state-of-ther the game Hanabi. However, the implementation of OBI. relies on belief model that experiences covariate shift. Moreover, during ad-hoc coordination, OBL or any other neural policy may experience test-time covariate shift. We present two methods addressing these issues. The first method, off-team belief learning (OT- BL), attempts improve the accuracy of the belief model of a target policy лт on a broader range of inputs by weighting trajectories approximately according to the distribution induced by different policy Tg. The second, off-team off-belief learning (OT-OBL), attempts to compute an OBL equilibrium, where fixed point error is weighted according the distribution induced by cross-play between the training policy æ and a different fixed policy my instead of self-play æ. We investigate these methods in variants of Hanabi.
results in ZSC
Introduction
A core objective of multi-agent reinforcement learning (MARL) is to develop methods for coordinat- ing with previously unseen partners such other artificial agents or humans. In partially observable fully cooperative settings, policies that were learned through self play develop efficient but opaque arbitrary conventions that make collaboration with novel partners, or even independent training runs of the same algorithm, extremely difficult [ 4 To study this issue, Hu et al. (6) introduced the zero-shot coordination (ZSC) setting in which the goal is to maximize the intra- alga--tt cross-play (intra-AXP) F ZSC evaluates algorithms under
i.c. by independently training N groups of agents and evaluating teams made up of one member of each group. For an algorithm achieve good intra-AXe AXP, it must tend to find compatible strategies across different, independent training runs.
Off-belief learning (OBL) [7] is recent MARL algorithm designed produce high intra-AXP. OBL takes in an input policy and aims to produce a uniquely defined output policy πi It effectively
1 Work done while Meta AI (FAIR).
Note that the term "zero-shot coordination" is used ambiguously in the original paper 6 both informally and specific problem setting. To avoid confusion, in this paper we use it strictly for the technical setting.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

uses To to reinterpret past actions as if they were taken by To- When To is fully random policy, this forces n to only take actions based on grounded information. This also means that क has no incentive to communicate private information since any behavior will be reinterpreted as having come from To. OBL then iterates upon this x1 by training hierarchy where each level builds upon the conventions induced by optimal play at the level below. For each Ty, OBL trains a belief model Ba, through supervised learning, which is then used to interpret past actions when training #6+1-
While OBL achieves strong results in ZSC on Hanabi [T], there are two problems with the current formulation of OBL 7. First, the belief model B=, is trained via supervised learning on the distribution of trajectories induced by self-pla play between Te. However, is evaluated on the trajectories induced by self-play with the OBL policy at the next level, TT+1- When the distributions of xy and Te+1 differ substantially, this will bring B,, off-distribution, which may cause to be unreliable and produce undefined outcomes. Second, OBL policies may be brittle when paired with a novel test time partner, e.g. human or another Al agent. Note that the second problem is prevalent in most neural policies trained with RL against specific partners.
To mitigate the covariate shift experienced by Bid we propose off-team belief learning (OT-BL). Rather than training B., only the distribution induced by Te, we introduce a belief- selief-bootstrapping operator that allows us to train our belief model off team on the distribution partly induced by Tt+1- We evaluate OT-BL empirically and show that it ameliorates the covariate shift experienced by belief models when training OBL and outperforms vanilla OBL in a 2-life variant of the card game Hanabi.
Next, to mitigate the test time covariate shift experienced by He when paired with unseen agent, we propose off-team off-belief learning (OT-OBL), OT-OBL builds on the insight that OBL can be applied tooff team. In another word, thanks to the reinterpretation idea of OBL, on data generated from self-play games of Tı but also the data generated by pairing Ti with any other policies in cross-play mode. It allows OBL to be trained broader distributions to reduce side effects of the test time covariate shift.
not only train
2 Related Work
2.1 Prior Work on Zero-Shot Coordination
Zero-shot coordination (ZSC) was first introduced by Hu et al. [6 and further analyzed by Treutlein et al. [21] and led to a series of works meant to address the challenges of the setting. In their original paper, Hu et al. [6] introduced other-play, method for training policies that are invariant with respect to the symmetries of the game, thus avoiding conventions that rely on arbitrary symmetry breaking. Other works have trained hierarchies of policies for 7SC (5), leveraged diverse populations train robust common hest response 12 19]. and investigated the inductive bias of different architectures for leveraging action features [13 One of the most significant breakthroughs in 7SC is off-he
learning (OBL) 7 which we detail in section 3.3 and upon which our work builds directly.
2.2
Belief learning
One crucial requirement for scaling OBL to large settings is learning beliefs about the private infor- mation of other players. Belief states can be explicitly tracked. Modern approaches to this problem involve deep sequence models, such as recurrent networks [20] or transformers [24]. However, the standard training setup for these models leverages Monte Carlo sampling. In contrast, OT-BL trains belief's models via approximate dynamic programming. In this respect, OT-BL most similar to the work of Sokota et al. [16), who propose a framework for belief fine-tuning, In this framework, belief model for the current time-step is used to bootstrap a belief model for the next time-step using approximate dynamic programming, Sokota et al. [16] show that belief fine-tuning can improve the quality of belief models and adapt them to new policies at inference time. Belief fine-tuning and OT-BI. are similar in that they use bootstrapping to learn beliefs; however, whereas Sokota et al. [16) examined belief fine-tuning in an inference-time context, we present OT-BL. as a mechanism for performing oef-t train-time learning.
2

---

3 Background
3.1 Turn-Basee Dec-POMDPSs with Public Actions
This work considers turn-based decentralized partially observable Markov decision processes (Dec- POMDP) 3 with public actions. Dec-POMDPP have state space s E S, observation functions of = (s) for each agent through which they obtain partial observation of the underlying state, action space A, reward function R: S 5x A + R and transition function T: S x A + AS that defines of N players takes an action; this action is observable to all players. We further denote a trajectory as T = (81, 01,, f1,- 8t) and each players action observation history (AOH) 7. (of, " r", , When equals to terminal T, the subscript is dropped and the two notations are simplified as t and r' respectively. Each agent acts according their policy n' (alrz) that returns distribution actions given action observation history and the joint policy is simply defined as π = (Tr,.. ~*). An important quantity in
the distribution of next state given current state and joint action. At each time step, exactly
the belief function Bg (7x 7T) = P( |T σ) that returns an agent's 's belief over the distribution of the state given their AOH and the group 's joint policy.
3.2 Problem Settings
The ad-hoc team play setting's goal is to cooperate effectively with unknown teammates, where each teammate can contribute meaningfully to the task [17]. This setting has been used in numerous environments including Overcooked [4], RoboCup 2D [2), and Hanabi [5] 7) Some works leverage small amounts of data to guide agents towards equilibria or "social conventions" For experiments involving ad-hoc team play, we choose policies that are amenable to
Ad-Hoc Teamplay
11 22
meaningful coordination.
Zero-Shot Coordination and Intra-Algorithm Cross Play Training in self play has been shown to give rise to policies that employ "arbitrary" conventions. Inspired by this, Hu et al. [6] proposed a zero-shot coordination (ZSC) in which the goal to maximize intra-algorithm cross play (intra-AXP). Intra-AXP measures the cross play score between independently trained agents under the same algorithm.
Human-Al Coordination Multiple studies have evaluated Al agents in coordinating with humans, [Σ] One proxy setting for human-AI coordination is to evaluate with an agent trained on human data
including Hanabi 6 15], Overcooked 4
7 We use this proxy setting in this work.
3.3 Off-Belief Learning
Intuition. At every step, OBL reinterprets the AOH as if generated by playing with a fixed, given policy To. When To is fully random policy, past actions themselves are treated as conveying no information, since the probability of an action independent of the trajectory.
Implementation. In practice, OBL trains sequence of policies starting with a uniform random policy. Keeping track of the exact belief is computationally expensive. Thus. OBL implemented by first training a neural network belief model on xo using supervised learning. This model is used to sample fictitious trajectories T' by evaluating it on πı 's data, i.e. Tf B., (Tę (T)),, where Tr. ~ Po,(T) (T.). At every timestep I, the policy कं is queried to produce an action on the fictitious trujectory. This action is used to perform a fictitious transition, which is used to calculate the target value for training OBL. See figure 2 for a visual representation the fictitious transition.
4 Problem Setting
We define the tef-team belief learning (OT-BI) problem setting as
min
ΣΣ E E C(B( T/), Bay (- 17)))
(1)
BEO
where θ the hypothesis class belief models. In words, the goal of the tef- belief learning problem setting is to learn an accurate belief model 1 for the policy mr, where accuracy is weighted by the distribution over AOHs induced by different policy Tb.
3

---

7T
Z2
T C
22
B Fy
B RT
# t+
7/4
T
Figure 1: Visualization of belief training schemas by vamila OBL (left) and OT-BL (right). Red arrows indicate values used for belief training. We only illustrate one time step, but we train on all time steps in given trajectory, т. Left: Vanilla OBL trains BRy to approximate the true conditional distribution P(relr)) using trajectories sampled from Pay. independently for cach time step t. Right: OT-BL uses to generate trajectories and trains Buy on the distribution Pa, (72)). In order to train Brg. we train only on fictitious transitions (indicated in blue) sampled from Bxy and use xy to generate the next action.
Similarly, we define the general OT-OBL problem setting follows
min Σ E B Biarrrtt), (((T), 4E
(n [R(T,, a) E ((an)
(2)
where è is the hypothesis class of action-value functions and
i(T) max '(T', .))
where i is the acting player. In words, the goal of the off-team off-belief problem setting learning is to learn an action value function that has low OBL TD-error assuming that teammates played according joint policy गт in the past, where error is weighted by the distribution over AOHs for agent is induced by the joint policy (ml "
5 Off-Team Learning
Off-team learning is a difficult problem because it requires weighting the loss values by the reach probabilities induced by Tg. However, we can't simply naively train our learned belief model, Bzy, on data generated by Te. since this would result in belief model for na rather than Ty. Similarly, we cannot simply naively train our policy on data generated by nr because such a policy would leverage the correlations between mi 's actions and its private information. We give rough heuristics for approximating solutions for these problems under the names off-team belief learning (OT-BL) and off-team off- off-belief learning (OT-OBI), which use bootstrapping perform forward induction on the beliefs and wack induction the values (respectively), and trajectory resampling.
Say we have learned a belief model Bay up to time t 1, and a value function & for time t and thereafter. We describe off-team learning's procedure for generating data for time t in three steps.
Off-team learning samples trajectories according to the behavior policy Ts thereby guaranteeing that the samples T/_1 are weighted by the behavior distribution.
Off-team learning corrects for the fact that T(-1 was played to by my rather than тт. It does so by resampling T using the learned belief model, B., (Te-1 T/-1). taking advantage of the belief model being accurate up to time 1.
3. We propagate this trajectory forward in time by sampling an action a ~ тT(((-1) from the target policy, IT. and sampling a trajectory for the next time step 7/ ~ T(T a) using this sampled action. This T/ is distributed approximately according to Bay (T²"). Furthermore, with the exception of the last action, which is generated with probability dictated by m. Finally, we train 33 to maximize the likelihood of Ti given 7²" and perform TD-update between ((7("1;0) and R(T(-1,0 a) + max q(Ti',a) a').

---

yiu maxx mma)
n + P yy ymmax Qu ((22
8
71
A1
1-1
Figure 2: Visualization of policy training schemas for vanilla OBL (left), and OT-OBL on Pa. (7²) (right) in a 2-player environment. Here superseript indicates the active player while -i refers to their partner. The red arrow is the computed target value. The bluc colors represent fictitious transition. Left: Vanilla OBL generates trajectories by unrolling the training policy ı for both players. It then samples fictitious trajectories from oi Bay (-Im)) compute the training target. Right: OT-OBI. instead generates data using for only different policy ma. It performs the same OBL training but only
the players and allows its partner
time steps when is active player.
These procedures are rough heuristic approximations for the problems described in the previous section but do not solve them exactly. We give pseudocode in Algorithms and 2 and visualizations are provided in Figure and Figure
Algorithm 1 OT-BL: Train belief model B.G for target policy that is robust when evaluated on behavioral policy my (i.e. to mitigate out-of-distribution errors when used reinterpret trajectories generated by Tb). The main difference between OT-BL and normal belief training is marked in red.
I: procedure OT-BL(TT, To. d)
b Ny! the target policy for which we want to train a belief model.
the behavioral policy which the belief model may be evaluated at test time.
b d: size of the dataset
2:
Initialize belief model Buy
3: Initialize dataset D
4: Initialize environment
To P(To)
5:
while len(D) < do
6:
Sample fictitious trajectory T. Bry (-1T,)
7:
Unroll the fictitious trajectory T/41 T at) with target policy of Tr Add training data from the unrolled fictitious trajectory (rft1 r(41) to dataset DD
8:
9:
Unroll the real trajectory T++1 = T(T,, a.) with behavioral policy a, ~ mb Reset To P(To) if T441 is terminal
10:
11:
end while
repeat
Train Brg, with loss: log ((Fel7)) witht mini-batch from dataset (Fê. Te}
14:
until Bay converges
15:
return Bry
end procedure
Experimental Setup
6.1 Hanabi
Hanabi [ is benchmark for Dec-POMDP research. is a cooperative game for 2-5 players played with 8 hint tokens, 3 life tokens, and deck of 50 cards, each having rank between I and 5 and one five colors. the 2-player variant, on which this paper focuses, each player holds cards, backwards, such that they can see their partners' hand but not their own. On their turn, player can either play, discard or hint. The goal is to play cards in rank order for each color, with each correctly played card providing one point for maximum score of 25, The team loses a life token each time
5

---

Algorithm 2 OT-OBL: Train a Q-function under the OBL setting, but also make it less prone to diit----iitritttion errors when evaluated with different partners such as Te. For simplicity, we show the procedure for data collection and policy update with a single transition, omitting the complicated distributed RL setup. We assume there are two players, i (current) and -i (next). The main difference between OT-OBL. and normal OBL is marked in red.
1: procedure OT-OBL(Q. Bay. Tb)
> q. л: the Q-function we are training and its associated (greedy) policy b BA,: The belief model of the imput policy 9y (Tu in original OBL)
The
Th. the policy that may potentially be a partner at test time
b Note: assume that player i uses policy x and -i uses policy Tb
2: Generate 1-step partial trajectory 7; P (TT) using joint policy (æ,, mb), & 2 O.
3: Sample fictitious trajectory Ti ~ (.17))
4: Unroll the fictitious trajectory T(41 = T -T(T a²) with of (a/r7)) get reward r/
5:
Unroll the fictitious trajectory 7/42 = T(((11 d(41) with t'+1 ~ ;(alT(ti). get reward
'++1
6:
Update Q-function ((a/lr/) t- r + mh + ?? max, ((a)7
7:
end procedure
a player attempts to play an unplayable card. Hinting consumes hint token and can be used to point all cards of a chosen color or rank in a partner's hand The team regains hint token when player discards card or completes stack by successfully playing the 5 of the corresponding color. player draws a new card after playing or discarding. The game terminates when all 5 stacks are completed, the deck is empty, or all three lives are lost. In the latter case the team loses all the points they have earned.
6.2 Training Details
We implement our algorithms based on the open-sourced code for OBL and extend it with ideas from synchronous training [0] training all models simultaneously, thus enabling effective imple- mentations of OT-BL. and OT-OBL. When synchronously training OBL models, every = 50 steps we save a copy of the model then query for and update all dependencies.
We follow practices in the original OBL paper 7 to train each policy and belief model. The backbone R2D2 R a distributed recurrent Q-learning method with parallel environment workers and centralized replay buffer and trainer. We briefly describe the loss functions for the Q-network and belief models here. For more details, please refer to Appendix Ar. the OBL paper
The Q-network is trained to minimize the TD-error between its Q-value and targets from fictitious transition: [() = Σ [G], Qo(arlT:)) where the target G¹2 is calculated on the 'imagined' fictitious transitions G/, = + +7 f and θ a slightly outdated target network.
We pre-compute the fictitious target G², and store them into the replay buffer to avoid branching LSTT rollouts into fictitious transitions at training time.
The belief models are trained predict the player's hand given ,'. At given timestep f, the belief model will take in oi as input and predict the hand of player auto-regressively from oldest card newest. We train all OBL models as recurrent models via supervised learning to optimize [(h(T)) = - En log pigp(ha ha I h1:k-1) where hy is the k-th card in hand and n is the hand size.
6.3 Evaluation
We repeat each training method five times with different seeds. For SP and XP, every team is evaluated on 5000 games and then averaged over five seeds. The XP score is the average score of all teams formed by all independently trained agents. To better reflect improvement in intra-AXP, we introduce
the self-play cross-play gap (SXG) metric, which is computed as SXG = 100 (SP-XP)
This
SP
c.gg "Your first and third card are rank 3."

---

% Timesteps with Valid Fictitious Sample
100.0
97.5
95.0
92.5
OTBL Penalty
OTBL Penalized
90.0
Vanilla Penalty
Vanilla Penalized
87.5
85.0
82.5
100
200
300
400
500
600
Epoch Num
Figure 3: Comparison between belief models trained with vanilla OBL (greens) and with OT-BL (blues) in 2-life Hanabi with and without penalty for invalid samples. Experiments with and without penalty sample = 1, 10 hands respectively. We plot the % of timesteps with valid fictitious sample (standard error is shaded) when training OBL level 1. Normally trained belief models struggle to gather a valid sample; it steadily declines in unpenalized cases and drops to only 83% of timesteps in the penalty cases. OT-BL able to consistently gather a valid sample during training (> 99.5% of the time), regardless of penalty. OT-BL mitigates the covariate shift experienced during OBL training.
normalizes increases in XP score relative to the theoretical maximum self-play score. Additionally, SXG acts as a proxy for the covariate shift as it compares the evaluated training distribution (SP) with the test distribution (XP). The higher the SXG the larger the covariate shift.
6.4 Experimental Design for OT-BL
We design experiments for OT-BL to emphasize its advantage over the normal belief learning procedure when evaluated off-team. In normal OBL, the belief model is trained on trajectories from Ty but utilized on trajectories from m++1 to produce fictitious transitions, making it vulnerable to covariate shift. With OT-BL, half of environments used to train B.,, follow typical belief model training. The other half uses OT-BL to train B,, off-team on Preta (T²))
Ideally, we want to evaluate our algorithm on environments with a larger covariate shift. Therefore, make a few modifications the Hanabi environment exacerbate the covariate shift. First, we reduce the total number of life tokens from 3 to 2, making the game harder so that the average score and average trajectory length of different OBL levels differ more significantly. It is particularly challenging for the belief model only trained on shorter trajectories to generalize to longer unseen trajectories. Second, we consider sampling valid hands. Prior works including OBL utilize a checker provided in the Hanabi challenge [ to check if hand satisfies the known public knowledge. If hand invalid, the environment can fail to produce key input features. We further modify the 2-life Hanabi environment to remove the checker and allow invalid samples to corrupt the environment as is, which only affects the fictitious transition. Specifically. if sampled card contradicts the public knowledge of that card, the observation encoder will just encode the card knowledge for that card to be all zero instead of rejecting the sampled hand. We call this invalid card penalty.
Finally, we measure the belief improvements through the percentage of valid samples for fictitious transitions. For this we focus on the level policy training process the OBL hierarchy, where there the largest covariate shift. To train the level policy, we need to unroll a belief model Br., on trajectories generated by the learning policy π1. At convergence, πi can achieve 20 points. However, the belief model is trained on No, a uniform random policy with average score close to zero. We compute the percentage of timesteps where Br, able to produce valid sample when training m as proxy for the amount the belief model has deteriorated due covariate shift.
7

---

without penalty
with penalty
Level
OBL (XP SXG)
OT-BL (XP SXG)
OBL (XP SXG)
OT-BL (XP SXG)
20.17 0.15 (0.84)
21.05 + 0.02 (0.05)
16.71 0.06 (1.56)
21.15 0.02 (0.09)
2
22.92 + 0.21 (2.18)
23.41 = 0.02 (0.56)
21.24 = 0.11 (4.28)
23.48 + 0.02 (0.38)
3
23.05 + 0.30 (3.27)
23.76 1 0.02 (0.46)
22.53 1 0.23 (4.43)
23.83 1 0.01 (0.33)
Table 1: XP and SXG (in parentheses) for OBL and OT-BL in 2-life Hanabi with and without invalid card penalty. OT-BL always achieves higher XP scores. OT-BL, also has a very small SXG < 1% which 18x lower than corresponding vanilla OBL. The + represents the team's 's standard error.
Level
OBL (XP & SXG)
OT-OBL(XP & SXG)
OT-OBL+ Robust Belief (XP & SXG)
21.50 0.05 (0.42)
21.31 (0.19)
21.61 + 0.03 (0.32)
2
23.75 = 0.02 (0.46)
2.07 (0.17)
23.78 + 0.01 (0.21)
3
24.10 + 0.02 (0.41)
1 (0.17)
24.22 0.01 (0.17)
21.03 + 0.10 (0.91)
21.22 1 0.01 (0.21)
24.29 4 0.01 (0.16)
23.99 + 0.11 (1.28)
24.24 0.01 (0.12)
24.30 0.01 (0.16)
Table 2: XP and SXG (in parentheses) for vanilla OBL, OT-OBL, and OT-OBL with belief models trained on broader distributions in 3-life Hanabi. Vanilla OBL overfits leading to drop in XP scores. OT-OBL prevents overfitting and continues to improve its XP score. OT OT-OBL almost always has a higher XP scores than vanilla OBL. OT-OBL has a very small SXG < 1% and this gap is up to 10 times lower than that of vanilla OBL. By training our belief models on broader distributions, we further increase our XP score while maintaing a low SXG. The + represents the team's standard error.
6.5 Experimental Design for OT-OBL
OT-OBL for Intra-AXP We design experiments to evaluate OT-OBL's performance in intra-AXP. For normal OBL, the policy, my. is trained on trajectories from Pa, (rī) but evaluated on Pr.., (T)). This means Ty experiences test time covariate shift. To evaluate OT-OBL in intra-AXP, when training a given policy ne, we train on real trajectories from from P. . (Tī), ), (T²), Paz,ees (7/) and use OT-OBL to train off-team. By training on broader distributions, we increase the support of our policies. Since π-- and Te+1 are OBL policies, they are also reasonable and do well in Hanabi. We also test variant where
OT-OBL and robustify the belief by also training the belief model on PR (T?). Pac (T)), P Premeey (T.). We refer to this variant as OT-OBL- + robust belief.
OT-OBL for ad-hoc teamplay and human-AI coordination This set of experiments aims to demonstrate the ability for OT-OBLto improve test time performance with only small amount of data from test time agents. These experiments address the ad-hoc teamplay and human-Al coordination settings where there are large covariate shifts. We utilize rank bot for the ad ad-hoc teamplay setting and clone bot, bot trained via supervised learning human data, as proxy for human-A AI coordination. See Appendix A.3 and A.4 for more details on training rank and clone bot respectively. Each setting we assume access to only small number, 3200, of full trajectories generated by the target agents. We retrain a new policy, म, with a belief model trained on level 3 of sequentially trained OBL with the typical OBL training procedure. In this case half of the games in our distributed RL environment are the real target trajectories and we train x in an off-policy manner with OT-OBLon these trajectories. By training on these target trajectories, we can partially mitigate the covariate shift experienced when x is evaluated. We compare our results with a vanilla sequentially trained OBL level 4.
7 Results and Discussion
7.1 OT-BL
The the XP and SXG results of OT-BL on 2-life Hanabi is shown in Table Without the invalid card penalty, both OBL with vanilla belief learning and OT-BL are able to achieve reasonably high XP scores in 2-life Hanabi. However, OT-BI. consistently achieves higher XP scores in 2-life Hanabi across all levels. At level 3, OT-BI. able to achieve final XP score of 23.76 + 0.02. Lastly, the SXG for OT-BLix is always lower than that of vanilla OBL (8- 16x lower).
8

---

Training Type
XP SXG
Wank bot
w/ clone bot
w/ Vanilla OBL
Vanilla OBL
23.66 + 0.06 (1.42)
14.47 = 0.67
16.19 19 + 0.21
OT-OBL w/ Rank Bot
23.59 + 0.08 (1.38)
15.66 = 0.56
16.36 + 0.23
23.67 0.04 23.64 0.04
OT-OBL w/ Clone Bot
23.54 + 0.06 (1.13)
14.33 + 0.63
16.64 + 0.20
Table 3: Evaluating OT-OBL in ad-hoc teamplay (w/ rank bot), proxy human-AI coordination, and w/ vamilla OBL. By training on small number of target trajectories, we can improve ad-hoc teamplay scores with rank bot and proxy human-Al coordination scores. OT-OBL also doesn't i significantly hurt XP scores, the SXG, or the unseen target policies in ad-hoc teamplay or duman-Al coordination. Additionally, by training with OT-OBL, only broadening the policies support.
we still achieve very similar scores with vanilla OBL, indicating The here represents the team's 's standard error.
In Table 1 we also show the performance of OT-BL on 2-life Hanabi with invalid card penalty on the right. Similarly, OT-BL consistently achieves higher XP scores across all levels. OT-BL is able to achieve small SXG (always < 1%) and this gap is up to 18x lower than vanilla OBI. OT-BL achieves similar results between the penalized and unpenalized environments since the belief model is more on distribution, thus it almost always samples valid samples (> 99.5% of the time). However, OBL with vanilla belief leamning suffers significantly from the penalty, especially for level I where the covariate shift from a uniform random policy to learned OBL policy is the largest. This indicates that the test time covariate shift is lower for OT-BL when compared to vanilla OBL. The reduced covariate shift improves training and leads to better test time results.
We plot the percentage of timesteps B., is able to produce a valid sample when training n1 in Figure 3 to illustrate the deteriorating belief model in normal OBL and show the benefit of OT-BL. Withour penalty, vanilla OBL steadily decreases in the percentage of timesteps producing valid samples. With penalty, vanilla OBL is unable to produce a high percentage of valid samples and can never recover. This high percentage of invalid samples contributes to the steep score drop for OBL level 1 in the invalid state experiments. On the other hand, OT-BL mitigates the belief model's training covariate shift and consistently samples valid states (over 99.5% of the time). This contributes to the high scores for OT-BL and explains why OT-BL maintains high scores in penalized Hanabi.
7.2 OT-OBL
Intra-AXP Results We present the XP and SXG results of OT-OBL in table All variants, normal OBL, OT-OBL, and OT-OBL + robust belief all achieve high XP scores. However, at level 4 vanilla OBL starts to overfit, whereas OT-OBL and OT-OBL- + robust belief continue to improve. This results in OT-OBL achieving XP score of 24.24 + 0.01 and OT-OBL robust belief achieves a XP score of 24.30 + 0.01. As Vanilla OBL overfits, the SXG increases, indicating an increase in the test time covariate shift. On the other hand, both OT-OBL and OT-OBL + robust belief always have SXG< 1% and at level 5 the SXG is almost O. By level 5, the SXG of OT-OBL and OT-OBL + robust belicf are at least 8 times lower than that of vanilla OBL. Overall, by training broader distributions, we reduce the test time covariate shift, achieve goof intra-AXP performance, and improve OBL in off-team evaluation.
Ad-Hoe Teamplay and Proxy Human-AI Coordination Results We present the results appling OT-OBL Ad-hoc teamplay and Human-Al coordination in tablee3 With OT-OBL and only 3200 target policy trajectories, we can increase rank bot to 14.47 0.67 to 15.66 + 0.56 and increase clone bot scores from 16. + 0.21 to 16.64 + 0.20 without heavily impacting XP scores or the SXG. Additionally, vanilla OBL in XP achieves near identical scores to OT-OBL trained on additional trajectories is paired with vanilla OBL. Therefore, the additional trajectories merely broaden the training policies support and don't 't negatively impact policy training. Overall, by training on target distributions off-policy and off-team with OT-OBL we are able to reduce the test time covariate shift and increase ad-hoc teamplay and proxy human-Al coordination scores.
Failure Cases Bombing Out Bombing out is an important failure case in Hanabi, where if three unplayable cards are played then team loses all of their points and the game ends prematurely. In Hanabi, an agent always has alternatives to playing a card (e.g. discarding or hinting), thus a bombout occurs when an agent misunderstood the hints given by a partner or if its Q-value predictions are inaccurate. Both of which can be caused by covariate shifts. We present bombing out results in the

---

Total number of bombouts
Number of bombouts by OBL/OT-OBL
% Bombouts by OBL/OT-OBL
Cross-play Pair
Vanilla OBL x Rank Bot OT-OBL w/ Rank X Rank Bot
6891
1619
23.49%
6048
1208
19.97%
Vanilla OBL x Clone Bot OT-OBL w/ Clone Clone Bot
6144
663
10.79%
5239
473
8.28%
Table 4: Evaluating total number of bombouts and percentage of bombouts by vanilla OBL or OT-OBL in ad-hoc teamplay with two different partners. Each pair is first evaluated with 5000 games and then averaged over 5 independently trained OBL or OT-OBL models, By training with OT-OBL, we are able to significantly reduce instances where our trained agent causes bombouts.
ad-hoe teamplay and proxy-human Al coordination setting in table 4 Notice, that OT-OBL reduces the instances where our agent causes bombouts by 25.38% when partnered with rank bot and 28.66% when partnered with clone bot.
8 Limitations
The largest set of limitations of the approaches proposed in this work is that they are only heuristic approximations for solving the problems that this work introduced and do not converge to the exact solutions. Additional limitations include the fact that OT-BL assumes is also distribution when we produce fictitious moves with OT-BL. As a result, π is too off distribution OT-BL might not mitigate covariate shift. Moreover, although our work provides positive step to mitigate the large covariate shift that can occur in ad-hoe teamplay and human-Al coordination with little target policy data, there is still exists covariate shift.
Conclusion
We present off-team belief learning, method that allows us to approximate training beliefs off-teamm. Belief models are very ubiquitous in POMDPs and Dec-POMDPs. However, they can experience covariate shift when deployed on unseen distributions. We demonstrate that standard belief model training suffer from covariate shift, but OT-BL reduces it. Overall, by training OBL with OT-BL, we are able to achieve strong results in intra-AXP 2-life Hanabi and minimize the gap between SP and XP.
We also present off-team off-belief learning, method that allows us to approximate train action values functions off-m The goal of off-team training is to train with various teams with the goal of improving performance with unseen teams. With OT-OBL, we can train policies on broader distributions, increase their support, and improve test time performance. By training OBL with OT-OBL we were able to achieve near-optimal intra-AXP performance and improve ad-hoc teamplay and proxy human-AI coordination.
10 Broader Impact
We do not believe this work raises broader impact concerns.
References
[1] N. Bard, N. Foerster, S. Chandar, N. Burch, M. Lanctot, H. F. Song, E. Parisotto, V. Dumoulin, Moitra, E. Hughes, Dunning, S. Mourad, H. Larochelle, M. G. Bellemare, and M. Bowling. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216, 2020. ISSN 0004-3702.
[2] Barrett, A. Rosenfeld, Kraus, and Stone. Making friends on the fly: Cooperating with new teammates. Artificial Intelligence, 172,132-171, 2017. ISSN 0004-3702, doi: https:
10

---

[3] D. S. Bernstein, R. Givan, N. Immerman, and S. Zilberstein. The complexity of decentralized control of markov decision processes. Math. Oper: Res., 27(4):819 840, Nov. 2002. ISSN 0364-765X. doi: 10. 00.1287/mooo:27481992999
UtLhttps: org/ 10. 1287/moor 277
4.819.. 897
[4] M. Carroll, R. Shah, M. K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan. On the utility of learning about humans for human-ai coordination. In H. Wallach, H. Larochelle, A. Beygelzimer, d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http::ttpptccceediiiiiaee///// pdf
[5] B. Cui, H. Hu, L. Pineda, and J. Foerster. K-level reasoning for (human-ai) zero-shot coordina- tion in hanabi. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 35, pages 41 4190-4203, Curran Associates, Inc., 2021.
[6] H. Hu, A. Lerer, A. Peysakhovich, and J. Foerster. "Other-play" for zero-shot coordination. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4399 4410. PMLR, 13-18 Jul 2020.
[7] H. Hu, A. Lerer, B. Cui, L. Pineda, D. Wu, N. Brown, and J.N. Foerster. Off-belief learning. ICML, 2021. URL httth:t//aatiiiii orgo/oo/////o. hts//bbs/21000 220000001000000)0
[8] S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney, Recurrent experience replay in distributed reinforcement learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[9] D. Kingma and J. Ba. Adam: method for stochastic optimization, 2014.
[10] M. Lanetot, V. 7ambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Perolat, D. Silver, and Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 4190-4203. Curran Associates, Inc., 2017.
[11] A. Lerer and A. Peysakhovich. Learning existing social conventions via observationally augmented self-play. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES '19, page 107-114, New York, NY, USA, 2019, Association for Computing Machinery. ISBN 9781999890 17836336322..
[12] A. Lupu, B. Cui, H. Hu, and J. Foerster. Trajectory diversity for zero-shot coordination. In International Conference on Machine Learning, pages 7204-7213. PMLR, 2021.
[13] M. Ma, J. Liu, S. Sokota, M. Kleiman-Weinet, and J. N. Foerster. Learning to coordinate with humans using action features. CoRR, abs/2201, 2022. URL/https: htaps:://axxxxi org/abs/ 2201. 12658
[14] Schaul, Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL
oww.iii.orrr/ooo abs/ /ttp/////.../....//eeeebbbetttep 1t///111..0055 05952
[15] H. C. Siu, J. Peña, E. Chen, Y. Zhou, Lopez, K. Palko, K. Chang, and R. Allen. Evaluation of human-ai teams for learned and rule-based agents in hanabi. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P. Liang. and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 16183-16195. Curran As- sociates, Inc., 2021. URL httpps:/pproceeeinggss
86e8f 2ee85772882cca127777222116666663 Papeoo pdf

---

[16] S. Sokota, H. Hu, J. Wu, J. Z. Kolter, J. N. Foerster, and N. Brown. A line-tuning approach belief state modeling. International Conference on Learning Representations, 2022. URL
http::///penreviee..eettt www.:e/orummddddd22/0oo.....
[17] Stone, G. A. Kaminka, Kraus, and J. S. Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Proceedings of the Twenty-Foute Fourth AAAI Conference on Artificial Intelligence, AAAI'10, page 1504-1509. AAAI Press, 2010.
[18] D. Strouse, K. McKee, M. Botvinick, E. Hughes, and R. Everett. Collaborating with humans without human data. In M. Ranzato, A. Beygelzimer, Dauphin, Liang, and W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 14502-14515, Curran Associates, Inc., 2021. URL http:tts///occeeednnngg www.eee.nrrip...e.. cecprrps.cc/ppper/222 hit1:977770464442211111744444422
pdf
[19] D. Strouse, K. McKee, M. Botvinick, E. Hughes, and R. Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34, 2021.
[20] I. Sutskever, O. Vinyals. and Q V. Le. Sequence to sequence learning with neural networks. CoRR, abs/ 1409.321,, 2014. URL
www.:ogg////ao.... wwwp:/wwwwww.pg/////////.. 3215
[21] J. Treutlein, M. Dennis, C. Oesterheld, and J. Foerster. A new formalism, method and open issues for zero-shot coordination. arXiv preprint arXiv:2106.06613, 2021.
[22] M. Tucker, Y. Zhou, and J. Shah. Adversarially guided self-play for adopting social conventions. ArXiv, 0tss22001.59999 2020.
[23] H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning, In D. Schuurmans and M. P. Wellman, editors, Proceedings of the Thirtieth AAAI Conference Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2094-2100, AAAI Press, 2016. URL http: aaai org/ ocs/ index.ddexx.////// 12389
[24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. CoRR, abss/100.00362, 2017. URL http:://aaax org/ abs /abb//106... 03762
[25] Z. Wang. T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas. Dueling network architectures for deep reinforeement learning. In M. Balcan and K. Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings. pages 1995-2003. JoI.R.org, 2016. Uttp::t:://prooeee
mlr. press/ v48/ 3/8//angf16
html
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets?
Did you include the license the code and datasets? [No]
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answees below.
1. For all authors...
12

---

(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "tBtxd44a96", "reviewer": "Reviewer_nA7t", "summary": "This paper investigates the problem of off-policy learning in cooperative games. Within this area, they address a problem they call \"off-team (OT) learning\", where the teammates that generated the experience differ from the target teammates. They accomplish this by disentangling the belief in the trajectory, specifically the previous joint actions, from action evaluation. As a result, a policy can be learned that may adapt to a target team through its belief maintenance. Their algorithm is evaluated in a variety of Hanabi settings and shows some empirical improvements.", "questions": "1. \"Off-team learning\" relies on the learned belief model bein accurate up to time t-1. This seems unrealistic in practice to rely on as an assumption when the algorithm is jointly learning it alongside the policy. Can the authors comment on the need for accuracy in the belief model both in terms of the theoretical guarantees of the algorithm and the empirical performance? \n\n2. Coordination games may exhibit many diverse equilibria. How does this algorithm cope with target teammates that play equilibrium vastly different than that expressed by the behavioural teammates?\n\n3. The authors claim L224: \"The goal of ZSC is to evaluate the consistency of an algorithm across different seeds.\" This is not true. The goal of ZSC is to coordinate productively with unseen opponents. If the authors' statement was true any deterministic algorithm solves this problem.  \n\n4. In L236 the authors discuss the robustness of their algorithm given covariate shift with respect to the environment dynamics. However, it seems to me, that is mostly orthogonal to the work. In the abstract it seems the focus is on only differing teammates. Shouldn't the emphasis be on covariate shift to radically different teammates? And/or the combination of the two changes? Could the authors explain this focus and why they de-emphasized these other facets? \n\n5. L283, could the authors explain \"only a small number, o=3200, of trajectories\"? How many trajectories are 3200 observations? Why is it measured in observations? \n ", "limitations": "The broader impact section could be improved to reflect the impact belief-modelling style algorithms can have on biases. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n - Their algorithms are described in the text, graphics, and pseudocode allowing for diverse learning styles to approach the material.\n - Evaluations include some amount of ablations/sensitivity-analysis to better understand the merits of the proposed algorithm.\n\n**Weaknesses**\n - The paper is dense with jargon and heavily marked symbols with detailed meanings making it more challenging to read than I believe is necessary. \n - Maintaining and correcting for beliefs as a way to make strategic adjustments has been in many flavors, making this work's originality small. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fv5iPxNQqLd", "reviewer": "Reviewer_RFqN", "summary": "The paper introduces the concept of off-team learning, where a target joint policy is learned using a behavior joint policy to address covariate shift in off-belief learning (OBL). First off-team belief learning (OT-BL) is proposed to address covariate shift, where the belief model is iteratively updated on fictitious transitions. Second, off-team reinforcement learning (OT-RL) is proposed to address test time covariate shift with respect to unseen agents, where Q-values can be learned with belief models of any distribution. Both approaches are evaluated in the Hanabi challenge. The belief model of OT-BL is shown to produce more valid fictitious samples than OBL, while OT-RL is shown to outperform OBL in various settings.", "questions": "None", "limitations": "Limitations and potential negative societal impact are adequately addressed in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- **Originality**: The idea and concepts presented in the paper are novel, interesting, and highly relevant to current applications and challenges of multi-agent systems.\n- **Quality**: The paper is technically sound. The explanations and experiments are presented well and the results clearly support the claims. \n- **Clarity**: The paper is well-written, clearly structured, and easy to follow. \n- **Significance**: The experiment design and results indicate a clear improvement over OBL.\n\nThe authors did a great job on motivating and presenting their approach. Since a fundamental issue is addressed, a smaller toy problem like the cooperative communication game from the OBL paper would have been helpful.\n\nMinor comments:\n- The tables should highlight the best performing result in boldface for better readability.\n- Text and labels in Fig. 1 and 2 are very small and hard to read when printed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BrrpQkHVyil", "reviewer": "Reviewer_s5QM", "summary": "This paper identifies weaknesses in Off-belief Learning [ICML 2021] that lead to large distribution shifts at test-time evaluation in zero-shot coordination (particularly when evaluated with human players.) The paper contributes two extensions (off-team belief learning and off-team reinforcement learning) which together improve cross-play evaluation and zero-shot coordination with another agent (rank bot) and human proxy (clone bot).", "questions": "During the discussion period, I'm particularly keen to hear from the authors on the following prioritized list of questions that could improve my current rating of this paper:\n\n1) What additional evidence can the authors provide that clone-bot is a good human proxy?\n\n2) Assuming clone-bot is a good human proxy, what evidence is there that the small improvement in proxy human-AI coordination (line 327) is a significant advancement?\n\n3) Please clarify the issues (a)-(e) regarding specific details raised in the section on strengths and weaknesses above.\n\n4) Please justify the very strong claim in the abstract that \"Zero-shot coordination is a ... pre-requisite for human-AI coordination\".", "limitations": "The paper includes a sufficient broader impact section. However, the closing sentence regarding \"working in purely cooperative settings\" is a short sighted conclusion. I encourage the authors to consider the issues raised in Section 5 of \"Open Problems in Cooperative AI\" Dafoe et al. Arxiv 2020.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed methods are original extensions to an increasingly specific approach to zero-shot coordination in Hanabi. However, the diminishing returns in improvements to this system and limited scope of its evaluation weaken the significance of this finding. Most notably the improvement in ad-hoc teamplay evaluation with rank-bot (line 326) is insignificant when considering the variance of both methods evaluated and the effect size of the improvement in proxy human-AI coordination (line 327) is tiny. Furthermore, no evidence is given in this paper that the clone bot used for proxy human-AI coordination is a good human proxy.\n\nThe overall clarity of the contribution made is significantly weakened by the paper's over reliance on references to \"Off-belief Learning\" [Hu et al. ICML 2021] causing the paper to not be a standalone reference to reproduce the work. For example (but not limited to) the zero-shot coordination task or cross-play metric used throughout the paper are not formally defined. For details on the critical process of training the policy and belief model readers are deferred in line 209 to \"follow practices in the original OBL paper [Hu et al. ICML 2021]\" and the core contributions of this paper (OT-BL and OT-RL) are then defined just by how they differ to this past work. \n\nThe specific details of how the proposed extensions differ from the past work are also incomplete and potentially incorrect in parts:\n\na) In Figure 1 left should the arrows into \\pi from \\tau_t and \\tau_t+1 be reversed or the same as OT-BL (Figure 1 Right)? \n\nb) In Figure 1 Right, the blue fictitious transitions use \\pi in the illustration and \\pi_L in the caption? Algorithm 1 also uses \\pi and not \\pi_L. Is it just the caption on Figure 1 that is incorrect?\n\nc) The loss in Equation 1 is not defined.\n\nd) Is it intentional that the only difference in Figure 2 left (OBL) and Figure 2 right (OT-RL) is the policy used to sample real actions?\n\ne) On line 273, does P_{pi_L} mean both players play using policy \\pi_L?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept