Human-AI Coordination via Human-Regularized Search and Learning
Authors: Hengyuan Hu
David J Wu
Adam Lerer
Jakob Nicolaus Foerster
Noam Brown
Venue: Submitted to ICLR 2023 (2023)
Keywords: human-ai collaboration
multi-agent
search
deep reinforcement learning

Abstract: We consider the problem of making AI agents that collaborate well with humans in partially observable fully cooperative environments given datasets of human behavior. Inspired by piKL, a human-data-regularized search method that improves upon a behavioral cloning policy without diverging far away from it, we develop a three-step algorithm that achieve strong performance in coordinating with real humans in the Hanabi benchmark. We first use a regularized search algorithm and behavioral cloning to produce a better human model that captures diverse skill levels. Then, we integrate the policy regularization idea into reinforcement learning to train a human-like best response to the human model. Finally, we apply regularized search on top of the best response policy at test time to handle out-of-distribution challenges when playing with humans. We evaluate our method in two large scale experiments with humans. First, we show that our method outperforms experts when playing with a group of diverse human players in ad-hoc teams. Second, we show that our method beats a vanilla best response to behavioral cloning baseline by having experts play repeatedly with the two agents.

Under review as a conference paper at ICLR 2023
HUMAN-AI COORDINATION VIA HUMAN-REGULARIZED SEARCH AND LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the problem of making Al agents that collaborate well with humans in partially observable fully cooperative environments given datasets of human be- havior. Inspired by piKL.
aattaandddtt--eulariizee search method that improves upon a behavioral cloning policy without diverging far away from it, we develop a three-step algorithm that achieve strong performance in coordinating with real humans in the Hanabi benchmark. We first use regularized search algorithm and behavioral cloning to produce better human model that captures diverse skill levels. Then, we integrate the policy regularization idea into reinforcement learn- ing to train a human-like best response to the human model. Finally, we apply regularized search top of the best response policy at test time to handle out- of-distribution challenges when playing with humans. We evaluate our method in two large scale experiments with humans. First, we show that method out- performs experts when playing with a group of diverse human players in ad-hoc teams. Second, we show that our method beats a vanilla best response to behav- ioral cloning baseline by having experts play repeatedly with the two agents.
INTRODUCTION
One of the most fundamental goals of artificial intelligence research, especially multi-agent re- search, is to produce agents that can successfully collaborate with humans achieve common goals. Although search and reinforcement learning (RL) from scratch without human knowledge have achieved impressive superhuman performance in competitive games (Silver et al.) 2017 Brown & Sandholm 2 prior works (Hu et al.) |220 Carroll et al. (019) have shown that agents produced Carroll
by vanilla multi aggett reinforcement learning do not collaborate well with humans.
A canonical way to obtain agents that collaborate well with humans is to first use behavioral cloning (BC) (Bain & Sammut 1996) to train a policy that mimics human behavior and then use RL to train best response (BR policy) to the fixed BC policy. However, such an approach has few issues. The BC policy is hardly a perfect representation of human play. It may struggle to mimic strong players' performance without search (Jacob al. (022). The BC policy's 's response to new conventions developed during BR training is also not well defined. Therefore, the BR policy may develop strategies that exploit those undefined behaviors and confuse humans and causes humans to diverge from routine behaviors or even quit the task because they believe the partner is non-sensible.
Recently, Jacob et al. (2022) introduced piKL, search technique regularized towards BC policies learned from human data that can produce strong yet human-like policies. In some environments, the regularized search, with the proper amount of regularization, achieves better performance while maintaining or even improving its accuracy when predicting human actions. Inspired by piKI,, we propose three-step algorithm to create agents that can collaborate well with humans in complex partially observable environments. In the first step, we repeatedly apply imitation learning and piKL (piKL-IL) with multiple regularization coefficients to model human players of different skill levels. Secondly, we integrate the regularization idea with RL to train human-like best response agent (piKL-BR) to the agents from step one. Thirdly and finally, at test time, we apply piKL on the trained best response agent to further improve performance. We call our method piKL3.
We test our method on the challenging benchmark Hanabi (Bard et al., 2020) through large-scale ex- periments with real human players. We first show that it outperforms human experts when partnering

---

Under review as a conference paper at ICLR 2023
with group of unknown human players in ad hoc setting without prior communication or warm- up games. Players were recruited from a diverse player group and have different skill levels. We then evaluate piKL3 when partnered with expert human partners. We find that piKL3 outperforms an RL. best response to a behavioral cloning policy (BR-BC) - -a strong and established baseline for cooperative agents in this setting.
2 RELATED WORK
The research on learning to collaborate with humans can be roughly categorized into two groups based on whether or not they rely on human data. With human data, the most straightforward method is behavioral cloning, which uses supervised learning to predict human moves and executes the move with the highest predicted probability. The datasets often contain sub-optimal decisions and mistakes made by humans and behavioral cloning inevitably suffers by training on such data. A few methods from the imitation learning and offline RL community have been proposed to address such issues. For example, conditioning the policy reward target (Kumar al. 2019 Chen et al., 2021) can help guide the policy towards imitating the human behaviors that achieve the maximum future rewards at test time. Behavioral cloning with neural networks alone may struggle to model sufficiently strong humans, especially in complex games that require long-term planning (Mcllroy- Young al. 2020). Jacob al. (2022) address this issue by regularizing search towards behavioral cloning policy. The proposed method, piKL, not only improves the overall performance as most search methods do, but also achieves better accuracy when predicting human moves wide variety of games compared to the behavioral cloning policy on which it is based.
Human data can also be used in combination with reinforcement learning. Observationally Aug- mented Self-Play (OSP) (Lerer& & Peysakhovich, (019) augments the normal MARL training pro- cedure with a behavioral cloning loss on a limited amount of data collected from a test time agent to find an equilibrium policy that may work well with that agent. OSP increases the probability of learning conventions that are compatible with the test time agents. However it may not be able to model partners with diverse skill levels given large aggregation of data from various players. We can also use RL to train a best response policy to the behavioral cloning policy (Carroll et al. 2019). This method is the optimal solution given perfect human model. In practice, however, RL is prone to overfitting to the imperfections of the human model. In addition, RL alone may not be sufficient
in practice to learn superhuman strategies (Silver al. 2018 Brown & Sandholm 2019).
A parallel research direction seeks to achieve better human-Al coordination without using any hu- man data. Some of them take inspiration from human behavior the human learning process. Hu iet al.] (2020) enforce the RL policies to not break the symmetries in the game arbitrarily, a com- mon practice of human players in certain games. Inspired by humans' learning and reasoning pro- cess, Off-belief learning (Hu et al.) 2021c) and K-level reasoning (Costa-Gomes & Crawford 2006) Cui et al. 2021b) train sequences of policies with increasing cognitive capabilities. Both methods achieve strong performance with a human proxy model trained with behavioral cloning. Another group of methods use population-bas
training and various diversity metrics (Strouse et al. [02T] Lupu et al. 2021: Tang et al. (202T) to first obtain a set of different policies and then train common best response that may generalize better to human partners than best response to a single RL policy.
BACKGROUND
3.1 DEC-POMDP AND DEEP REINFORCEMENT LEARNING
We consider human-Al coordination in decentralized partially observable Markov decision process (Dec-POMDP) (Nayyar et al. 2013). A a state space S, joint action space A = AA' ... x AA,, a transition function T S x + S, reward function r S x A - E and a set of observation function of ?2'ss), E S for each agent i. We further assume that the joint actions a and rewards r are observable by all agents. We then define the trajectory of true states until time step as T (Sp. ag, ro. ........ st) and its partially observed counterpart (action-observation history, AOH) for agent as T/ = (of ag. To..... An agent's policy x' (rt) P(a (r.) maps each possible AOH distribution over the action space of
consists of N agents indexed by (1.....).
2

---

Under review as a conference paper at ICLR 2023
that agent. We use π to denote the joint policy of all agents and -' to denote the joint policy of all other agents excluding agent i.
Deep multi-agent RL (MARL) has been successfully applied in many Dec-POMDP environments.
Deep MARL algorithms often consist of a strong RL backbone such as (recurrent) DQN (Kaptur
owski al. 2019) PPO (Schulman
al. 2017) and additional modules such centralized
value function (Yu et al. 2021), value-decomposition (Sunehag et al.] [017] Rashid et al.) 2018) to handle challenges posed by having multiple agents. In this paper, we use recurrent DQN to
train a best response to fixed policies. Specifically, a recurrent network is trained to model the expected total return for each action given the input AOH, Q(F/. a) Irrr((rerr))// where R(T)) Eeze" (r'=)(('=0))))))) is the sum of discounted future reward by unrolling the joint policy π on the sampled true game trajectory until termination. The joint policy is the greedy policy derived from each agent's Q-function.
3.2 SEARCH AND REGULARIZED SEARCH IN DEC-POMDP
Search has been critical to achieve superhuman performance in many games (Silver et al. 2018 Brown Sandholm, 2019 Bakhtin al. 2021). SPARTA (Lerer et al. 2020) and its faster and more generalized variant Learned Belief Search (Hu et al. 202Tb) are competitive and efficient search algorithms in POMPPSS SPARTA assumes that a joint blueprint policy (BP) π has been agreed on beforehand. In single-agent SPARTA, one agent performs search at every time step assuming that their partners follow the BP. Specifically, the search agent keeps track of the belief B(TT)) P(T.IT/, n )) which is the distribution of the trajectory of states given the AOH and partners' policies. It picks the action a' that returns the highest sum of undiscounted future rewards assuming a' is executed at time and everyone follows the joint BP afterwards, i.e.
a arg max Qr (T/ a) arg max E Blrr) Bo a) + Rx (T(+1)),
where r(T,, a) is the reward at time t after executing a and R (T++1) 1) is the sum of future rewards following joint policy л. This notation assumes deterministic transition function the randomness can be absorbed into the belief function.
The belief tracking in SPARTA is computationally expensive and may have null support when part- ners deviate even slightly from the BP. Learned Belief Search (Hu et al.] 2021a| LBS) mitigates those problems by using a neural network belief model B trained with data generated by the BP with some exploration. For environments where the observation can be factorized into public and private parts, such as Hanabi, LBS also proposes to use a two-stream architecture where one stream with LSTM takes the public information as input while the other stream consists of only feed-forward layers takes the private information. The outputs of the two streams are fused to compute Q- Q-values. This special architecture further reduces the computation cost as it no longer needs to re-unroll the LSTM from the beginning of the game for each sampled T..
Although the learned belief technique was originally proposed to speed up SPARTA, it becomes critical in scenarios where the game trajectory does not exactly follow the assumed joint policy. For example, when playing with humans, humans' real moves may differ from our model's predictions and the learned belief model can often generalize well in those cases. In this paper we use LBS as the search component in piKL..
Jacob et al. (2022) show that the output policy of search algorithms can diverge quite far from the underlying blueprint policy used for rollouts and value estimation, which is undesirable in envi- ronments where collaborating with human partners is crucial. In general, they propose to sample actions following
Q (TÉ
P(a) x Tane (ale())))) exp
(2)
where Qraa (Tē, ()) is the value output of search algorithm like SPARTA, Monte Carlo tree search or regret matching using Tioll as the blueprint policy for rollouts and/or value estimation, Tane (a|r?) is the anchor policy that we want our final policy to be close to and À is parammer controlling the degree of regularization. In fully cooperative games where mixed strategies are not necessary, Jacob et al. (2022) show that a greedy variant that select the arg max works better in practice.
3

---

Under review as a conference paper at ICLR 2023
4 METHOD
In this section we introduce piKL.. We first use piKL.-IL with a probability distribution over the regularization parameter to model human players with varying skill levels. Then, use RL. regularized toward the behavioral cloning policy to train human-compatible best response piKL- BR. Finally, use piKL-LBS at time with high regularization toward the BR fix severe mistakes when playing with real humans.
4.1 PIKL-IL FOR MODELING DIFFERENT LEVELS OF HUMAN PLAY
Algorithm piKL-IL: modeling human with different skill levels. P(A) can be discrete uniform distribution over a set of values or over a set of Gaussian distributions centered around those values. PiKL-LBS(X,, Troll, B, Tame) is function act following Eq. 2 its greedy variant. It samples from the learned approximate belief model
B(rz)) estimate Q
1: procedure PIKL-IL(TBCc P(A), k, d)
BBC behavioral cloning policy trained from human data;
b P()) distribution of X;
k: number of iterations
d: size of the dataset
2: PIKLL-IL f- TBC
3:
for 8 f 1...., do
4:
Train a belief model B from self-play games of TpiKL-IL
5:
Initialize dataset D = 0 while (Dn do
6:
7:
Sample X; ~ P()) for every player independently
8:
Generate game where player follows PiKL-LBS(A,, TIKIL B, TBC)
9:
Add the game y to dataset 'D end while
10:
11:
Train a new policy T' with behavior cloning on
12:
TpiKL-IL f- m
13:
end for
14:
return TpiKL-IL
15:
end procedure
PiKI.-IL is a search-augmented imitation learning method. It first trains an imitation policy TBC via behavioral cloning (Bain & Sammut] 1996) on a dataset collected from the population of humans we want to model. Then piKL-IL iteratively improves policy TiKKL-IL- alternating between generating higher quality data with PiKL-LBS and training a better model using the generated dataset to produce T-LL--LL- Each iteration of piKL-LBSS uses лBC as the anchor policy Tane and TpKKL-IL as the rollout policy Troll in Eq. 2 so that we always anchor the generated data to never differ too much from human play, while using the best rollout policy far generate the next. The pseudocode is in Algorithm
piKL was shown maintain the same or even higher prediction accuracy on human moves while achieving much higher performance with certain As, indicating that it may be better at modeling stronger human players (Jacob al. 2022) than behavioral cloning. As À becomes smaller, the pre- diction accuracy drops while the performance keeps increasing, moving closer to the unregularized search policy. We therefore use distribution of Às generate a spectrum policies with strength ranging from average human players to exceptional policies that still resemble human behaviors reasonably well. When training new policy on the generated data, we can condition the policy on the À so that iterations as well as in the piKL-BR of Section 4.2
explicitly control the distribution of different skill levels in the subsequent
Theoretically, we should apply multi-agent PiKL-LBS but it is too computationally demanding to generate enough data for imitation learning. Instead we run single-agent piKL-LBS with learned beliefs independently for both players. Prior work (Jacob al. 2022) shows that although running PiKL-LBS independently for more than one player lacks theoretical guarantees because both players unsoundly assuming that the other player is playing according the pre-search policy when both

---

Under review as a conference paper at ICLR 2023
players in fact play according to the post-search policy, the algorithm still achieves high performance since all policies are regularized towards the same TBC, thus the learned belief models are still in practice good approximation of the true beliefs despite the shift in the underlying policies. For theoretically sound version, we could alternatively run single-agent piKI.-I.BS on only one player and only collect training data only from that player's perspective while fixing the other player to only play the pre-search policy that the learned belief assumes they will.
Once we have collected enough data, we can train a new model with imitation learning and proceed to a new iteration. The process can be repeated until TpiKL-nI. stops improving. Note that the anchor policy always the same human behavior-cloned policy prevent the final policy from drifting away from human conventions.
The algorithm presented here in the iterative form. However, if computational resources permit, can be formulated as an asynchronous RL algorithm similar to AlphaZero (Silver et al. (0118) where TikkL-IL is constantly trained with data from replay buffer while many parallel workers generate games with piKL-LBS and add them to the buffer.
4.2 PIKL-BR FOR A HUMAN-LIKE BEST RESPONSE
popular approach to human-A coordination to train a best response to a human model. This BR training is similar to standard single-agent RL in POMDP settings as the partners are fixed policies and thus can be viewed as part of the environment. In practice, this method has a few issues due to the imperfection of the human model well as the overfitting problem in RL.
Given a normally distributed dataset which the majority of the humans have intermediate skill levels, the vanilla behavioral-cloning model often converges to intermediate average score in self-play. Moreover, as observed in Jacob et al. (2022). BC often nontrivially underperforms even the average of the players it is trained on. This can make it hard for the BR agent learn the true best response to dtronger-than-aveeee
players, or even to average players. We can address this problem by training a BR PBR against the final piKL-IL policy TIKKL-IL instead of the original human behavior cloning policy TBC-
Similar to how single-agent RL can overfit to its exact training environment, an RL best response may overfit to its fixed neural partner, including finding unusual or -ffoofiiitiibttionn actions that happen to cause partner to perform slightly better actions but that might not generalize actual human players. In this case, instead of greedily picking an action that has slightly higher return as in normal RL, it would be better to err on the side of what humans tend to do in order to remain in-distribution. To address these issues, we propose to add piKL regularization during BR training.
Specifically, we train policy BpikI_BR to best response to TI. via Q-learning, but we modify the Qelaninn update as
Q(T a1) + re( a) + Q(T d(+1);
(3)
where d/+1 arg max (eaax(QQrin) ( + A log Tec(Ti+1
(4)
a
and where the exploration policy is e- Grreedy(((T/..o a) + A log TBC The difference from the normal Q-learning is highlighted in red. At test time, is set to O. The À here can be set smaller value than that in pIKI.-II. because the main purpose is no longer modeling human moves but rather regularization and tie- bie-breaking when multiple actions have small differences in expected return.
It is worth noting that if we run PIL with the same small X for many iterations, then the final TiK-I-n. with the small À input converges to the same policy as piKL-BR. PiKL-BR is the amortized model-free version of pIKL-IL and this step can be omitted if there are enough resources to run PiKL-ILL for enough iterations with additional As.
4.3 PIKL-LBS FOR ROBUSTNESS AGAINST OOD ERRORS
The TakL--R policy is strong human-like policy that performs well with piKL-IL and humans who play similarly. When playing with a diverse group of human players in real life, however, it may suffer from duttof-distribution (OOD) errors when encountering trajectories that have low probability under training distributions. The actions produced by T.kkilBrR on OOD input sequences can be arbitrarily bad as the neural network has never been trained on such data.
5

---

Under review as a conference paper at ICLR 2023
À Input
10
Self-Play
21.81 = 0.05
22.06
0.04
22.36
= 0.04
22.66 + 0.03
w/ piKL.-BR
22.99 + 0.04
23.10 + 0.03
23.20 + 0.03
23.34 + 0.03
The performance of piKL-IL in self-play and in cross-play with piKL-BR. The piKL-IL model is a single neural network model whose input contains the mean of Gaussian distribution from which the À of the player sampled. It evaluated with all possible input conditions (A input). Each cell (mean standard error) evaluated over 5000 games with different random seeds. For reference, TBC learned on the original human dataset gets 19.72 + 0.10 in self-play.
Table I:
However, search or other model-based planning algorithms can mitigate this problem by avoiding the most devastating mistakes, because often takes only few steps of simulated rollouts to directly observe the negative outcomes caused by those mistakes. Inspired by this observation, we run piKL- LBS test time on top of BR. We use "piKI_BR for both of Tane and Toll in Eq. 2 when is our turn, and assume the partner acts according to piiKL..II. on their turn. The belief model is trained on data generated by cross-play between piKL--BR and piKL-II.. Since the main purpose of this step is to avoid catastrophic OOD errors that are usually associated with substantially lower Q-values, we can set high so the search policy stays close TpiKL-BR in situations when the Q-values do not substantially differ.
5 EXPERIMENTAL SETUP
We implement and test our method in the Hanabi benchmark environment (Bard et al. 2020). In this section, we introduce the game rules of Hanabi, as well as how we implement piKL and best response to TBC (BR-BC) baseline.
Hanabi is 2 to 5 player card game. In this paper consists of five color suits and each suit has ten cards divided into five ranks with three 1s, two 2s, two 3s, two 4s and one 5. At the beginning, each player draws five cards from the shuffled deck. Players can see other players' cards but not their own. On each turn, the active player can either hint a color or rank to another player or play or discard a card. Hinting a color or rank, will inform the recipient which cards in their hand have that specific color/rank. Hinting costs an information token and the team starts with eight tokens. The goal of the team to play exactly one card of each rank to 5 of each color suit, in increasing order of rank, The order of plays between different color suits does not matter. A successful play scores the team one point while a failed play one costs one life. If all three lives are lost, the team will get 0 in this game, losing all collected points. The maximum score is 25 points. The team regains a hint token when a card is discarded or when a suit is finished (playing all 5 of suit successfully). The player draws new card after a play or discard move. Once the deck is exhausted, the game terminates after each player makes one more final move.
the standard 2-player version. The deck
We acquire a dataset of roughly 240K 2-player Hanabi games from BoardGame human policy тh. The dataset contains all the games played in a certain period on that online platform and we do not perform any filtering. The dataset is randomly split into training set of 235K games, a validation set of 1K games and test set of 4K games. The average score of games in the training set 15.88. The policy Ti parameterized by Public-LSTM neural network (See Hu et al. (2021b) or Section 3.2). The policy is trained to minimize the cross-entropy loss (())= ED En no (aflrr)) Note that treats the AOH of each player r; as a separate data point for training. Similar to prior works (Hu al. 2021c), we apply color shuffling Hu et al. (2020) for data augmentation. Every time we sample 7, DD we generate a random permutation f of the five colors, e.g. f: a be e e + d e, and apply f to both the input and the target of the training data. This model is trained with Adam (Kingma & Ba 2014) optimizer until the prediction accuracy on the validation set peaks. The converged Ta gets 19.72 0.10 in self and 63.63% in prediction accuracy on the test set. In evaluation, we take the arg max from the policy instead of sampling, which also explains why it achieves higher average score than the training set.
train the
PiKL-LBS requires a learned approximate belief model. In Hanabi, the belief model takes the same AOH 7¹ the policy and returns a distribution over player i's own hand. The hand consists of
en www.roagaaaaaeeeeooooooaooo

---

Under review as a conference paper at ICLR 2023
cards and we can predict them sequentially from the oldest to the newest based the time they are drawn. The belief network o consists of an LSTM encoder to encode sequence of observations and an LSTM decoder for predicting cards autoregressively. Note that the belief is a function of partner's policy. The belief for a given policy π partnering with is trained with cross-entropy loss ((6) -Ry.=DD(s..)) E, E, Po(plt 1 CI,".. -cj-1), where Cj is the j-th card in hand to predict. (( p) is an infinite data stream generated by cross- cross-play using x and p and 7* D means that we only use data from xt perspective for training. In piKL-IL, we use π = p Trll-
We set P(A) to be a uniform mixture of truncated Gaussian distributions to model players of diverse skill levels. Specifically, we use Gaussian distributions, N ( truncated at 0 and 2/r with (1, o) = 1/4) (2, 2/4), (5, 5/4), (10, 10/4) and each Gaussian is sampled with equal probability. We generate d' = 250K games in each iteration to train the new policy and we find that one outer iteration (k = 1 in Algo. [ is sufficient to achieve good performance in Hanabi. In every LBS step, perform M = 10K Monte Carlo rollouts evenly distributed over [A] legal actions. We sample M/|A| valid private hands from the belief model to reset the simulator for rollouts. Invalid sampled hands are rejected. With this setting, each game with 2 player running PiKI.-L.BS independently takes roughly 5 minutes with 1 GPU and we use 500 GPUs in parallel for 42 hours to generate the entire dataset. To better imitate policies under different λs, we feed the f of the Gaussian distribution from which the À is sampled to the policy network in the form of one-hot vector concatenated with the input. The self-play performance of the piKL-IL model conditioning on different input is shown in the top row of Table [ Clearly, piKL-IIL performs significantly better than Th and the score increases as regularization decreases.
The BR is trained under a standard distributed RL setup where many parallel workers generate data with cross-play between the training policy and the fixed II. policy. The generated data is added into a prioritized replay buffer Schaul et al. (2015) and the training loop samples mini-batches of games to update the policy with TD errors. We use the same Public-LSTM architecture for the BR policy as it will also be used in piKL-LBS at test time. The BR policy explores with e-greedy to a distribution of e sampled every new game while the IL policy does not explore but samples a new À input from {1, 2, 5. 10) every game. The À in Eq. is set to 0.1. The tross-play performance between the converged piKL-BR and pIKL-IL is shown in the bottom row of Table As expected, piKL-BR is better at collaborating with piKL.-IL than piKL-IL itself and the gap shrinks as the regularization À decreases. The reasons are that piKL-BR is trained with lower regularization and RI. can optimize for multi-step best response while search can only optimize for one step.
We run PiKL-LBS on the PiKL-BR policy with high regularization À = 2. The search assumes that our blueprint is Thp and our partner always follows ML, the final output policy of PiKL-IL. To avoid predicting the À input for partner model TIL. we replace it with an imitation learning policy T... trained on the same dataset as Ti but without the À input. The belief model is trained the same way as in piKL-IL but with π = TBR and per step is reduced to 5K to make faster and suitable for real world testing.
Ti in D(T, p). The number of Monte Carlo rollouts
Finally, we train an unregularized À = 0 best response to the vanilla behavioral clone policy TBC as our baseline. This agent achieves 23.19 + 0.03 in cross-play with TBC convergence. This score is quite high considering that its partner TBC is much worse than the PiKL-IL policy, indicating that the unreguarlized BR may be overfitting to the imperfect human model.
RESULTS
We carry out two large scale experiments with real humans to evaluate piKL3. The first experiment focuses on ad-hoc team play with diverse group of players without any prior communication (zero- shot). In the second experiment, we invite a group of expert players to play multiple games with PiKL3 and the BR-BC baseline in alternating order to further differentiate the gap between them.
The experiments are hosted on our customized version of the open sourced Hanab, Live platform The modified platform disables chat, observe and replay functionalities. Additionally, participants cannot create games themselves nor invite others to form team. All games are created automatically following the design of the experiments below. We send each player an instruction document of the platform make them familiar with the UI in advance.
https: thub. com Hanabi -iiee-eavveeaaa -liee
7

---

Under review as a conference paper at ICLR 2023
w/ Human Experts
w/ BR-BCC
w/ piKL.
All Testers (56)
14.54 + 1.47
16.73 1.27
17.18 1.28
Newcomer (2)
0,00 + 0.00
0.00 0.00
10.00 + 7.07
Beginner (17)
9.12 + 2.65
14.82 2.42
14.47 + 2.63
Intermediate (23)
14.57 + 2.27
19.48 + 1.64
18.52 + 1.79
Expert (14)
23.14 + 0.60
16.93 2.41
19.29 2.14
Table 2: The performance of different groups of players partnering with a group of testers. Testers recruited from diverse sources represent the general population with different skill levels and different conventions Hanabi, Each tester is matched with one the available human experts, one BR-BC baseline agent and one piKL3 agent in random order. The bottom 4 rows shows the results for cach subgroup of testers. The number in parentheses after the group name is the headcount of that group. Each cell contains mean 1 standard error.
w/ BR-BC
w/ piKL3
21.21 + 0.59
22.23 + 0.52
Experts 22.52% + 3.96%
31.86% % + 4.38% %
Table 3: The performance of experts playing with BR-BC baseline and piKL3 in alternating order repeatedly for maximum of 20 games in total. The cells contain mean + standard error (top row) averaged over 111 (BR-BC) and 113 (piKL.3) valid games, and percentage of perfect games (bottom row).
6.1 AD-HOC TEAM PLAY
In the first experiment, we recruited players with different skill levels from diverse sources. We posted invitations the board game Reddit, the forum of BoardGameArena, Twitter and Facebook Ads, as well as on two popular Discord channels where enthusiasts discuss conventions and organize tournaments. This group of players are referred to lesters. A $40 or S80 gift card was sent the testers who successfully completed the required games to encourage participation. Meanwhile, we recruited a group of expert human players from a well-known Discord group study how well humans can do when playing with unknown partners. The experts have all played Hanabi for more than 500 hours. The experts were paid proportional with the festers. Each tester signed up for 45-60 minute time slot. During their session, they were automatically matched with the BR-BC baseline agent, the piKI.3 agent and one of the available experts in random order. Usernames of all players including Al agents were randomly decided, to maintain anonymity of the participants and of which players were the experts or the agents. Both AI agents sampled sleep time t proportional to the entropy of softmax(() and waited for at least t seconds before sending the action. This further helped to hide the identity of AI agents and to mitigate the potential side effect that piKL3 and the BR-BC need different amounts of time to compute an action.
the time they spend waiting for and playing
The results shown in Table 2 From the overall result in the first row, and piKL. outperformed human experts in this task, indicating that playing with a diverse range of players in the zero-shot ad-hoc setting is challenging for humans. Still, worth noting that the 2-player no-variant version of Hanabi is not a particularly popular variant within the advanced Hanabi community and many people mostly play with people they know so that they can discuss and perfect strategies after games.
that both BR-BC
We let testers self-identify as one of four skill levels spanning from newcomer who has just learned the rules to expert who has played extensively. The results for each skill level group together with the number of players from each group are shown in the table as well. Although the standard errors are large due to the small number of games within each group, we can see a trend that the AI agents generally worked better with non-expert players, while experts have a clear lead when collaborating with other experts. This is likely because experts' behaviors are more predictable and the community has converged to a few well-known convention sets that are easy to identify for the experts who follow them closely in forums and discussion channels. It might also reflect the fact that our training data matches a more diverse pool of players with fewer experts in it, since those experts tend to play on sites other than BoardGameArent which is only source of training data.
8

---

Under review as a conference paper at ICLR 2023
The difference between BR-BC and piKL3 in this set of results is not significant given the standard errors. Unfortunately, it is challenging to collect more games in this setting due to both the difficulty to recruit enough people with good intent and the logistic overhead to manage anonymous human- human matches.
We hypothesized that piKL. might work better with experts thanks to its abilities to model stronger human players and to be more robust against OOD errors. We design a different experiment to verify this in the next section.
6.2 REPEATED GAMES WITH EXPERTS
In this experiment, we recruited a group of expert players to play with the two AI agents in alter- nating order for a maximum of 20 games. Each expert started randomly with one of the agents and switched to the other one after each game. They were aware of the matching rule and which Al was in their current game. As an incentive to do well, the players were compensated with $0.6 for every point they got. A terminated game with all life token lost counted as 0 points.
We collected 111 games for the BR-BC agent and 113 games for the piKL3 agent. The numbers are slightly different for the two agents because some games terminated unexpectedly due to platform related technical reasons. The average score and the percentage of perfect games for each agent are shown in Table A perfect game is where the team achieves full score. Although the improvement may seem small numerically, the mechanics of Hanabi makes it increasingly difficult to improve as the score gets closer to a perfect 25. In RL training, for example, learning 20-point policy from scratch takes roughly the same amount of time and data as improving that policy to a 21-poitt one.
PiKL3 outperformed the BR-BC in terms of both average score and percentage of perfect games. Although the statistical significance of both results is somewhat limited given the amount of data p = 0.097 and p- 0.058, respectively, both are strongly suggestive and consistent with our ini- tial hypothesis. It is also encouraging to see that piKL3 can achieve more perfect games with the experts. Experienced human players are particularly excited about perfect games as they are often significantly harder than getting 23 or 24 in a given deck.
Due to the limitation on budget and the overhead of managing experiments with humans, we were unable to collect enough games to perform ablations for each component of piKI.3, nor to include more baselines. In this experiment, we focused on demonstrating the effectiveness of this combina- tion compared to the popular BR-BC bascline. In practice, researchers may any combination of the three components of PiKL3 based on the properties and challenges of their specific domains.
Most existing works targeting human-Al coordination in Hanabi (Hu et al., 2021c; Cui et al. 2021al la) do not directly use human data. They get around 16 points with a similar but slightly stronger TBC while piKL. and BR-BC get around 23. Therefore, despite being interesting, it will not be fair comparison to include them. Previously, Hu et al.] (2020) also tested their agent with human players. However, these results are not directly comparable as they use different scoring method that keeps all the points when losing all life tokens. Additionally, the population of the human testers have a profound impact on the numerical results.
CONCLUSION AND FUTURE WORK
We present PiKL3, three-step algorithm centered around the idea of regularizing search, imitation leamning and reinforcement learning towards policy learned from human data. We performed two large-scale experiments with human players in the Hanabi benchmark to demonstrate the effective- ness of this method and its superiority over the popular baseline of training ordinary best response to plain imitation policy. We also for the first time report human experts' perhaps surprisingly low performance on zero-shot ad-hoc team play with diverse population.
The main limitation of this method is that it requires large amounts of data to learn the initial human policy used anchor and blueprint in piKL search. Therefore, an interesting future direction is to extend this method onto more domains, especially ones with less high quality human data. Another direction would be to create personalized regularization that adapts to each individual player in repeated games in an attempt to better model them.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15. pp. 103-129. Oxford University Press, 1996.
Anton Bakhtin, David Wu, Adam Lerer, and Noam Brown. No-press diplomacy from scratch. In A. Beygelzimer, Y. Dauphin, Liang, and Wortman Vaughan (eds.), Advances Neu- ral Information Processing Systems, 2021. URL httptt:///pppeeeeeeeeee net. http//ewwwee/.trrrrreeeee forum?id= #q7wIzt30UE.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
Noam Brown and Tuomas Sandholm. Superhuman Al for multiplayer poker. Science, 365(6456):
885-890, 2019.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL httppsss//prrooedangngn mww.eeeiiiinnnn.nnnneeeeeeeeeeee co/ /c//papp//211///lee
pdf
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaugham (eds.), Advances in Neural Information Processing Systems, 2021. URL htttp:://openeviee. id=a APPM4B9d
Miguel Costa-Gomes and Vincent P. Crawford. Cognition and behavior in two-person guessing games: An experimental study, In American Economic Review, volume 96(5), pp. 1737-1768, 2006.
Brandon Cui, Hengyuan Hu, Luis Pineda, and Jakob Foerster. K-level resoning for zero-shot coor- dination in hanabi. In Thirty-Fiih Fifth Conference on Neural Information Processing Systems, 2021a.
Brandon Cui, Hengyuan Hu, Luis Pineda, and Jakob Foerster. K-level reasoning for zero-shot co- ordination in hanabi. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. -228. Curran Associates, Inc., 2021b. URL htttps::/proceee/innaaeee neur Lpsurrpps.... pps cc/
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "'other-play" for zero-shot coordination. In International Conference on Machine Learning, pp.
PMLR, 2020.
Hengyuan Hu, Adam Lerer, Noam Brown, and Jakob Foerster. Learned belief search: Efficiently improving policies in partially observable settings. arXiv preprint arXiv:2/06.0900 2021a.
Hengyuan Hu, Adam Lerer, Noam Brown, and Jakob N. Foerster. Learned belief search: Efficiently improving policies partially observable settings. CoRR, abs/2106.00006, 2021b. URL https:
//arxiv.org 02006
Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, David Wu, Noam Brown, and Jakob Foerster. Off-belief learning, In International Conference on Machine Learning. PMLR, 2021c.
Athul Paul Jacob, David Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Jacob Andreas, and Noam Brown. Modeling strong and human-like gameplay with KL-regularized search. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 06695-9722 PMLR, 17-23 Jul 2022. BRL.tttppp htttps://prooeediii ngs ml www.s//evvvvv... 62 http:///////aaa222 html
10

---

Under review as a conference paper at ICLR 2023
Steven Kapturowski, Georg Ostrovski, John Quan, Rémi Munos, and Will Dabney. Recurrent expe- rience replay distributed reinforcement learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htt.hhttps::/oppneeeeiiie net/forumt lenenniddrrryyyeoa yT
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Aviral Kumar, Xue Bin Peng, and Sergey Levine. abs/1912.13465, 2019. URL htthtt:://aaxii http:///wwww...orr//////////////
Reward-conditioned policies.
CoRR,
Adam Lerer and Alexander Peysakhovich. Learning existing social conventions via observation- ally augmented slaf- In Proceedings of the 2019 AAAI/ACM Conference on Al, Ethics, and Society, pp. 107-114, ACM, 2019.
Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games, In AAAI Conference on Artificial Intelligence, 2020.
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7204 7213. PMLR, 18 24 Jul 2021. URL httpp::/ppocceeee/ee /
www.nngs........m.mm..mme...mmem press/v139/
Lupu2la.hir html
Reid Kellroy-YYungy Russell Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Learn- ing personalized models human behavior in chess. Technical report, Microsoft, Inc., Au- gust 2020. URL
http://www..cooooooooocoooo///// zed hrtpr//peeetcttaaooonroononinnnr
Ashutosh Nayyar, Aditya Mahajan, and Demosthenis Teneketzis. Decentralized stochastic control with partial history sharing: A common information approach. IEEE Transactions on Automatic Control, 58(7):1644-165, 2013.
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foer- ster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th In- ternational Conference on Machine Learning, ICML 2018, Stockholmsmässsan Stockholm, Swe- den, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4292 4301. PMLR, 2018. URL http::::::pcpccceiptpet ngs m prwss/ v80 rashiaihhcce 8a html
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay. arXiv e-prints, art.
Nov 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 707.06377, 2017. URL http://arxiv.orr//eee
1707. 06347
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Mathhe Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 35047 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419))1140) 1144, 2018
DJ Strouse, Kevin R. McKee, Matthew Botvinick, Edward Hughes, and Richard Everett. Col- laborating with humans without human data. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:://penrest net /
wttp:ew...w.eeeteeeeteee Forumiidu Kof Kof-nkm@58

---

Under review as a conference paper at ICLR 2023
Peter Sunehag. Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zam- baldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo. Karl Tuyls, and Thore Grae- pel. Vallu-dccommositionn networks for cooperative auuli- learning. CoRR, abs/1706.05296, 2017. http::p:////.t.ooor//aaa////////
C5226
Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Shaolei Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward ran- domization. In International Conference on Learning Representations, 2021. URL https:
//openreview. lett://fooummmd=wTT@ccoo.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre M. Bayen, and Yi Wu. The surprising effectiveness of MAPPO in cooperative, multi-agent games. CoRR, abs/2103.01 2021. URL 2103.
https://arxiy ottp://o/////////../.....
12


Reviews:
[{"review_id": "rJDvssuPHWJ", "reviewer": "Reviewer_hc8j", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work extends piKL (an existing work for learning human-like policies) to coordinate with humans. They showcase their results on Hanabi benchmark. ", "summary_of_the_review": "I found that the work attempts to solve an interesting problem, relevant to the HiL community however lacks good readability to be accepted as a paper in ICLR. I would recommend the authors to update the write up to accordingly motivate the problem and highlight the contributions.", "strength_and_weaknesses": "Strengths : \n1. I agree to the problem statement of being of high importance with upcoming HiL methods where coordination with the human model / policy should be accounted for. \n\n2. I like the fact that the work attempts to model humans of varying skill levels. \n\nWeaknesses : \n\n1. I was quite disappointed with the background section as the paper mentions piKL several times and admits to extend that work, but fails to provide any background details about this existing work. \n2. Why Partial Observability is so important? Again, I feel with improved related work sections this could have been solved. The work mentions two key things in their setup - need for coordination with real humans as well as that agents exist in partially observable environments. Are there works that have already solved full observability? \n3. The work keeps mentioning SPARTA several times in the related work section & some of its details - but not piKL? Is there a reason for it? \n4. How feasible is such a technique in other environments (other than Hanabi) where 200k + simulations may not be available?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qD5VNrNZxr", "reviewer": "Reviewer_6xiK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the multi-agent human-AI collaboration setting, and the challenge of coordinating with humans. Specifically focusing on the Hanabi benchmark task, the authors propose the piKL3 method which as the following components:\n(1) a human model that captures diverse skill levels controlled via a \\lambda regularization term, \n(2) train a human-like best response model, \nand (3) further apply PiKL on the trained best-response model. \nThey evaluate their method on Hanabi, showing marginal improvement over a BC baseline. ", "summary_of_the_review": "Because I found the paper quite difficult to follow and limited in terms of novelty and performance of the proposed method, I lean towards reject. However, because of my unfamiliarity with the related works, this is a low confidence review. ", "strength_and_weaknesses": "\nWhile I believe multi-agent human coordination is a challenging and interesting problem, I overall found the paper quite difficult to follow. \nMy concerns include:\n\n1. The gains of the proposed pikl3 method over baselines are not that strong at all, and is only evaluated on one task (Hanabi)\n2. The methodological contribution largely seems to be via a regularization term, and applying piKL to multiple stages of the multi-agent task, which seems quite limited in novelty to me\nMinor: \\lambda  controls the regularization of unconstrained search towards human data, which is a bit different than different human \"skill levels\", as the spectrum is more about the degree of human-like behavior, making the overall point of different levels a bit confusing\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "dg_qS0ieBsV", "reviewer": "Reviewer_yAwz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a framework for training RL agents to collaborate with human players in 2-player Hanabi games. The key challenge of training such a policy is that human data is finite and fixed, which means that there needs to be a human-like behavior policy for the RL to learn to collaborate with. Prior work shows that such human-like policy trained with naive behavior cloning may make mistakes that can be exploited by the RL agent, resulting in the RL agents exhibiting strategies that don't mesh well with real human players. The paper proposes to use piKL, a prior search-based method that generates strong yet human-like plays, as the human policy to train the RL policies with. On the side of training the human policy, the method introduces (1) an iterative training scheme to improve piKL (the human policy) by feeding its successful play back to its dataset and (2) a way to learn human policies with different expertise levels by tuning the regularization strength. On the side of RL training with the human policy in-the-loop, to prevent RL overfitting to the human policy, the method proposes to further regularize the trained human policy by preferring “human-like” actions instead of greedily choosing the best move. Finally, the method also proposes to augment the RL policy with a search-based strategy to guard against catastrophic failure caused by out-of-distribution moves by real human players. The method is evaluated primarily via playing with real human players. The authors built a customized game server and solicited players of varying expertise level to play against the trained agent. The paper shows that the proposed method is able to outperform the RL policies trained with naive BC-based human policies. \n", "summary_of_the_review": "Overall I quite like this paper. See my main comments for the remaining concerns.", "strength_and_weaknesses": "Overall I quite like the spirit of this paper — for human-machine collaboration, human data is inherently scarce, especially for methods that rely on massive simulation and trial-and-error imitation learning. Training a human-like policy from a fixed set of human demonstration to generate reactive response for RL learning seems to be a good strategy. The proposed method is also conceptually simple — use a constrained BC method to plug “holes” in the human policy to prevent the RL learner from exploiting these errors to develop un-human behaviors. Finally, the scale of human study is laudable. \n\nI don’t have major concerns. A few minor ones: \n- Does \\lambda truly capture the level of expertise in eq.2? Intuitively it’s an interpolation between “optimal play” and “human-like” play instead of different expertise levels. Besides, how much does training the policy against these policies matter in practice? There doesn’t seem to be an ablation study on this.\n-  The general recipe for learning human-like behavior policy to train RL collaborators seems to be general and can be applied to other domains. The authors also seem to agree with this point in the “future work” section. The authors should discuss which aspect of the proposed method needs to be adapted in order to accommodate new domains.\n- A piece of related work that’s missing from the paper is  Wang et al. [1]. They explored a similar venue of idea, including elements such as learning a human-like behavior policy to train RL collaborators and modeling diverse collaboration strategies, albeit in a robotics domain. It’d be great if the authors could compare and contrast their work with [1]. \n\n[1] Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration, Wang et al., 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XutjiMMdL1", "reviewer": "Reviewer_ywYA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper looks to improve on the piKL method for producing good policies for human-AI collaboration. The proposed method, piKL3, uses piKL in three ways to produce a final collaborative policy. The first part, piKL-IL, is an iterative imitation learning algorithm which takes as initial input a behavior-cloned policy based on human data. At each iteration, the algorithm generates data sampled from piKL agents based on the current policy with various values of $\\lambda$. Then, a new policy is trained via behavior cloning on the generated data and the process continues. The idea is to produce a policy with similar advantages to piKL but that better approximates a range of human skill levels. The second part of the algorithm, piKL-BR trains a best response to the output of piKL-IL with additional regularization that encourages the best response to be similar to the original behavior-cloned policy. Finally, the third part further improves the output of piKL-BR by using piKL again with the piKL-BR policy as both the anchor and rollout policy. Empirically, piKL3 seems to improve human-AI performance on Hanabi against a baseline of simply training a best-response to the behavior-cloned policy.", "summary_of_the_review": "Overall, the paper is good except that it could use a better evaluation with ablations to justify the complex, multi-stage algorithm that is proposed. Therefore, I don't think it is quite ready for publication.", "strength_and_weaknesses": "The writing of the paper is quite good and the algorithm is described clearly. There are some nice algorithmic insights in the various pieces of the piKL3 algorithm. The empirical results in Table 3 also show a clear improvement over the baseline of BR-BC. \n\nMy main concern with the paper is that piKL3 is both complex and computationally expensive, and I'm not sure the limited evaluation justifies all the complexity. I understand that the human experiments are expensive to run, but it seems like there are a lot of other ways the authors could have done ablations or better evaluated whether all pieces of the algorithm are fully necessary. For instance, one could train a proxy agent using BC on some held-out data (possibly from a different population) not used for the algorithm. Ideally, there would be experiments showing what happens if you train piKL-BR directly on the BC policy rather than on the piKL-IL policy, or what happens if you don't add piKL on top of piKL-BR, etc. With the current evaluation, it's very hard to tell where the improvement over BR-BC is coming from—it could be a combination of all parts of the algorithm, or just one. The authors argue that \"in practice, researchers may use any combination of the three components of piKL3 based on the properties and challenges of their specific domains.\" However, given how expensive some components are (most researchers in academia do not have access to 500 GPUs), it would be ideal if readers could get a better sense of which components are the most important and decide if the computational expense is worth it for each one.\n\nOther specific issues:\n * In Table 1 I don't understand exactly how $\\lambda$ can be varied when evaluating piKL-IL self-play. Isn't the output of piKL-IL, as you describe, a single neural network policy which ends up being the average of piKL policies with various values of $\\lambda$? How can the $\\lambda$ value of this be changed when evaluating it?\n * Where is footnote 3?\n\nTypos:\n * Page 6: \"whose input contains the mean of Gaussian distribution\" -> \"mean of Gaussian distributions\"\n * Page 9: \"all life token lost\" -> \"all life tokens lost\"\n * Page 9: \"the population of the human testers have a profound impact\" -> \"the population of the human testers has a profound impact\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject