Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper looks to improve on the piKL method for producing good policies for human-AI collaboration. The proposed method, piKL3, uses piKL in three ways to produce a final collaborative policy. The first part, piKL-IL, is an iterative imitation learning algorithm which takes as initial input a behavior-cloned policy based on human data. At each iteration, the algorithm generates data sampled from piKL agents based on the current policy with various values of $\lambda$. Then, a new policy is trained via behavior cloning on the generated data and the process continues. The idea is to produce a policy with similar advantages to piKL but that better approximates a range of human skill levels. The second part of the algorithm, piKL-BR trains a best response to the output of piKL-IL with additional regularization that encourages the best response to be similar to the original behavior-cloned policy. Finally, the third part further improves the output of piKL-BR by using piKL again with the piKL-BR policy as both the anchor and rollout policy. Empirically, piKL3 seems to improve human-AI performance on Hanabi against a baseline of simply training a best-response to the behavior-cloned policy.
strength_and_weaknesses: The writing of the paper is quite good and the algorithm is described clearly. There are some nice algorithmic insights in the various pieces of the piKL3 algorithm. The empirical results in Table 3 also show a clear improvement over the baseline of BR-BC. 

My main concern with the paper is that piKL3 is both complex and computationally expensive, and I'm not sure the limited evaluation justifies all the complexity. I understand that the human experiments are expensive to run, but it seems like there are a lot of other ways the authors could have done ablations or better evaluated whether all pieces of the algorithm are fully necessary. For instance, one could train a proxy agent using BC on some held-out data (possibly from a different population) not used for the algorithm. Ideally, there would be experiments showing what happens if you train piKL-BR directly on the BC policy rather than on the piKL-IL policy, or what happens if you don't add piKL on top of piKL-BR, etc. With the current evaluation, it's very hard to tell where the improvement over BR-BC is coming from—it could be a combination of all parts of the algorithm, or just one. The authors argue that "in practice, researchers may use any combination of the three components of piKL3 based on the properties and challenges of their specific domains." However, given how expensive some components are (most researchers in academia do not have access to 500 GPUs), it would be ideal if readers could get a better sense of which components are the most important and decide if the computational expense is worth it for each one.

Other specific issues:
 * In Table 1 I don't understand exactly how $\lambda$ can be varied when evaluating piKL-IL self-play. Isn't the output of piKL-IL, as you describe, a single neural network policy which ends up being the average of piKL policies with various values of $\lambda$? How can the $\lambda$ value of this be changed when evaluating it?
 * Where is footnote 3?

Typos:
 * Page 6: "whose input contains the mean of Gaussian distribution" -> "mean of Gaussian distributions"
 * Page 9: "all life token lost" -> "all life tokens lost"
 * Page 9: "the population of the human testers have a profound impact" -> "the population of the human testers has a profound impact"
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of the writing and experiments that are done seems to be quite good. I do think the evaluation is not comprehensive enough. The ways the authors use piKL also seems to be novel. In general there are enough experimental details for reproducibility.
summary_of_the_review: Overall, the paper is good except that it could use a better evaluation with ablations to justify the complex, multi-stage algorithm that is proposed. Therefore, I don't think it is quite ready for publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers the multi-agent human-AI collaboration setting, and the challenge of coordinating with humans. Specifically focusing on the Hanabi benchmark task, the authors propose the piKL3 method which as the following components:
(1) a human model that captures diverse skill levels controlled via a \lambda regularization term, 
(2) train a human-like best response model, 
and (3) further apply PiKL on the trained best-response model. 
They evaluate their method on Hanabi, showing marginal improvement over a BC baseline. 
strength_and_weaknesses: 
While I believe multi-agent human coordination is a challenging and interesting problem, I overall found the paper quite difficult to follow. 
My concerns include:

1. The gains of the proposed pikl3 method over baselines are not that strong at all, and is only evaluated on one task (Hanabi)
2. The methodological contribution largely seems to be via a regularization term, and applying piKL to multiple stages of the multi-agent task, which seems quite limited in novelty to me
Minor: \lambda  controls the regularization of unconstrained search towards human data, which is a bit different than different human "skill levels", as the spectrum is more about the degree of human-like behavior, making the overall point of different levels a bit confusing

clarity,_quality,_novelty_and_reproducibility: I found the paper quite unclear to follow, with the Experiment details section particularly confusing. I was also confused about the  difference between the "2nd" and "3rd" stages of pikl3. There are too many acronyms/sub-methods (pikl-IL, pikl-LBC) that make the overall prose quite challenging to follow. I would encourage the authors to add more subsections and outlining to the experiments section.
summary_of_the_review: Because I found the paper quite difficult to follow and limited in terms of novelty and performance of the proposed method, I lean towards reject. However, because of my unfamiliarity with the related works, this is a low confidence review.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The work extends piKL (an existing work for learning human-like policies) to coordinate with humans. They showcase their results on Hanabi benchmark. 
strength_and_weaknesses: Strengths : 
1. I agree to the problem statement of being of high importance with upcoming HiL methods where coordination with the human model / policy should be accounted for. 

2. I like the fact that the work attempts to model humans of varying skill levels. 

Weaknesses : 

1. I was quite disappointed with the background section as the paper mentions piKL several times and admits to extend that work, but fails to provide any background details about this existing work. 
2. Why Partial Observability is so important? Again, I feel with improved related work sections this could have been solved. The work mentions two key things in their setup - need for coordination with real humans as well as that agents exist in partially observable environments. Are there works that have already solved full observability? 
3. The work keeps mentioning SPARTA several times in the related work section & some of its details - but not piKL? Is there a reason for it? 
4. How feasible is such a technique in other environments (other than Hanabi) where 200k + simulations may not be available?

clarity,_quality,_novelty_and_reproducibility: I found that the paper lacked clarity regarding the overall objective and the methodology proposed. Without the introduction of piKL, it was extremely challenging to realize the different variants as piKL IL, piKL - LBS, piKL BR. I feel that I could have appreciated the work much better had it been more clear.
summary_of_the_review: I found that the work attempts to solve an interesting problem, relevant to the HiL community however lacks good readability to be accepted as a paper in ICLR. I would recommend the authors to update the write up to accordingly motivate the problem and highlight the contributions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a framework for training RL agents to collaborate with human players in 2-player Hanabi games. The key challenge of training such a policy is that human data is finite and fixed, which means that there needs to be a human-like behavior policy for the RL to learn to collaborate with. Prior work shows that such human-like policy trained with naive behavior cloning may make mistakes that can be exploited by the RL agent, resulting in the RL agents exhibiting strategies that don't mesh well with real human players. The paper proposes to use piKL, a prior search-based method that generates strong yet human-like plays, as the human policy to train the RL policies with. On the side of training the human policy, the method introduces (1) an iterative training scheme to improve piKL (the human policy) by feeding its successful play back to its dataset and (2) a way to learn human policies with different expertise levels by tuning the regularization strength. On the side of RL training with the human policy in-the-loop, to prevent RL overfitting to the human policy, the method proposes to further regularize the trained human policy by preferring “human-like” actions instead of greedily choosing the best move. Finally, the method also proposes to augment the RL policy with a search-based strategy to guard against catastrophic failure caused by out-of-distribution moves by real human players. The method is evaluated primarily via playing with real human players. The authors built a customized game server and solicited players of varying expertise level to play against the trained agent. The paper shows that the proposed method is able to outperform the RL policies trained with naive BC-based human policies. 

strength_and_weaknesses: Overall I quite like the spirit of this paper — for human-machine collaboration, human data is inherently scarce, especially for methods that rely on massive simulation and trial-and-error imitation learning. Training a human-like policy from a fixed set of human demonstration to generate reactive response for RL learning seems to be a good strategy. The proposed method is also conceptually simple — use a constrained BC method to plug “holes” in the human policy to prevent the RL learner from exploiting these errors to develop un-human behaviors. Finally, the scale of human study is laudable. 

I don’t have major concerns. A few minor ones: 
- Does \lambda truly capture the level of expertise in eq.2? Intuitively it’s an interpolation between “optimal play” and “human-like” play instead of different expertise levels. Besides, how much does training the policy against these policies matter in practice? There doesn’t seem to be an ablation study on this.
-  The general recipe for learning human-like behavior policy to train RL collaborators seems to be general and can be applied to other domains. The authors also seem to agree with this point in the “future work” section. The authors should discuss which aspect of the proposed method needs to be adapted in order to accommodate new domains.
- A piece of related work that’s missing from the paper is  Wang et al. [1]. They explored a similar venue of idea, including elements such as learning a human-like behavior policy to train RL collaborators and modeling diverse collaboration strategies, albeit in a robotics domain. It’d be great if the authors could compare and contrast their work with [1]. 

[1] Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration, Wang et al., 2021

clarity,_quality,_novelty_and_reproducibility: Quality: great execution of a conceptually simple idea. Impressive empirical evaluation.
Clarity: great writing.
Originality: the problem setting is rarely studied and seems to be important for the future research of human-AI collaboration.
summary_of_the_review: Overall I quite like this paper. See my main comments for the remaining concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces piKL3, a three-step algorithm that performs well while playing the game of Hanabi with an ad-hoc team of human players. The experimental results show that the algorithm has the best human-AI performance in the game of Hanabi.

One of the main criticisms of the paper is that the proposed algorithm is very complex and computationally intensive. While it shows performance gains, reviewer ywYA says that there is not enough analysis in the paper to justify different components of the algorithm. I would ignore other complaints which are all very minor or irrelevant. As reviewer ywYA says, while the paper is very interesting, it is not yet ready for publication. I encourage the authors to incorporate the feedback from the reviewers, add more extensive analysis and resubmit to a future venue.

justification_for_why_not_higher_score: The paper has an algorithm which does ~1 point better than the previous methods. But there is not enough analysis to justify why we need such a complex algorithm for 1-point improvement. Hence I recommend reject.
Decision: Reject