Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors train agents with self play that have diverse preferences that differ from the original task and show that having a richer pool of agents generated with this technique yields policies that are better adapted to human coordination. 

strength_and_weaknesses: The empirical results are excellent. In particular, I think the combination of ablations, human experiments, scripted bots, and imitation-trained policies go beyond most any other works and clarify key issues that were not carefully analyzed in previous works. For instance, the authors clearly show that the techniques used in most evaluations (imitation-learned policies and human interaction) are highly confounded since people adapt, and the imitation-learned policies don’t show much diversity. Their method only weakly improves over baselines in these tasks. In contrast, the use of specific scripted probes and a more qualitative evaluation revealed large discrepancies between actual coordination performance in the most important edge cases. 

The algorithmic contribution is a weakness as it depends on significant hand-tuning of custom features specific for these specific Overcooked environments. I do not see how this approach could be easily adapted to a new tasks (or even an Overcooked level with different dynamics). On its own, I do not think this algorithm is a sufficient contribution to literature. I would have also liked to see comparisons or thoughts on more model-based towards generating diversity in Overcooked for example: Wu, Sarah A., et al. "Too Many Cooks: Bayesian Inference for Coordinating Multi‐Agent Collaboration." Topics in Cognitive Science 13.2 (2021): 414-432.


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written with significant details available in the appendix. The work is original. The units in Figure 5 should be explained in the text 

summary_of_the_review: Based on the new method alone I would not accept this paper. However, the thoroughness of evaluation sets a new standard and I feel that I learned something new and important from these empirical analyses. I would like to cite this paper in the future and that should be sufficient for acceptance. I would raise my score further if the authors can more greatly emphasize these contributions in their work
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the Hidden-Utility Self-Play algorithm to explicitly model human bias via a modification to the reward function used during self-play training. This is used in the domain of the cooperative game Overcooked, where a particular reward modification is shown to be effective both in experiments where it is paired with learned human models and in experiments with actual humans. The framework is demonstrated to be especially effective with real human partners.
strength_and_weaknesses: Strengths
- This paper focuses on the benefits of incorporating human biases in cooperating with them, an important area.
- The paper is written clearly, and the appendix incorporates many details of the experiments that aid reproducibility.
- The ablation studies and other experimental details are presented in a thorough way.

Weaknesses
- It is not clear how well the approach presented here would generalize and requires domain knowledge to be effective.
- The human study, which shows the most compelling evidence for the effectiveness of this framework, needs more details. How were the volunteers recruited, how did they participate in the experiment, how long did the experiments take, what was their experience with Overcooked, etc.? The volunteer group is also younger and heavily male. How may this impact the results?
- The "Distant Tomato" regime has a clear optimal policy - would just telling the human participants what that is have substantially impacted results? Specifically, is this a question of a "hidden reward function" (as the framework presents) or inadequate skill or knowledge of the domain by the human participants?
clarity,_quality,_novelty_and_reproducibility: This paper shows the effectiveness of a variation on a popular technique in which there is a hidden reward function to augment training, expands the Overcooked domain with two new layouts, and shows the effectiveness in human studies; these are all reasonably novel contributions. There are adequate details for reproducibility (although, as mentioned above, more details on the human study should be included, possibly in an expanded Appendix with some detail in the main paper).
summary_of_the_review: This is an interesting paper that shows the effectiveness of incorporating models of human biases in a particular domain. The main weaknesses are the question of how domain-specific this reward is (and how much domain knowledge it requires) and inadequate details about the human study. For example, in Section F.4.1, slowing down the speed of the simulation is noted for making the users more comfortable in contributing; these are the kinds of factors that could substantially impact the effectiveness of a solution, in addition to algorithmic contributions. The qualitative details in F.4.2 could also be augmented with quantitative metrics of the feedback, if available.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for learning a collaborative agent that is able to handle arbitrary and/or suboptimal human reward functions. The method leverages self-play, specifically fictitious co-play, where the policies the target policy plays alongside are learned according to different reward functions. A variety of reward functions are used to train the pool of policy in order to train the target policy alongside a variety of different preference types and to teach the target policy to be robust to different, potentially previously unseen, preferences. The proposed method is evaluated on the Overcooked task, a standard benchmark for human-agent collaboration, alongside learned human models, scripted policies, and real humans. Across the different "humans", the proposed method outperforms the baselines. The experiments ablate several aspects of the approach: how the policies in the pools are trained and the batch size used to train the target policy. The results suggest that both filtering and MEP policies (trained with a single reward function) trained policies are important and large batch sizes are needed to stable target policy performance. 
strength_and_weaknesses: Strengths:
- Overall a well written paper.
- The method is well motivated for why it is needed and is well grounded in the existing literature.
- The experiments appear to be solid and compare against important baselines to evaluate different aspects of the proposed contributions. The authors made sure to explicitly evaluate how well the approach works when the target policy is deployed to collaborative with agents that have reward functions not included in the policy training pool. Additionally, the authors evaluate against a variety of "humans", including real humans. Assessing the method across different "humans" provides a good idea of how the method extends to the real world. 
- The use of random reward search to create a diverse pool of policies seem simple, but also effective. 
- The authors discuss limitations of the current approach and propose future work to address the limitations.

Weaknesses:
- It would be nice to see a discussion about what the ablation study results tell us about the proposed method. What does it mean for the MEP policies to be helpful? What does it mean about the method for a larger batch size to be important? How does the need for a large batch size align with what the baseline methods needs?
- It is unclear to me how relevant the paper is to the topic areas ICLR focuses on. There is little to no discussion of representations nor representation learning methods. The paper seems like it is a better fit for CoRL, AAAI, AAMAS, and ICML.
clarity,_quality,_novelty_and_reproducibility: In terms of clarity/quality, it would be helpful to clarify the following points:
- It would be beneficial to more clearly point the reader to where "Asylum. Adv.", "Coord. Ring", and "Counter Circ." are specified in the appendix so that they may better follow the results and take aways. I missed the pointer to the appendix and it made following the results and take aways more challenging. 
- It was unclear to me exactly how the target policy's adaptation occurs. Is there any adaptation happening online while the policy is interacting with one of the humans? Or is the policy "adaptive" because it has experienced co-play with agents trained according to a variety of different reward functions? Is the contribution only to train the policy pool on random reward functions?
- It would be nice have it made clear what the criteria for a "fair comparison" (6. Experiments | Baselines), which is used to motivate the pool size. 
- Why was 5 standard deviations used instead of 1 when assessing performance? (6. Experiments | Baselines)

In terms of novelty:
- The paper builds upon existing methods, but modifies how the pool of policies is trained. Based on the given literature review, the introduction of random rewards for policy pool training appears novel. However, I am not an expert in self-play and fictitious co-play. 

In terms of reproducibility:
- The authors have not made code available and it would be helpful for them to release their code. 
- The appendix has a lot of implementation details and looks sufficient for reproducibility. However I did not attempt to reproduce and have not previously implemented the baselines, so cannot definitively say if sufficient information is given.

summary_of_the_review: Overall the paper is strong with some areas for improvement in terms of clarity. The idea is relatively simple and based on the results appears to improve policy performance, especially in experimental conditions including real humans. My only concern about acceptance to ICLR is its fit. The paper seems like a better fit to CoRL, AAAI, AAMAS, or ICML as it does not have a representation nor representation learning focus.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel method called Hidden-Utility Self-Play (HSP) to solve a realistic and challenging problem that there exist human biases in human-AI interaction. The main contribution of this paper is proposing a hidden reward function to model human biases. The experiments are solid and sufficient to verify the performance improvement of HSP.
strength_and_weaknesses: ## Strength
1. This paper is well written and easy to follow. The related works are sufficiently discussed.
2. The experimental settings and the baseline algorithms are described in details.
3. Although this paper is an application work, it still includes some theoretical analysis as the base to support the proposed algorithm. The theoretical analysis is neat and sufficient to support the idea.
4. No obvious flaws in my view.


## Weaknesses
1. The construction of the hidden reward function is implemented as the counting-based method which is too simple. Since the hidden reward function is belonging to a type of intrinsic reward, there are plentiful literatures that study on the construction or learning this reward. I suggest the authors can have a review on these and improve the work.
2. In the procedure of policy filtering, the greedy policy selection suffers computational burdern as the authors illustrate. I wish the authors give a discussion on the possible way to improve it in the future work, since I don't wish the performance improvement is just based on more computation which is not good for the following works.
clarity,_quality,_novelty_and_reproducibility: The clarity of this paper is good and every claim is well justified.

The novelty of this paper is good, mainly depending on the novel practical research problem I have never seen before. Nevertheless, I have to say that adaption to the unknown agent has a long history of research in the multi-agent learning. The research problem discussed in this paper can be seen as a special case, where an agent is replaced by human being.

The originality of this paper is good, because of the novel research problem and the experimental design.

The reproducibility of this paper is generally good, since it provides the details of experiments. However, I am concerned about the experiment done with human beings, which could be difficult to be reproduced.

Overall, the quality of this paper is high.
summary_of_the_review: This paper proposes a novel practical research problem and a novel algorithm to solve it. Besides, it has done solid and sufficient experiments to verify the performance of the proposed algorithm.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper explores how to accommodate for human biases when training agents for human-AI collaboration without data from human policies. The proposed method is demonstrated in Overcooked (a popular choice recently in studies of human-AI collaboration) but is in principle more general then this game alone provided domain-specific knowledge of events likely to motivate human behavior. This requirement of domain-specific knowledge weakens the original motivation and benefit of not needing data from human policies. All reviewers saw positive contributions from the paper suggesting the paper is of interest to ICLR attendees, but the authors may reach a larger audience by also promoting the work in more core AI venues.
justification_for_why_not_higher_score: Approach is motivated by not needing data from human policies, but does assume some domain specific knowledge of likely human motivations.

Core problem likely of interest to only a sub-community within ICLR, arguably would reach a larger audience at other conferences.
justification_for_why_not_lower_score: Unanimous support for the paper across all reviewers with a variety of contributions identified by all.
Decision: Accept: poster