Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper conducts a user study in which humans play alongside a rule-based AI bot (SmartBot) and a learning-based AI bot (Other Play) in the cooperative card game Hanabi, and then are asked to rate the bots along several axes. The key finding of interest is that while both bots achieve similar performance in play with humans (about 14 or 15 on average), human players significantly preferred the rule-based AI bot, and this preference increased in strength with additional expertise (i.e. the most expert humans most strongly preferred the rule-based bot.)
main_review: I like this paper a lot: much work has been done in multiagent RL, and recently it has started to look into human-AI teaming as well, but so far the focus has primarily been on optimizing the reward obtained, with at best qualitative evaluations of other factors. These other factors seem very important for the deployment of human-AI teaming in the real world, so it is good to see a paper explicitly aim to quantify and analyze these factors. The survey design and analysis seem sensible (though I do not know much about these topics, and might be missing something). The paper is very clear; I especially appreciated the abstract, which laid out exactly what I should expect from the paper (in contrast to most abstracts, which seem to be vague and hiding information about the proposed method).

Ambition
--------

My one critique of the paper is that it could have been more ambitious and had a broader scope. For example, it could have studied multiple domains, to check whether the detected effects generalize to these other domains. As the authors mention, alternatives include StarCraft and Overcooked; the authors could adapt the [existing website](https://humancompatibleai.github.io/overcooked-demo/) for Overcooked in which humans can play against trained agents.

Given the results, it would also be nice to test learning agents that are specifically trained to play with humans. It sounds like participants disliked Other Play because of its alien play style and because it “ignores” information provided by humans. An agent that is trained to play with humans is less likely to fall into such a trap. (That being said, I am not aware of any learning agents that are trained to play with humans on Hanabi, so this may not have been feasible.)

It would also be interesting to learn more about qualitative reasons for participant’s responses -- it seems there were multiple free response questions. Did participants mostly not answer these? If they did, what did they tend to say?

Finally, given that this is (to my knowledge) the first paper studying other aspects besides reward in human-AI teaming for agents based on deep reinforcement learning, it might have been nice to mention other qualitative results that have been informally reported, such as in the cooperative matches for OpenAI Five and Capture the Flag, in which some bots and humans play together on a team. I know that there are some public comments on at least OpenAI Five’s cooperative games. This could capture a broader range of qualitative effects that we may expect to see in human-AI teaming.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a comparative study on the objective and subject performance of a rule-based agent and a RL-learned agent for playing the card game Hanabi with human teammates. The authors selected each of the best-performing existing Hanabi AI from the rule-based methods and learning-based methods, and conducted a user study where human participants are invited to play the game with both types of AI agents and the team performances as well as the human’s subjective feedback are collected. The results show that, despite no significant differences in performance, the rule-based AI agent is preferred by human participants over the learning-based agent.


main_review: Pros: This work is the first to study the human teammate’s subjective evaluation of different types of AI agents for the card game Hanabi. The user study is carefully designed and controlled to offer a comprehensive analysis of the two selected candidate AI agents. The results of the user study are presented in detail with clarity and the claims are supported with statistical analysis. The findings imply that human subjective evaluations are also important metrics for designing AI agents since the task performance is not indicative of subjective evaluations.

Cons: One concern for this paper is that the selection of candidate AI agents for testing is too limited (one from each category) and therefore the results may not generalize across different agents within the same game. Especially that RL training is an unstable process and the paper only tested one policy trained by RL to represent the full range of learning-based agents.
On the other hand, the findings do not seem to be too surprising since the rule-based AI agent is designed with hard-coded human knowledge and therefore are expected to play smoothly with human teammates while the RL-based policy was trained to optimize for zero-short coordination but the test scenarios in this user study is few-shot coordination. Moreover, the RL agent was not explicitly designed to perform well with experienced human teammates who have developed their conventions for communication. 

While I agree with the implication from the findings of this paper that human experiences/evaluations are important metrics on their own, I do not think this directly indicates that subjective metrics should be incorporated into RL training to achieve good human evaluations. Experiments and comparisons with RL agents trained or fine-tuned with human teammates are needed to show whether it is necessary to directly optimize for human subjective evaluations. 

limitations_and_societal_impact: The authors commented on how this work could influence future research on AI agents and how this can have a larger impact on the users.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents the results of a user study that enlisted 29 participants to play the cooperative game of Hanabi. The results suggest that humans prefer to play with rule-based agents than with reinforcement learning (RL) agents despite the average game score being nearly identical with either type of agent.

main_review: The paper is well written and deals with an important problem, which is to understand how humans perceive their interaction with RL-based agents in cooperative settings. The game of Hanabi is a great choice for studying this problem as it is one of the standard benchmark problems for evaluating RL algorithms in cooperative settings. The choice of the agents used in the study is also good: Smart Bot (SB) and Other-Play (OP), which are the current state-of-the-art rule-based and RL-based algorithms for playing Hanabi. 

The results of the study show an interesting trend: objective metric of number of points scored is similar for both agents, but subjective metrics such as whether the agent "understands" the human player's intention mostly favor the rule-based agent. I have played Hanabi with older RL agents such as policies trained with DQN Rainbow and I know that it can be frustrating to play with RL agents. However, I have not played with agents trained with the newer OP algorithm. When I read the OP paper I had thought that it would be fun to play with OP agents because humans were shown to score much higher with OP agents (while I can score around 5 or 6 with DQN Rainbow, the OP paper showed that experienced players can score ~15). This submission is telling me that "hey, you might score ~15 with OP agents, but you will still prefer to play the SB agent." I think this is an interesting result that needs to be investigate more carefully. The research community needs to get to the bottom of this result and understand what is going on. 

After my initial excitement with the result, I started to be concerned with its validity for the authors enlisted a small number of people to perform their study. A careful analysis show that the results might be too preliminary and we would need a larger study to confirm the trend observed in the current submission (see details below). 

Participants of the study played 3 games with each agent (either SB or OP and then with the other agent). The results suggest (according to the paper) that there is a learning effect, which is expected: participants score higher as they play the second block of games. The experimental design deals with learning effects by controlling the number of people that gets to play each agent first. It would be helpful to know exactly how many of the 29 participants played OP first and how many played SB first. Small variations in the distribution could affect the results due to the strong learning effect of the design. A cleaner approach is to use a between-subjects design, where each participant plays either SB or OP. Then the learning effect won't be carried across different treatments of the study. 

I am also concerned with the incentive design of the study. Each participant received a 10 USD gift card and the highest-scoring participant received an additional 50 USD gift card. I wonder if a different compensation model (e.g., participants receive more if score higher in the game, independently of the other participants' scores) would change the results. Figure 1 shows that the distributions of scores of OP and SB are different. In particular, there is a peak for the OP results for the score of 24 (maximum is 25). I wonder how people who played those matches ranked OP compared to SB. Would the participants who scored higher with OP than with SB still prefer to play with SB -- even if OP gives them a higher monetary reward? 

A careful analysis of the distributions shown in Figure 1 also shows that the number of participants enlisted in the study is probably too small to support significant conclusions. For example, no participant scored 15 with SB and I don't see why 15 would be too different from 14 or 16. The lack of scores 15 makes the distribution look bimodal, but I suspect this is just the effect of a small number of participants. 

The distinction between experienced and inexperienced players seems to play an important role in these user studies (e.g., the one presented in the OP paper). However, no one has proposed an effective metric of experience for Hanabi that allows others to reproduce the results. There could even be a cultural effect of how people regard their own experience in the game. Perhaps one can formulate a Hanabi test to allow for a concrete evaluation of one's experience?

In short, the results presented in this paper are interesting, but perhaps too preliminary for publication. I suggest that the study is repeated with a between-subject design, a more lenient reward function to make people prefer higher scores, and with many more participants. It would also be helpful to report demographic data about the participants and also the effect size of the comparisons.
limitations_and_societal_impact: The main limitation of the paper is the pool of participants used in the study. The number of participants enlisted is very small and we have no information about their diversity. Which part of the world's population is represented in the pool of participants? The paper doesn't present any demographic data and this is certainly a limitation.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper evaluates a human behavior with and perception of Hanabi agents when paired with human players, comparing a top-performing rule-based and learned agent. Results show no significant differences in team performance when comparing the two agents. But humans show a preference for the rule-based agent in terms of subjective ratings of trust, perception of agent skill, understanding of the agent, and preference as a partner. Some results suggest high skill (by self rating) players more strongly distrust the learned agent when compared to novices. Together the results describe the current state of how Hanabi AI agents are perceived as teammates, opening future investigations of ways to bridge this trust/preference gap.

main_review: The paper has several key strengths:
- Topic relevance: The paper investigates an important (at least in near term) question around how human-AI collaboration is experienced by humans.
- Rigor: Overall the empirical analysis is well-described and uses statistical tests where appropriate.

The paper has some modest weaknesses:
- Limited generality: Any human study will have necessarily limited generalization to other problem domains. The value in this work is the insight into the problem and as a good demonstration of appropriate methodology for others to apply to other domains.
- Scale: The study is limited to 29 participants, which is modest for the variety of questions to be addressed. Regardless of statistical power it would be more compelling to see a larger scale replication of this work, particularly including a more balanced sample of participant skill levels.
- Representativeness: Results reporting do not give a sense as to whether the sample should be considered "representative". Or what a representative population might be for something like Hanabi. This makes it hard to assess the whether the outcomes would transfer to the general population of human-AI teaming for Hanabi.
- Missing results: Many negative results are not included in the main text. That is fine, but it would help to include these in the appendices. The narrative of the results may change depending on what other measures of cooperation and task load (TLX) were *not* significantly different.

Below are comments specific to the review criteria prompts.

# Originality

The work extends previous contributions by doing a comparison among agent authoring methods. The task/methods are not new and that is a good thing for this sort of empirical evaluation. 

The topic of human-AI teaming in terms of human perception has not been adequately answered in the past for this domain in specific or more generally. Related work is adequately cited and highlights this point. Related work also appropriately cites relevant materials for the algorithms studied.

# Quality

The empirical analysis uses appropriate statistical tests and shows significant effects. This is clearly a complete piece of work (aside my desire to see a larger population used in the experiment, which is beyond the scope of this paper per se).

I would ask the authors to include the non-significant effects in a supplement or appendix to understand how they fit with the narrative presented. The focus on only positive results is reasonable, but means it is unclear whether contradictory results on related questions are being omitted.

The authors are clear on the ways the work may not generalize due to the test domain (Hanabi), while still addressing what aspects of the work may generalize (expert distrust of AI).

One comment: Figure 1 makes note that "no player achieved a perfect score with OP". This comment is true, but overlooks that seemingly many more players achieved a near-perfect score with OP compared to SB. So it's not clear what point this note is intending to make.


# Clarity

The submission is clearly written and generally well organized.

One clarification (if possible without violating IRB and privacy concerns): how was the experiment population selected? Or what is it's composition? This would help understand the limitations to the generalization of the results to other populations.

Given many NeurIPS readers are not necessarily well-versed in experiment design and analysis it would help to inform the reader more about the choice of statistical tests. For example, why use a one-sample t-test? 

Others could readily reproduce this work, aside from the choice of sample population.

# Significance

The results are important empirically and drive the need for others to extend this work and incorporate this type of evaluation in future efforts with human-AI teams. Others *should* use these results to inform how human-AI agent teams are assessed; the paper reveals a clear gap in prior approaches to evaluation.
limitations_and_societal_impact: The paper addresses the potential societal impacts of human-AI teaming should the paper results generalize. The work itself is an evaluation and thus is not necessarily likely to have negative societal impacts by sharing these results.
review_title: null

Meta-review: Reviewers were unanimous that the paper addresses an increasingly important area of research (human-AI collaborations), focuses on an understudied but crucial aspect of the research (i.e., how do humans perceive and assess their AI teammates?), and offers insightful results. While they found the paper well-written overall, reviewers suggested several avenues for further clarification and improvement, in particular, they would like the authors to further elaborate on their choice of the learning agent, the number and representativeness of participants in the study, and their arguments around ecological validity.  Given the authors’ comprehensive response to the reviews, I am confident they can revise their paper accordingly, so I recommend acceptance.
Decision: Accept (Poster)