Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi

Abstract: Deep reinforcement learning has generated superhuman AI in competitive games such as Go and StarCraft. Can similar learning techniques create a superior AI teammate for human-machine collaborative games? Will humans prefer AI teammates that improve objective team performance or those that improve subjective metrics of trust? In this study, we perform a single-blind evaluation of teams of humans and AI agents in the cooperative card game Hanabi, with both rule-based and learning-based agents. In addition to the game score, used as an objective metric of the human-AI team performance, we also quantify subjective measures of the human's perceived performance, teamwork, interpretability, trust, and overall preference of AI teammate. We find that humans have a clear preference toward a rule-based AI teammate (SmartBot) over a state-of-the-art learning-based AI teammate (Other-Play) across nearly all subjective metrics, and generally view the learning-based agent negatively, despite no statistical difference in the game score. This result has implications for future AI design and reinforcement learning benchmarking, highlighting the need to incorporate subjective metrics of human-AI teaming rather than a singular focus on objective task performance.


Reviews and author responses:

 Thank you for the response and the suggestions. We will include the suggested changes if the paper is accepted, either in the main paper or in the supplemental materials as appropriate.

 Thank you for your detailed response and for clarifying about the statistical tests of the study. I will adjust my score accordingly. 

If the paper is accepted, please include the following information in the final version: 

"Regarding the specific participants brought up by the reviewer: There were 3 participants who obtained a score of 24 with OP, with one of these participants obtaining 24 twice. All three replied at the extreme end of our Likert scale (i.e., 1 or 7) with a preference for SB.

Interestingly, their cumulative scores for OP and SB, respectively, were: Participant A: 57 and 28, having played with OP first Participant B: 68 and 48, having played with SB first Participant C: 70 and 35, having played with OP first For reference, our winner (Participant D) obtained a cumulative score of 68 and 54 with OP and SB, respectively, having played with OP first, and preferred SB with a Likert scale rating of 6. Their self-rated Hanabi experience level was 7, 7, 6, and 7 for Participant A, B, C, and D, respectively."

As well as demographics information and effect sizes.

This paper presents the results of a user study that enlisted 29 participants to play the cooperative game of Hanabi. The results suggest that humans prefer to play with rule-based agents than with reinforcement learning (RL) agents despite the average game score being nearly identical with either type of agent.
  The paper is well written and deals with an important problem, which is to understand how humans perceive their interaction with RL-based agents in cooperative settings. The game of Hanabi is a great choice for studying this problem as it is one of the standard benchmark problems for evaluating RL algorithms in cooperative settings. The choice of the agents used in the study is also good: Smart Bot (SB) and Other-Play (OP), which are the current state-of-the-art rule-based and RL-based algorithms for playing Hanabi. 

The results of the study show an interesting trend: objective metric of number of points scored is similar for both agents, but subjective metrics such as whether the agent "understands" the human player's intention mostly favor the rule-based agent. I have played Hanabi with older RL agents such as policies trained with DQN Rainbow and I know that it can be frustrating to play with RL agents. However, I have not played with agents trained with the newer OP algorithm. When I read the OP paper I had thought that it would be fun to play with OP agents because humans were shown to score much higher with OP agents (while I can score around 5 or 6 with DQN Rainbow, the OP paper showed that experienced players can score ~15). This submission is telling me that "hey, you might score ~15 with OP agents, but you will still prefer to play the SB agent." I think this is an interesting result that needs to be investigate more carefully. The research community needs to get to the bottom of this result and understand what is going on. 

After my initial excitement with the result, I started to be concerned with its validity for the authors enlisted a small number of people to perform their study. A careful analysis show that the results might be too preliminary and we would need a larger study to confirm the trend observed in the current submission (see details below). 

Participants of the study played 3 games with each agent (either SB or OP and then with the other agent). The results suggest (according to the paper) that there is a learning effect, which is expected: participants score higher as they play the second block of games. The experimental design deals with learning effects by controlling the number of people that gets to play each agent first. It would be helpful to know exactly how many of the 29 participants played OP first and how many played SB first. Small variations in the distribution could affect the results due to the strong learning effect of the design. A cleaner approach is to use a between-subjects design, where each participant plays either SB or OP. Then the learning effect won't be carried across different treatments of the study. 

I am also concerned with the incentive design of the study. Each participant received a 10 USD gift card and the highest-scoring participant received an additional 50 USD gift card. I wonder if a different compensation model (e.g., participants receive more if score higher in the game, independently of the other participants' scores) would change the results. Figure 1 shows that the distributions of scores of OP and SB are different. In particular, there is a peak for the OP results for the score of 24 (maximum is 25). I wonder how people who played those matches ranked OP compared to SB. Would the participants who scored higher with OP than with SB still prefer to play with SB -- even if OP gives them a higher monetary reward? 

A careful analysis of the distributions shown in Figure 1 also shows that the number of participants enlisted in the study is probably too small to support significant conclusions. For example, no participant scored 15 with SB and I don't see why 15 would be too different from 14 or 16. The lack of scores 15 makes the distribution look bimodal, but I suspect this is just the effect of a small number of participants. 

The distinction between experienced and inexperienced players seems to play an important role in these user studies (e.g., the one presented in the OP paper). However, no one has proposed an effective metric of experience for Hanabi that allows others to reproduce the results. There could even be a cultural effect of how people regard their own experience in the game. Perhaps one can formulate a Hanabi test to allow for a concrete evaluation of one's experience?

In short, the results presented in this paper are interesting, but perhaps too preliminary for publication. I suggest that the study is repeated with a between-subject design, a more lenient reward function to make people prefer higher scores, and with many more participants. It would also be helpful to report demographic data about the participants and also the effect size of the comparisons. The main limitation of the paper is the pool of participants used in the study. The number of participants enlisted is very small and we have no information about their diversity. Which part of the world's population is represented in the pool of participants? The paper doesn't present any demographic data and this is certainly a limitation. 

 Thank you for the review and the suggested changes.

We acknowledge and agree with many of the concerns expressed by the reviewer. In particular, it is certainly true that a larger sample size and more diversity in self-rated Hanabi experience could provide additional insights to our findings. Future iterations of our analysis will include automating our experiment and facilitating our data collection process to perform the suggested replication as well as probe additional questions. Nonetheless, as noted, our pool size was sufficient for the statistical analysis performed, and the concern about the number and variety of questions addressed is mitigated by our post-hoc multiple comparisons corrections.

Regarding generalizability and representativeness, we agree that these concerns of ecological validity are valid, and appreciate them being brought up. The primary objective of this paper was to bring attention to cases where objective performance metrics alone are insufficient for effective human-AI teaming, which we believe is generalizable and applicable to other problem domains. It is necessary to begin accounting for human-perception factors in Human-AI team settings since it will be critical for adoption and deployment in real-life scenarios.

The issues of ecological validity are, however, quite important in the context of Hanabi, but not easily addressed, and will likely take substantially more work beyond what we can provide in this paper (or any single paper). As the reviewer notes, it is unclear what a representative population for Hanabi players might look like. How to answer this question is less clear here than for games like Chess, Go, and StarCraft, given the tight coupling and interdependence between teammates, the development of team-specific conventions that occur across Hanabi-playing groups, and the way that players adapt to teammates of varying levels of experience and convention knowledge. Even the idea of “player strength” for individuals is very contextually-dependent and difficult to isolate, unlike in the aforementioned cases where systems like Elo, dan, and MMR may be used. Given these properties, even a significantly expanded sample could suffer from the same critique. What we can say is that based on our demographic data, our sample population comes from a variety of backgrounds, including multiple disconnected groups of Hanabi players and several non-players. We have also begun some very preliminary work in isolating the effects of deck ordering and determining individual player strength in Hanabi, but that is beyond the scope of this study. Similarly, future work might look at generalizations of such teaming trends to non-Hanabi team environments, such as Overcooked. 

Regarding the inclusion of non-significant results - we can certainly add a more complete statistical analysis to the appendix. Additional analysis of other data that we did not include at all – e.g., TLX measures and full analysis of open-ended text responses, is ongoing.

Regarding the statement “no player achieved a perfect score with OP,” we agree that the added statement may be superfluous, particularly without the context of the number of near-perfect scores with OP. We will consider how to reframe it, or we will remove it.

Regarding the experiment population - the participants came from a fairly diverse pool, since many (but not all) subjects were referred to us via word-of-mouth among several distinct and unrelated groups of Hanabi players (as well as individuals who did not regularly play Hanabi, or play Hanabi at all). Participants included students at several universities (affiliated to the researchers and not), unaffiliated researchers, unaffiliated working professionals, and several individuals from an online Hanabi community. We can add some additional information on the recruitment of subjects to our supplementary materials, within the bounds of the IRB protocol.

Regarding the use of the one-sample t-test, the test was used for post-hoc survey responses because we were determining whether the sample mean from the survey responses was different from the null hypothesis of no difference between the two agents (or a neural response of 4 on a 1 to 7 scale). We will add additional commentary describing our methodology to Section 3.2. 

Thank you again the time and effort for this review.


 Thank you for the comments and the suggested changes.

Regarding the selection of AI agents for evaluation, we acknowledge the reviewer’s comment on limited variation between agents of each category (learning and rule-based). Our justification in agent selection is as follows:

* The agents selected gave the participants the highest chance of achieving a high score since they are the “state-of-the-art” in their respective categories
    * Hanabi is a game designed with an explicit goal/objective: achieving the highest maximum score. In our experiments, the maximum sore is 25. 
    * Participants were incentivized to achieve the highest possible score via our reward structure.
    * Following our literature search, we selected the learning (OP) and rule-based (SB) agents based on their objective Self-Play performance (Table 1) and have therefore shown to be “good” (in the objective sense) Hanabi players. 
    * Thus, we posit that human-AI teaming in Hanabi would prove most effective if participants were partnered with AI teammates that were optimized for their incentives (achieving a maximum score and reward).
        * Anecdotally, we note that playing a cooperative game – for the most part – is more fun when we have an effective partner that increases our chances of winning (i.e., having a partner that is helping you achieve a higher score is more pleasant than having a partner which only helps you obtain an average score.).
* Lack of readily available implementations of Hanabi AI bots, as mentioned in the limitations of our work (Section 5.3), 

We also performed some preliminary testing with other learning-based agents (from [24]), but they 1) performed much worse than OP, and 2) were much more strongly disliked by human teammates than OP. From this, we concluded that it would not add much to the results to use a wider variety of learning agents, when OP appears to already be an example of best-in-class for cross-play. We can provide additional commentary for our agent selection in Section 3.1.

We would also like to clarify that we do not consider our experiment to be an instance of few-shot coordination from the agent’s perspective (though it might be considered as such from the human perspective). We obtained the pre-trained learning-based agent (OP) from its authors [18] and made no attempt at re-training prior to the experiment or after any game was played. We note that the OP agent was trained via self-play using different random seeds. As such, the agent could not have been exposed to any of the participants’ idiosyncrasies during its training process nor calibrated post-game. Thus, we consider the use and evaluation of the learning-based agent to be fair given that it was used in the exact scenario it was made to succeed in (zero-shot coordination).

We agree with the comment that the RL agent was not explicitly designed to perform well with experienced humans, though in fairness, we did not find any agent in the literature that was designed to play well, while using a human teammate’s level of experience as a prior. It might be argued that the SmartBot agent “was designed to” play well with experienced humans, but it simply had one set of rules to follow, among many possible conventions that humans actually use. Further work into agents that consider human experience (and convention use) for both learning and non-learning agents would be valuable, but outside of the scope of this work.

Regarding the incorporation of subjective metrics into RL training, we would like to clarify that we are not arguing that incorporation of subjective metrics is the only way to achieve good human-AI teaming. Incorporating subjective measures during training could result in a positive increase in human-perception of their AI teammates (e.g., increased trust between Human-AI teams). However, other approaches might also be taken to improve teaming. For example, AI teammates with legible/transparent and/or interpretable/predictable policies may also be perceived more favorably relative to current, state-of-the-art black-box methods, and self-play training paradigms could potentially be modified to account for these factors without direct human intervention during the training itself.

Regardless of which approach is used, we believe it is necessary to begin accounting for human-perception factors in Human-AI team settings since it will be critical for adoption and deployment in real-life scenarios. Our paper indicates a lack of such evaluations in the RL literature, even when dealing with multi-agent systems that hint heavily at an eventual goal of human-AI teaming or human-like intelligence, and we supply an example of the potential negative impacts that can exist when such factors are not considered. We can incorporate additional commentary can to Section 5.2 to make our intent clearer.

Finally, we agree with the reviewer that a learning-based Hanabi bot fine-tuned for human teammates could potentially improve their subjective evaluations. However, we have not encountered such an agent for Hanabi, and our current evaluation seemed a reasonable precursor to such work, given the lack of human evaluations of RL agents in general.

As Hanabi is quickly becoming de-facto RL benchmark for training and testing cooperative AI bots – learning and rule-based alike – given its simple cooperative nature, we believe it is a sensible starting point for evaluating Human-AI teams beyond objective metrics and eliciting the AI research community to think about human-perception factors (e.g., trust and understanding) when developing new methods. 

Future iterations of our analysis would expand to additional environments which have AI bots fine-tuned for Human-AI teaming readily available (e.g., Overcooked [9]).

Thank you again for the time and effort for this review.



 Thank you for the comments and suggested changes.

To address the primary concerns of the validity of the results due to sample size and various other aspects of the experimental design, we will first provide a broader response to the sample size issue, and then dive into the specifics presented, as well as other questions.

### Regarding sample size:

We first note that our study were limited by the requirement for experimenter presence for the experiments (typically ~3 hours per subject). Experimenters were required for logistical reasons of our software interface, to answer clarification questions, and collect observations on subject responses. The latter was particularly needed since audio/visual recording of subjects requires far more stringent human subjects research approval, and more post-processing time. An automated approach would allow an expansion of our subject pool, and is something we are working towards, but such a setup would also trade off with the kind of data that would be collected, as well as data quality. Since this type of study is relatively new for Hanabi, we opted for the more labor-intensive approach for this initial study, in the hope that it would improve data quality over the alternative.

With that said, we agree with the reviewer that a larger sample size could provide additional insights. It appears that there are two concerns here that we would like to address separately: whether the conclusions we have reached are likely true based on our data (statistical power), and whether or not the conclusions generalize (ecological validity).

In terms of statistical power – how many samples we need to correctly reject a null hypothesis – our statistical tests may be considered in two groups: preference differences (Likert scale, 1-7), and performance differences (game score, 0-25).

For preference differences, if we assume that we are looking for a rating difference of 2, with a standard deviation of 2 (using typical values of alpha = 0.05, power = 0.8), one-tailed testing would require 13 samples. It is debatable whether a “sample” should be a game (of which we had 174) or a subject (of which we had 29), but we had nearly double the number that was needed in the more conservative case. Expanding the assumed sigma out to 3 gives a required sample size of just 28 for single-tailed testing.

Calculating an appropriate sample size to detect score differences is more difficult. The standard deviation of scores is higher (ours were ~7 out of 25 for both bots), but the standard deviation likely varies with team composition. Experienced human teams would likely achieve more consistent scores. However, we may make a few estimates. Holding our alpha and power constant, using a sigma of 7 (Figure 1), it would take 152 samples (fewer than our number of games, but more than our number of subjects) to detect a score difference of 2. Again, assuming samples = subjects, we would have only been able to detect mean score differences of 5 points, which is very large in Hanabi. Thus, depending on how one counts “samples,” it may indeed have taken many more samples than we had to correctly reject the null hypothesis for differences in game score, but all of this is wrapped up in preexisting difficulties of measuring scores and score variance in Hanabi teams. We should point out that our conclusion regarding game score was simply that the data _did not_ support a difference between the bots, and we did not reject the null hypothesis.

Given these considerations for the power analysis, we believe there may be some marginal concern the score statistics, but there should not be much concern for the statistics presented on preference insofar as internal consistency for the paper.

Perhaps the greater concern is that of ecological validity – whether the results generalize to “real-world” human-AI Hanabi teams. We share this concern, but would point out that this is a very difficult question to answer precisely for Hanabi. Given the extremely group-dependent nature of Hanabi, it is not obvious how this would be addressed; increasing sample size does not obviously remove this issue. For example, [25] involved 246 completed surveys and 904 Hanabi games (not necessarily distinct humans) but recruited only from Reddit’s r/boardgames and r/cardgames, which might weaken ecological validity due to potentially similar group conventions among people from a common forum. Our subject pool came from several distinct and disconnected groups of Hanabi players, as well as non-players from a variety of locations (we can add this to the supplement), which may strengthen our ecological validity in comparison, though it is difficult to measure. It that would take much more thorough consideration for demographic and gameplay factors other than just sample size to properly address these concerns.

In summary, concerns about the sample size are mainly pertinent to ecological validity, not statistical validity for our paper. We acknowledge the difficulty, and we will adjust our discussion to more explicitly frame the challenges. Still, we believe that our underlying thesis – that human preferences should be considered alongside raw performance when evaluating AI for human teaming – remains valid and worth bringing to the attention of AI researchers.

### Specific concerns about study design:

* To balance out exposure effects, 15 of the 29 participants played with SB first and the remaining 14 played with OP first. We noted this counterbalancing in line 154. Both groups had a relatively even distribution of Hanabi experience.

* We explicitly considered the learning effect by including game number and block number as factors in our mixed-effects model (lines 182-185), which was not significant for game score (lines 202-204). Our analysis thus indicates that there might be some learning effect, but not a strong one for game score. Statistical tests for post-experiment survey responses sometimes showed significance due to block and/or game number (lines 218-221) and we intend to include our full statistical results in the supplementary materials for transparency.

* Since our statistical model was second order, all pairwise crossed effects (e.g., block and game number) were also tested as factors, and the specific significant results that we focused on in the paper came out despite these being part of the model.

* The between-subjects approach suggested by the reviewer does avoid the learning effect, but as noted, agent order was counterbalanced and learning effects were part of the statistical model. Furthermore, the between-subjects approach makes the direct comparison of the two agents by subjects who experienced both impossible, which was an element we considered important for this study, given the aforementioned logistical constraints.

* Regarding incentive structure: We acknowledge that this is not something we explored exhaustively. We note that through informal, post-experiment discussions, we learned that most participants enlisted in our study out of pure curiosity and “for science,” and none mentioned the monetary reward in their comments about agent preference or their own actions.

* It would be another study entirely to determine the point at which monetary compensation would begin to affect subject behavior, and such a point would likely vary significantly based on subject demographic, which we were not particularly restrictive on for our recruitment.

* Regarding the specific participants brought up by the reviewer: There were 3 participants who obtained a score of 24 with OP, with one of these participants obtaining 24 twice. All three replied at the extreme end of our Likert scale (i.e., 1 or 7) with a preference for SB. 
    * Interestingly, their cumulative scores for OP and SB, respectively, were:
        * Participant A: 57 and 28, having played with OP first 
        * Participant B: 68 and 48, having played with SB first
        * Participant C: 70 and 35, having played with OP first
* For reference, our winner (Participant D) obtained a cumulative score of 68 and 54 with OP and SB, respectively, having played with OP first, and preferred SB with a Likert scale rating of 6. Their self-rated Hanabi experience level was 7, 7, 6, and 7 for Participant A, B, C, and D, respectively. 

* We agree that the game score distribution might lead to unfounded conclusions for distribution properties (e.g., bimodality). That is why we did not place much emphasis on properties of the score distribution, other than to note the lack of statistically-significant differences between agents – a conclusion that should be well-supported for a sample size of 174 games. Furthermore, our power analysis for the Likert scale subjective responses indicated that for the differences that we were looking for, our number of subjects and survey responses per subject is more than sufficient.

* Regarding the measurement of Hanabi experience: We agree that an empirical method of determining Hanabi experience can provide additional perspective, as well as other research regarding Hanabi agents. However, interpreting such a result should be taken with caution since Hanabi is a cooperative game and such a rating can be misleading because individual performance is largely dependent on the team. Furthermore, such a rating may not translate between game modes (e.g., 2-player vs 4-player), where individual idiosyncrasies and team dynamics can vary greatly. We have not found a metric for individual Hanabi skill that would be analogous to similar ratings of Elo (chess), dan (Go), or MMR (StarCraft), and the development of such a metric is something that we are considering.

* The effect sizes were included in some cases (lines 231-233), though we can certainly add all of our results (including demographics and effect sizes) to the supplementary material for completeness and transparency. 

Thank you again for the time and effort for this review.

 Thank you for the review and the suggested changes.

We appreciate that the reviewer considers this work significant and valuable enough to warrant broader and “more ambitious” testing, and we agree that further extensions should be done in more domains, with larger and perhaps specifically-chosen human populations, with more agents, etc. As far as our selection of Hanabi and decision to try to publish at this point, we note that Hanabi is quickly becoming de-facto RL benchmark for training and testing cooperative AI bots – learning and rule-based alike – given its simple cooperative nature. Some of its more important qualities/properties are highlighted in Section 2.1 (Lines 58-61) and additional properties are thoroughly discussed in [2], including many aspects of tight team coupling that do not exist to the same magnitude in most other team games. As such, we believe Hanabi is a sensible starting point for evaluating human-AI teams beyond objective metrics and eliciting the AI research community to think about human-perception factors (e.g., trust and understanding) when developing new methods. 

One of the key points that we bring up is the significant gap in the literature for human evaluations of multi-agent RL systems, even in cases where the putative end goal is, in fact, human-AI teaming or human-like teaming intelligence. Our current work already shows a stark example of the impact that the lack of such consideration can have.

We also discuss extensions into StarCraft, though we wanted to focus on existing agents that had some previous baseline for teaming, which StarCraft RL agents in the literature do not have (to our knowledge). The vast majority of the work on StarCraft is in 1v1 settings with no teaming component, apart from small-scale multi-agent RL settings, where agents only control one unit, and all units are AI-controlled.

We agree with the reviewer that a learning-based Hanabi bot trained with humans or specifically for human teaming could potentially improve their perception from their human-partners. However, we have not encountered such an agent for Hanabi. The Other Play agent was in fact one of the closest cases since it was trained specifically to play with previously-unseen teammates, and achieved state-of-the-art performance in cross-play. As our work suggests, more multi-agent RL work needs to be done that considers the possibility of a _human_ teammate, as opposed to the more theoretical notions of simply _previously unseen_ – but still AI – teammates assumed in nearly all zero-shot and few-shot RL settings. More work in this domain would be needed before such agents can be properly compared against each other, but this is a significant expansion beyond the scope of this paper.

As far as different domains, such as Overcooked, we would like to include such domains in future iterations of our analysis, as suggested by the reviewer. With an understanding of how human perceptions are affected in different environments, future iterations of our analysis could include developing methods for improved human-agent cooperation/understanding without requiring the use of domain-specific information injected into the learning-based agent (as was the case for both agents used in this study). 

Regarding subject commentary – we did include some examples of subject commentary (lines 240-244), though more thorough qualitative analysis did not fit within the allowed space for this paper. Further analysis along those lines, along with more specifics on subject workload, is planned for future work.

Commentary from the participants was obtained at three different phases during the experiments, the first two being required and the third optional, though the majority of subjects did respond to the third as well. During their post-game questionnaire/evaluation (i.e., after each game), participants were asked to describe what they believed the agent’s strategy to be. During their post-experiment questionnaire/evaluation (i.e., after the entire experiment concluded), participants were asked to describe why they chose one agent over the other (without knowing which agent they played with).

For the first two phases, the answer complexity ranged from a single statement/fragment to a few well thought out sentences. For the rule-based (SB) agent, most participants provided similar descriptions, that the agent seemed to take actions which it was immediately sure about (e.g., playing a card immediately after it was given a hint or providing a hint as soon as their human partner had a playable card). This is expected since the rule-based agent will always follow the same conditional logic when playing the game. For the learning-based (OP) agent, descriptions varied widely, with some participants stated they had “no clue” what the agent’s strategy was while others stated that the agent’s actions were initially counter-intuitive (e.g., discarding a card immediately after being hinted it was playable) but were explained in later moves (e.g., OP discarding its playable card because their human partner had the same card). This was also unsurprising, since the learning-based agent does not follow standard game rules or heuristics and is instead a black-box basing its decisions on its policy’s state evaluation.

Further analysis of subject responses is currently underway as follow-up work, but we will consider including raw responses in our supplementary material.

Thank you for the suggestion of including the OpenAI Five cooperative game comments in our discussion. We will try to include that as space allows.

Thank you again for the time and effort put into this review.


This paper conducts a user study in which humans play alongside a rule-based AI bot (SmartBot) and a learning-based AI bot (Other Play) in the cooperative card game Hanabi, and then are asked to rate the bots along several axes. The key finding of interest is that while both bots achieve similar performance in play with humans (about 14 or 15 on average), human players significantly preferred the rule-based AI bot, and this preference increased in strength with additional expertise (i.e. the most expert humans most strongly preferred the rule-based bot.)  I like this paper a lot: much work has been done in multiagent RL, and recently it has started to look into human-AI teaming as well, but so far the focus has primarily been on optimizing the reward obtained, with at best qualitative evaluations of other factors. These other factors seem very important for the deployment of human-AI teaming in the real world, so it is good to see a paper explicitly aim to quantify and analyze these factors. The survey design and analysis seem sensible (though I do not know much about these topics, and might be missing something). The paper is very clear; I especially appreciated the abstract, which laid out exactly what I should expect from the paper (in contrast to most abstracts, which seem to be vague and hiding information about the proposed method).

Ambition
--------

My one critique of the paper is that it could have been more ambitious and had a broader scope. For example, it could have studied multiple domains, to check whether the detected effects generalize to these other domains. As the authors mention, alternatives include StarCraft and Overcooked; the authors could adapt the [existing website](https://humancompatibleai.github.io/overcooked-demo/) for Overcooked in which humans can play against trained agents.

Given the results, it would also be nice to test learning agents that are specifically trained to play with humans. It sounds like participants disliked Other Play because of its alien play style and because it “ignores” information provided by humans. An agent that is trained to play with humans is less likely to fall into such a trap. (That being said, I am not aware of any learning agents that are trained to play with humans on Hanabi, so this may not have been feasible.)

It would also be interesting to learn more about qualitative reasons for participant’s responses -- it seems there were multiple free response questions. Did participants mostly not answer these? If they did, what did they tend to say?

Finally, given that this is (to my knowledge) the first paper studying other aspects besides reward in human-AI teaming for agents based on deep reinforcement learning, it might have been nice to mention other qualitative results that have been informally reported, such as in the cooperative matches for OpenAI Five and Capture the Flag, in which some bots and humans play together on a team. I know that there are some public comments on at least OpenAI Five’s cooperative games. This could capture a broader range of qualitative effects that we may expect to see in human-AI teaming. Yes.

This paper presents a comparative study on the objective and subject performance of a rule-based agent and a RL-learned agent for playing the card game Hanabi with human teammates. The authors selected each of the best-performing existing Hanabi AI from the rule-based methods and learning-based methods, and conducted a user study where human participants are invited to play the game with both types of AI agents and the team performances as well as the human’s subjective feedback are collected. The results show that, despite no significant differences in performance, the rule-based AI agent is preferred by human participants over the learning-based agent.

  Pros: This work is the first to study the human teammate’s subjective evaluation of different types of AI agents for the card game Hanabi. The user study is carefully designed and controlled to offer a comprehensive analysis of the two selected candidate AI agents. The results of the user study are presented in detail with clarity and the claims are supported with statistical analysis. The findings imply that human subjective evaluations are also important metrics for designing AI agents since the task performance is not indicative of subjective evaluations.

Cons: One concern for this paper is that the selection of candidate AI agents for testing is too limited (one from each category) and therefore the results may not generalize across different agents within the same game. Especially that RL training is an unstable process and the paper only tested one policy trained by RL to represent the full range of learning-based agents.
On the other hand, the findings do not seem to be too surprising since the rule-based AI agent is designed with hard-coded human knowledge and therefore are expected to play smoothly with human teammates while the RL-based policy was trained to optimize for zero-short coordination but the test scenarios in this user study is few-shot coordination. Moreover, the RL agent was not explicitly designed to perform well with experienced human teammates who have developed their conventions for communication. 

While I agree with the implication from the findings of this paper that human experiences/evaluations are important metrics on their own, I do not think this directly indicates that subjective metrics should be incorporated into RL training to achieve good human evaluations. Experiments and comparisons with RL agents trained or fine-tuned with human teammates are needed to show whether it is necessary to directly optimize for human subjective evaluations. 
 The authors commented on how this work could influence future research on AI agents and how this can have a larger impact on the users. 

The paper evaluates a human behavior with and perception of Hanabi agents when paired with human players, comparing a top-performing rule-based and learned agent. Results show no significant differences in team performance when comparing the two agents. But humans show a preference for the rule-based agent in terms of subjective ratings of trust, perception of agent skill, understanding of the agent, and preference as a partner. Some results suggest high skill (by self rating) players more strongly distrust the learned agent when compared to novices. Together the results describe the current state of how Hanabi AI agents are perceived as teammates, opening future investigations of ways to bridge this trust/preference gap.
  The paper has several key strengths:
- Topic relevance: The paper investigates an important (at least in near term) question around how human-AI collaboration is experienced by humans.
- Rigor: Overall the empirical analysis is well-described and uses statistical tests where appropriate.

The paper has some modest weaknesses:
- Limited generality: Any human study will have necessarily limited generalization to other problem domains. The value in this work is the insight into the problem and as a good demonstration of appropriate methodology for others to apply to other domains.
- Scale: The study is limited to 29 participants, which is modest for the variety of questions to be addressed. Regardless of statistical power it would be more compelling to see a larger scale replication of this work, particularly including a more balanced sample of participant skill levels.
- Representativeness: Results reporting do not give a sense as to whether the sample should be considered "representative". Or what a representative population might be for something like Hanabi. This makes it hard to assess the whether the outcomes would transfer to the general population of human-AI teaming for Hanabi.
- Missing results: Many negative results are not included in the main text. That is fine, but it would help to include these in the appendices. The narrative of the results may change depending on what other measures of cooperation and task load (TLX) were *not* significantly different.

Below are comments specific to the review criteria prompts.

# Originality

The work extends previous contributions by doing a comparison among agent authoring methods. The task/methods are not new and that is a good thing for this sort of empirical evaluation. 

The topic of human-AI teaming in terms of human perception has not been adequately answered in the past for this domain in specific or more generally. Related work is adequately cited and highlights this point. Related work also appropriately cites relevant materials for the algorithms studied.

# Quality

The empirical analysis uses appropriate statistical tests and shows significant effects. This is clearly a complete piece of work (aside my desire to see a larger population used in the experiment, which is beyond the scope of this paper per se).

I would ask the authors to include the non-significant effects in a supplement or appendix to understand how they fit with the narrative presented. The focus on only positive results is reasonable, but means it is unclear whether contradictory results on related questions are being omitted.

The authors are clear on the ways the work may not generalize due to the test domain (Hanabi), while still addressing what aspects of the work may generalize (expert distrust of AI).

One comment: Figure 1 makes note that "no player achieved a perfect score with OP". This comment is true, but overlooks that seemingly many more players achieved a near-perfect score with OP compared to SB. So it's not clear what point this note is intending to make.


# Clarity

The submission is clearly written and generally well organized.

One clarification (if possible without violating IRB and privacy concerns): how was the experiment population selected? Or what is it's composition? This would help understand the limitations to the generalization of the results to other populations.

Given many NeurIPS readers are not necessarily well-versed in experiment design and analysis it would help to inform the reader more about the choice of statistical tests. For example, why use a one-sample t-test? 

Others could readily reproduce this work, aside from the choice of sample population.

# Significance

The results are important empirically and drive the need for others to extend this work and incorporate this type of evaluation in future efforts with human-AI teams. Others *should* use these results to inform how human-AI agent teams are assessed; the paper reveals a clear gap in prior approaches to evaluation. The paper addresses the potential societal impacts of human-AI teaming should the paper results generalize. The work itself is an evaluation and thus is not necessarily likely to have negative societal impacts by sharing these results.


Meta-review: Reviewers were unanimous that the paper addresses an increasingly important area of research (human-AI collaborations), focuses on an understudied but crucial aspect of the research (i.e., how do humans perceive and assess their AI teammates?), and offers insightful results. While they found the paper well-written overall, reviewers suggested several avenues for further clarification and improvement, in particular, they would like the authors to further elaborate on their choice of the learning agent, the number and representativeness of participants in the study, and their arguments around ecological validity.  Given the authors’ comprehensive response to the reviews, I am confident they can revise their paper accordingly, so I recommend acceptance.

Decision: accept