Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi
Authors: Ho Chit Siu
Jaime Daniel Pena
Edenna Chen
Yutai Zhou
Victor Lopez
Kyle Palko
Kimberlee Chestnut Chang
Ross Emerson Allen
Venue: NeurIPS 2021 Poster (2021)
Keywords: Human-AI Teaming
Deep Learning
Reinforcement Learning

Abstract: Deep reinforcement learning has generated superhuman AI in competitive games such as Go and StarCraft. Can similar learning techniques create a superior AI teammate for human-machine collaborative games? Will humans prefer AI teammates that improve objective team performance or those that improve subjective metrics of trust? In this study, we perform a single-blind evaluation of teams of humans and AI agents in the cooperative card game Hanabi, with both rule-based and learning-based agents. In addition to the game score, used as an objective metric of the human-AI team performance, we also quantify subjective measures of the human's perceived performance, teamwork, interpretability, trust, and overall preference of AI teammate. We find that humans have a clear preference toward a rule-based AI teammate (SmartBot) over a state-of-the-art learning-based AI teammate (Other-Play) across nearly all subjective metrics, and generally view the learning-based agent negatively, despite no statistical difference in the game score. This result has implications for future AI design and reinforcement learning benchmarking, highlighting the need to incorporate subjective metrics of human-AI teaming rather than a singular focus on objective task performance.

Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi
Ho Chit Siu
Jaime D. Peña
Yutai Zhou
Edenna Chen
Victor J. Lopez
Kyle Palko
Kimberlee C. Chang
Ross E. Allen
Abstract
Deep reinforcement learning has generated superhuman Al in competitive games such as Go and StarCraft. Can similar learning techniques create a superior Al teammate for human machine collaborative games? Will humans prefer Al team- mates that improve objective team performance or those that improve subjective metrics of trust? In this study, we perform single- evaluation of teams of humans and AI agents in the cooperative card game Hanabi, with both rule- alsed objective
and learning- based agents. In addition to the game score, used metric of the human-A Al team performance, we also quantify subjective measures the human's 's perceived performance, teamwork, interpretability, trust, and overall preference of AI teammate. We find that humans have clear preference toward rule-basee AI teammate (SmartBot) over state-of-the-art learning- based Al teammate (Other-Play) across nearly all subjective metrics, and generally view the learning- bearning-based agent negatively, despite statistical difference in the game score. This result has implications for future Al design and reinforcement learning benchmarking, highlighting the need to incorporate subjective metrics of human-Al teaming rather than singular focus on objective task performance.
Introduction
Advances in artificial intelligence (AI) have resulted in agents that perform at superhuman levels within domains previously thought to be solely the purview of human intelligence. Such performance has been demonstrated through application of reinforcement learning (RL) environments such as board games [6. 39], arcade games [31], real-time strategy games [44], multiplayer online battle arenas [4], and simulated aerial dogfights [11].
In nearly all cases, these demonstrations of Al superiority are in purely adversarial, one- or two-player games. However, order to achieve wal-world applicability and adoption, AI must be able to demonstrate teaming intelligence, particularly with human teammates [23]. Due to the focus on adversarial games, teaming intelligence has been understudied RL research. Al teammates must also exhibit behavior that engenders an appropriate level of certain human reactions, such as trust, mental workload, and risk perception [26, 37]. Failure do so risks the same kind of misuse, disuse, and abuse that Parasuraman and Riley illustrated with traditional automation systems [37]. These issues are distinct from much of current multi-agent AI work, as AI that is able to team effectively with other AI agents has failed to work effectively with humans [9].
MIIT Lincoln Laboratory, {hochit. siu,, siniiten,,,jjpaaa,,,,a, yutai zhou, chestnut, ross allen)-011 mit.edu 1 MIT Department Electrical Engineering and Computer Science, edennadmit. edu U.S. Air Force Artificial Intelligence Accelerator, fvictor. lopez. 10, kyle.yplllll......e 11sus. af mil DISTRIBUTION STATEMENT Approved public release. Distribution is unlimited.
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

The objective of this paper is to evaluate human-Al teaming in the cooperative, cuperfect-informatne game of Hanabi. We consider not only the objective performance of a human-Al team, but also the subjective human reactions and preferences when working with different kinds of Al teammates. Based on the success of applying deep reinforcement learning to create superhuman AT adversarial games, we hypothesize that similar RI. techniques can render collaborative AI that outperform and are preferred over rule-based agents in human-Al Hanabi teams. Our results show that this hypothesis is not supported given the current state of the art of collaborative RL agents. Human participants show a clear preference toward rule-based AI even though the learning- Al perform no worse and are specifically optimized for teaming with previously unknown partners (e.g. humans) [21]. To the best of our knowledge, this is the first comparative study of objective performance of rule-based and learning- basedd Hanabi AI in human-teame -tamiing experiments, as well as the first quantified study of subjective human preferences toward such AI.
2 Background
2.1 Hanabi
Hanabi is a cooperative card game in which two to five players attempt to stack twenty-five cards into five different fireworks (piles), one for each suit (color) and by ascending rank (number). We consider only the standard, two-player version, where the deck is composed of 50 cards, five suits, each suit having three 1s, two 2s, two 3s, two 4s, and one 5, and each player is dealt hand of five cards. Hanabi's 's difficulty lies in the fact that players can only see their teammate's 's hand and never their own and communication about cards is strictly limited.
Games start with eight hint tokens and three bomb tokens. Each turn, a player may discard a card from their hand, play a card from their hand, or give a hint to teammate. Whenever card is played or discarded, it revealed and a new card is drawn from the deck. A correctly-playe card is discarded and the team loses one bomb.
card is placed
in the appropriate firework; an incorrectly-playy Hinting costs one hint token and allows a player to reveal either all cards of a certain suit or rank in their teammate's hand. Hint tokens are earned back when a card is discarded, or when 5 card is successfully played. The final score is the sum of the top card in each firework, for a maximum of 25 points. The game ends when all fireworks have been completed, the deck is empty (and each player has one additional turn), or the team loses 3 bombs.
Hanabi is purely cooperative game with imperfect information, and limited, codified communication. These properties make it an interesting challenge for teaming since players must consider the reasons for their teammates' actions and any implied information, while avoiding misinterpretations. Bard et al. [2] presents a more complete treatment of Hanabi and its properties as an Al problem.
2.2 Training and Evaluating AI Teams
Hanabi's uniquely collaborative nature has made the subject of several AI challenges in recent years [2, 46). In these challenges AI agents are paired with other teammate agents, and their performance measured based on the paradigms of self-play and cross-play [21].
Self-play (SP) is when an Al plays a game with a copy of itself as the opponent (adversarial games) or teammate (cooperative games like Hanabi). Self-play can be used as a form of RL training [1, 20],
and/or as a form of evaluation [2].
Cross-play (XP) is an evaluation-only paradigm where an agent is teamed with other agents (AI or humans) that were not encountered during training, measuring how well an agent can cooperate with previouslyunnsen teammates. SP-trained agents achieve high SP-scores by developing "secretive conventions" that are not understood by agents not present during training, thus completely failing in the XP setting [21]. Cross-play with humans (human-play) is of particular importance as it measures human-machine teaming (HMT) and is the foundation for the experiments in our paper.
Cross-plan can be evaluated under the zero-shot coordination (ZSC) [21] or ad-hoc teaming [40] settings. ZSC assumes that teams are formed between agents that have no prior knowledge of each other; therefore it is impossible to train an agent to bias towards teammate idiosyncrasies, though the agents may agree to a common training strategy beforehand. On the other hand, ad-hoe teaming attempts to achieve coordination by having agents update their policy while interacting with other

---

Table Survey of Rule-based and Learning Based Hanabi Al Performance in 2- 2-Player Games
Hanabi AI
Original Author
References
Self-Play
Cross-Play
Human-Play
Van den Bergh Self- Reconneeconntion
Van den Bergh 2016
[42, 45. 15]
13.8
10.8
Osawa 2015
[36. 15]
15.9
Piers
Walton-RRvers 2017
[45, 38]
17.3
11.2
Intentional
Eger 2017
[13, 8]
12.6
13.8
15.0
Expectimax
Bouzy 2017
[5, 155
19.0
MirrorSituational
Canaan 2018
[5]
20.1
12,4
RIS-MCTS
Goodman 2019
[15, 8]
20.6
13.3
WhFWThat
Wu 2018
[48, 2, 15]
19.5
FireFlower
D. Wu 2018
[47, 2, 14]
22.7
Implicature AI
Liang 2019
[28, 8]
18.9
15
SmartBot (SB)
O'D O'Dwyer 2019
[33, 2, 15]
23.0
IL- Valuebot
Sarmasi 2021
[38, 33]
18.0
ACHA
Bard 2020
[2, 20]
22.7
1.0
BAD
Foerster 2019
[14, 2, 20)
23.9
SAD
Hu 2019
[20, 21]
24.0
3,0
SAD+AUX
Hu 2019
[20, 21]
24.0
21.1
9.2
SPARTA
Lerer 2020
[27, 8]
24.6
Other-Play (OP)
Hu 2020
[21]
24.1
22.5
15.8
We provide Hanabi AI agents' scores as they reported in existing literature. This gives notional perspective of the current state of the art; however, we note that evaluation conditions have not been standardized throughout previous literature (i.e. number of game seeds for self-play, pool of teammates for cross-play etc.). Therefore some caution is needed when making direct comparisons between reported scores. Please respective literature for details evaluation conditions.
agents, and the the pool of partner agents can be known in advance (though it can be of arbitrarily large size, making such knowledge practically impossible to exploit).
2.3 AI for Hanabi
Table summarizes the most salient published Hanabi Al agents in 2-player games with game score in self-play, cross-play, and human-play. The table is separated into rule-based (top) and learning- -based (bottom) agent types. Rule-based agents have policy composed of a predefined set of rules to follow given any particular game situation, and the rules often derived from human domain knowledge. Learning-base based agents, on the other hand, use statistical learning methods to adjust the parameters of their policy. The mechanism governing what action the policy will choose is learned via the agent's experience, without the need for human domain knowledge.
Most early work focused on rule-based AI [36, 42, 45]. Goodman Competition [46] with a rule-based Monte Carlo Tree Search agent. O'Dwyer's SmartBot (SB) is the highest performing rule- based agent created date, in terms of self-play [33, 2, 15). Due to its state-o satee-o-fae afethh-art SP score, as well as a readily available implementation [33], we select SB as our rule-based agent in the Auman-AI experiments presented in Sections and 4. We found no previous work that evaluated SB in the cross-play or human-play settings.
the 2018 CoG Hanabi
Recently developed learning-based agents have demonstrated breakthrough Hanabi performance. Sarmasi et al. provide a collection of agents trained with imitation learning that nearly match the performance of the rule-based agents from which they were trained [38]. A sequence of publications [14, 20, 27, 21] offered reinforcement learning agents that each advanced the state the art self-play and/or cross-play performance at the time of publication. This culminated with the SPARTA [27] and Other-Play (OP) [21] agents; which, respectively, represent the highest performance to date in self-play and cross-play of any agent type.
The OP agent selected as the learning-basse based agent for study our human-Al Al experiments described in Sections 3 and 4. We choose OP, not only for its state-of-the-art cross-play performance, but also because it uses learning objective function specifically designed optimize for zero-shot coordination settings. Due to this optimization, expect that humans would subjectively prefer and

---

objectively perform better with OP agent teammates over other Al teammates that were not designed for the zero-shot coordination setting, such SmartBot.
handful of works have conducted human-play experiments; however these works are uncommon due the significant time and effort required to conduct such experiments. Both Eger et al. [13] and Liang et al. [28] propose rule-based AI derived from Gricean maxims [17] and achieved an average human-play score of approximately 15.0 over set of experiments that included hundreds of human participants. The learning-based Other-Play agent is arguably the highest performer in terms of human- play with an average score of 15.8, however the experiments were run with a much smaller pool of human participants [21]. No prior works were found that provide a comparative study of rule-based and learning- basedd--bdddd Hanabi Al in human experiments.
2.4 Human-Al Teaming and Metrics
interaction evaluations typically consider two broad categories of metrics: objective performance metrics (raw score, error rates, accuracy, time required, etc.), and subjective team- or human-focused metrics (situation awareness, trust, workload, etc.). Measurement of the former is heavily task-dependent, and are often the primary evaluation metries of Al systems. The latter, however, can also involve metrics for probing systems and teams in a quantitative way, with important implications for how technology is used and adopted [37].
One key metric of teaming is trust, which is defined by Lee and See as "the attitude that an agent will help achieve individual's goals in situation characterized by uncertainty and vulnerability" [26]. Potential difficulties with trust include trust calibration (whether one's trust of an agent is commensurate with its capabilities) and trust resolution (whether the range of situations where human trusts system is commensurate with its range of capabilities).
Closely related to trust are the notions of legibility (being expressive of one's intent) and predictability (matching one's expectations) [12]. measuring teaming, one might directly ask humans about their teammates' intent and their [12] argue that these two ideas trade off in the context of robot motion, but that legibility is the more important factor when working in close collaboration with human. Similar arguments may also be made in the context of Hanabi, where maximum legibility is key to guiding a team's actions in the tightly-coupled, imperfect- imperfect-informaatie information scenario.
expectations, particularly in relation to shared goals. Dragan et al.
Hoffman [19] proposes set of subjective and objective methods (overlapping with the aforemen- tioned ideas) to measure fluency in human-robot interaction, defined as the "coordinated meshing of joint activities between members of well-synchronized team." " Fluency encompasses subjective elements such as teamwork, trust, and positive perception; and objective measures of timing and concurrency.
Much of the contemporary work on learning based agents has not evaluated such teaming metrics, due to their focus on single-player [3] or adversarial games [39, 43]. In this work, we consider the use of both objective performance and subjective teaming based measurement of human-Al pairs in the context of Hanabi. We aim to provide a more thorough analysis of the human-AL -AI teaming data than previously-dic discusssd Hanabi HMT studies [13, 28, 21]. Rather than proposing new Al with better raw performance, we are interested in the subjective aspects of human-Al teaming with state-of-the-ard rule- and learning-based AI, and how they might drive future AI development.
3 Methods
3.1 Human-Al Teaming Experiment
Experiments consisted of two-player Hanabi games played by teams of one human participant and one Al agent. Experiments aimed to measure the objective team performance and subjective human reactions to different types of AI. The AI agents used in experiments were the Other-Play RL agent [21] (specifically the OP+ agent, hereafter referred to as the OP bot) and the "SmartBot" agent [33] (hereafter referred to as the SB bot), both of which have MIT Licenses. These agents were chosen because they were the top-performing learning-basse and rule-based Hanabi AI, respectively at the time of the experiment.

---

Participants were first introduced to the experiment and the rules of Hanabi as defined in Section 2.1. The experiment followed with brief familiarization game so participants could acquaint themselves with the game interface. Then, each participant played two sets (blocks) of three games, with each set using a different Al teammate. The participant was not informed which AT teammate played each block of games and the order of the Al teammates was counterbalanced over the course of the study. Participants answered Likert scale surveys after each game (Table 2, left), a NASA Task Load Index (TLX) survey after each block [18]. and a Likert scale survey after both blocks that directly compared their experience with both agents (Table 2. right), Likert scale survey questions were largely derived from compilation of similar questions in Hoffman et al. [19]. Unless otherwise noted, Likert scales were arranged with 1 corresponding to "strongly disagree" "
and 7 corresponding to "strongly agree." "
A total of 29 adult participants completed an experiment and each provided written, informed consent. The protocol was approved by the MIT Committee on the Use of Humans as Experimental Subjects (protocol E-2520) and the United States Department of Defense Human Research Protection Office (protocol M0T1.20200003). Participants received $10 USD gift card the end of their experiment, and the highest-scoring participant received an additional $50 gift card. Each experiment took approximately 1.5 to 2.5 hours. The total amount spent on participant payment was $350, with one participant being paid twice due to technical difficulties ending their initial session early. Experiments were conducted virtually, with all interactions occurring through eddo-cconeerecce online surveys, and the Hanabi game interface, adapted from [27]. No personallyiddentffaaln information was collected, and no significant risk to participants was expected.
Liang et al. [28] conducted human experiments with subjective survey questions; however the only significant result reported was that humans were more likely to mistake Implicature Al as another human. Although Hu et al. [21] conducted some human experiments with the OP bot and a self-play RL bot, their experiments only focused on game score, involved only one game with each agent, and did not consider participant expertise in Hanabi.
Table 2: Post-Game (Left) and Post-Experiment (Right) Evaluation Statements
7-point Likert Scale statement
G1
am playing well.
G2
The agent is playing poorly.
G3
The team is playing well.
7-point Likert Scale statement
G4
This game went well.
E1
Which agent did you prefer playing with? Which agent did you trust more? [trust] Which agent did you understand more? [legibility]
G5
The agent and have good teamwork.
E2
[fuency]
E3
G6
The agent is contributing to the success of the team.
E4
Which agent understood you better? [legibility] Which agent was the better Hanabi player? Which agent was more reliable? [predictabilit]] Which agent had better understanding of the game on average?
G7
understand the agent's 's intentions. |legibility]
E5
E6
G8
The agent does not understand my intentions. [legibility]
E7
G9
I feel comfortable playing with this agent.
E8
Which agent caused you have greater mental workload? [mental workload]
G10
do not trust the agent. [trust] The agent is not a reliable teammate. [predictability]
G11
G12
not confident in my gameplay.
Left: Statement order was randomized when presented to participants. Right: Rating is "strongly prefer first agent" and rating is "strongly prefer second agent." Items associated with particular human factors constructs have the construct name in brackets (not shown to participants).
3.2 Statistical Analysis
We evaluated both objective and subjective metrics of human-Al teaming, with the overall hypothesis that the Other-Play RL agent (OP) is preferred over- and would outperform the rule-based SmartBot (SB) agent. As an objective measure, we consider the team score during each game. For subjective measures, we consider outcomes related to perceived performance, teamwork, legibility, comfort, trust, and workload as measured by the Likert surveys. Linear mixed-effects regression models were used for both objective and subjective variables, following the recommendation of Norman [32].
5

---

Due to space constraints, we do not present the results of the TLX here, but include one question on mental workload from the post-experiment survey (Table 2, E8).
Both objective and subjective omnibus tests for post-game results used second-order mixed-effects models with fixed factors of (1) Al agent, (2) self-rated Hanabi experience, (3) block (the first second set of three games), and (4) game number (first, second, or third game within a block), and a random factor of participant number. Al agent and participant number were considered categorical variables. Post-experiment surveys were evaluated separately with one-sample (-tests, and a Holm -onffrorrro multiple comparison correction, since these tests were looking for differences from a neutral value on the Likert scale, and there were multiple hypotheses being tested. Additionally, correlations between self-rated experience, and post-game responses were checked against game score.
4 Results
We present the results of the teaming experiment. Our results indicate that despite no significant difference in objective performance between teaming with the two agents (Section 4.1), human sentiment results (Sections 4.2 and 4.3) show a clear preference toward rule-based agent over learning-based agent.
For context, we have colored most of the plots to show the self-rated Hanabi experience level of the participant with which the data associated, and we note that our participant pool is skewed towards -eter--sel-------eee-eeee the demographic survey statement "I am experienced in Hanabi" (see supplemental materials for demographic survey results).
players. Self-rated Hanabi experience was based on ratings of
4.1 Game Score
The mixed-effects regression did not support score differences due to agent type (1(158) 0.374), 0.709) or self-rated Hanabi experience (I(158) 1.228, p = 0.221) (Figure 1). This means that the data do not support OP under-performing relative to SB.
The dependent variable of game score only has significant effect of block (b 5.282, t(158) 2. 2.636, p = 0.009), and a hear-significant effect of game number (b 2.957, t f(158) = 1.887, p = 0.061). The positive regression slope in both cases indicates that this is likely learning effect over time; i.e. the human participant adapts his/her play over time.
OP Games
5B Games
16
25
Mezn standerd
deviation: 14.39
Mezn standard deviatio: 14.07
14
Median
Median 14.0
12
Dersity
10
10
20
25
0
35
20
Scores
Hanabi Experience
Figure 1: Game scores by agent type (left) and self-rated player experience (right). No significant differences were found when teaming with either agent, and correlation with self-rated experience was significant only for SmartBot games.
Since score is the primary performance metric of interest in Hanabi, and acts as the reward function for RL agents in this domain, we examined a few additional correlations with score. Note that since these are bivariate correlations, they not complete the aforementioned statistical regression.
Self-rated Hanabi experience and score had a small, but significant positive correlation when pooling both agents' games (p 0.0053, r 0.214). Correlation remains significant for the subset of SB games (p- = 0.0023, r 0.247), but not significant for OP games (p = 0.0867, r 0.18)), indicating that for this bivariate analysis, experience only correlates with score for SB, not with OP.

---

A small, but sigmificant positive correlation was found between subjective team performance (G3, G4) and score (p 0.0003, r 0.275 and p 0.0002, 0.280). However, subjective ratings of self- and agent-performance (G1, G2) were not significantly correlated to score.
When participants were ordered by the absolute differences in their total scores with each agent, the ten subjects with the greatest differences (max/min point differences of 57 and 17), six of the had better performance with OP, but when considering only the top five subjects in terms of score difference (max/min differences of 57 and 28), only the participant with the greatest difference performed better with SB, and all four others performed better with OP. full plot of scores by subject and agent type is in the supplemental materials.
4.2 Post-Game Sentiments
Significant effects in post-game subjective measures are summarized in Table Although the statistical model considers factors of block and game, which were sometimes significant, we do report these main effects or their interactions in the table, in order focus on the independent variables of greater interest.
In all cases where the interaction of agent type and Hanabi experience found to be significant, more experienced Hanabi players rated the Other-Play agent much more negatively than the SmartBot agent, while novices rated the two similarly (Figure 2). However, there was significant difference between novice and expert ratings of the SmartBot agent. Cases where neither agent nor experience significant factors are shown in Figure 3.
Table 3: Post-Game Sentiment Statistics (Statistically-Signiie Factors Only)
Dependent Variable
Factor
P
I am playing well (G1)
experience
2.698
0.008
The agent playing poorly (G2)
experience
1.44
< 0.0001
The team is playing well (G3)
agent:experience
-2.082
0.039
The agent and I have good teamwork (GS)
agent
2.578
0.011
agenteexpriiecce
-3.021
0.003
T understand the agent's intentions (G7)
agentxexperience
-2.73
0.021
The agent does not understand my intentions (G8)
experience
2.098
0.037
agent:experience
-3.166
0.002
I feel comfortable playing with this agent (G9)
agent
3.302
0.001
agentrexperience
-3.561
< 0.0001
The agent is not reliable teammate (G11)
experience
3.159
0.002
Degrees of freedom for t-tests 164 in all cases. Agent is agent type (SmartBot or Other- and experience self-rated Hanabi experience. Agent:experience is the interaction effect agent and experience
To specifically examine the effect of self-rated player experience, we performed post-hoc pairwise comparisons cases where experience was significant. Participants were pooled into "novice" (n = 10, self-rated experience of < 5) and "expert" (n = 19, self-rated experience of > 5) groups, and comparisons were made on ratings of each agent in the cases where an interaction effect was significant (G3, G5, G7, G8, G9). The groups did not rate SB significantly differently, but experts always rated OP worse than novices did. The difference in G3 "the team is playing well" (L(85) 3.551, < 0.001, effect size d = 0.752) was not stark the others (1(85) 5.068 to 5.855, p < 0.0001, [d] > 1.0), but all were still clearly significant, and all but one case had large effect sizes.
4.3 Post-Experiment Sentiments
Participants' direct comparisons of the agents (Table 2) are shown in Figure 4. The statements were presented to participants as comparing the "first" and "second" agents (without reference to agent type, which was not disclosed to participants). For ease of interpretation, we matched the responses to the agent type, and flipped the scale as appropriate. All positively-framed questions (E1 to E8) showed a strong preference SB over OP (corrected p < 0.05), while mental workload (E8) was split.
7

---

G3: The team 8 playing well (OP)
G3: The team playing well (5B)
G5: The egent and have good teamwork (OP)
G5: The egent and have good teamwork (58)
25
25
25
25
20
20
20
15
15
15
10
10
understand agent's
understand agent's
The agent
agent does understand my
(OP)
25
25
25
retf-rated Hanabi Experience
Frequencies
15
10
5-
5-
G9: feel comfortable playing with this agant (OP)
G9: feel comfortable
playing with this agent (SB)
20
15
15
10
10
5
Rating
Figure 2: Participant rating for post-game questions by self-rated Hanabi experience (SB vs OP), where statistically significant differences related to factors of agent and/or experience were found (Table 3). The scale ranges from 1-7, corresponding to "strongly disagree" to "strongly agree".
agent not Seitrated Experience
playing well
agent playing poorly
40-
Frequencies
36-
10
Rating
Figure Participant rating for post-game questions by self-rated Hanabi experience where statistically significant differences related to factors of agent and/or experience were found. The scale ranges from 1-7, corresponding to "strongly disagree" to "strongly agree".
We note there were three participants who obtained a score of 24 with OP, with one of these participants obtaining 24 twice (no player achieved a 25 with OP). All three replied at the extreme end of our Likert scale (i.e., or 7) for question E1 with preference for SB. Interestingly, their cumulative scores for OP and SB, respectively, were: Participant 6 (played with OP first, self-rated experience of 7): 57 and 28; Participant 19 (SB first, experience 7): 68 and 48; Participant 20 (OP first, experience 6): 70 and 35. The participant with the highest cumulative score (Participant 2, OP first, experience 7) had cumulative scores of 68 (OP) and 54 (SB) and preferred SB with Likert rating of All participant scores are provided in the Appendix (Figure 7).
Participant commentary indicated that low mental workload when working with OP was often caused by frustration with the agent and giving up on teaming with For example, after OP bot failed to act several hints from the human ("I gave him information and he just throws it away") participant commented that " At this point, don't It know what the point is, regarding working with the agent.
8

---

E1: Which agent did you prefer playing with?
E2: Which agent did you trust more?
E3: Which agent did you understand more?
E4: Which agent understood you better?
12
10-
Self-ratd Hamaai Experience
Frequencles
E7: Which agent hed better understanding game average?
EB: Which agent caused you have a groater mental workload?
ES: Which agent was Hanaloi player?
EG: Which agent was more
12
10.
Preference
Figure 4: Post-experiment questions. All showed statistically significant preference for the rule- based SmartBot (p < 0.05), except the question on workload (E8). The scale ranges from ("strongly prefer OP") 7 ("strongly prefer SB").
Discussion
5.1 Other-Play Agent Performance
Other-Play (OP) training is designed to avoid the creation of "secretive" conventions that can result from self-play training. However, even though OP agents can pair well with previously-unseen teammates, there are still assumptions placed on the types of teammates OP agents will encounter. Notably, OP assumes that teammates are also optimized for zero-shot coordination [21]. In contrast, human Hanabi players typically do not learn with this assumption. Pre- game convention-setiing and post-gand game reviews are common practices for human Hanabi players, making human learning more akin few-shot coordination. Furthermore, Hu et al. note that, due practical implementation details, OP does not always avoid secretive conventions. For example, some OP agents use color hints to indicate discarding of the 5th card [21]. Such conventions not legible human teammate without extensive observation.
5.2 Implications for Human-Al Teaming
The results shown here have important implications for the development of (earning-basede human- teaming agents. Regardless of the objective performance of Al systems, "teaming intelligence" is ultimately required for real-world deployment. The difference between rule-based and learning- based systems is particularly poignant here, where humans strongly favored rule-based agent over an RL agent many subjective metrics (and never significantly favored the RL agent) despite achieving objective scores that were not significantly different across the two agent types. Caution should be exercised when generalizing our results beyond Hanabi, though there is some indication of cross-domain consistency. The negative post-experiment sentiments expressed towards OP by some participants are similar to those expressed by participants during the DotA2 OpenaAl Five games where RL players exhibited playstyles that were not immediately understood by human players, though they were successful [34, 35].
An additional consideration is the user base of future learning learning-based systems. We note that between post-gams and post-experiment surveys, nearly statements/questions associated with human factors constructs (except workload) had differences related to the agent and/or the participant's experience level. Experienced players rated the OP agent particularly negatively (Figure 2, Table 3), despite similarity in scores. As domain experts are likely to be the first users of Al technology in operational settings, their perception of Al is key factor to its adoption. Part of the experts' negativity toward OP may come from the way human experts make decisions in uncertain situations.

---

Klein's 's recognition eeoogition-primed decision massing model [25] indicates that experts typically rely on base of knowledge to recognize "prototypical" situations and alleviate much of the mental burden of decision-makiin making. Working with an unpredictable and illegible teammate breaks experts' ability to rely on much of their knowledge base. This notion also supported by the scorefexperience correlations (Figure 1) which were only significant for SB games.
This work indicates a need for RL methods that produce legible and predictable policies. Methods in this direction generally involve either building inherently transparent models, or procedures for post- hoc analysis. The former includes human languagggguggdrrr driven policy specification [41], or explanation learning through construction of causal diagrams [30]. The latter includes saliency maps for visual domains, and action preference explanation through reward decomposition [24].
5.3 Limitations and Future Work
The tight coupling of coordination and performance. and the codification of communication, make Hanabi unique test bed. Games without these properties may elicit different responses from human-Al teams, so generalization of this experiment beyond Hanabi must be treated carefully.
well as the lack of readily available Due to the manual and dime-intensive experimental procedure implementations for many pre-existing Hanabi agents such as Intentional [13] and Implicature AI [28]-we we were limited in our pool of participants and agents. Future work could increase both by using an online, operator-free game-play and survey platform.
While we made an effort to recruit participant with diverse level of experience, the participant pool was still skewed towards those with higher self-rating (though is likely less biased than the pool from [28], who recruited exclusively from online gaming communities). Since Hanabi performance heavily dependent on the team, it is difficult to obtain an objective single-player skill rating. Still, the relatively tight clustering of many (self-rated) experts' responses to OP in particular is notable.
Game outcomes and the feasibility of perfect games (i.e. score 25) are dependent on the starting deck. We did not control for deck configuration to avoid the issue encountered in Hu et al. [21], who explored relatively narrow portion of the game space due to using only two deck seeds.
Interesting extensions to this Hanabi experiment include varying participants' knowledge of their teammates, increasing the number of games played, and adding additional metrics of teaming, such as positive listening and positive signalling [22, 29]. Modifications to the agents may include adding logic filters to prevent learning-based agents from making "obvious" mistakes, modifying the reward function to more heavily penalize "trust-breaking" moves. Beyond the domain of Hanabi, other candidates for Auman-Al teaming experiments include StarCraft (which date has only shown adversarial RL agent performance with humans [44]), Overcooked [9], and mixed cooperative- competitive environments like Bridge [49] and Diplomacy [16].
Conclusion
This study measured the game performance and human reactions in mixed human-Al teams in the cooperative card game Hanabi, comparing outcomes when humans were paired with a rule-based agent and when paired with reinforcemeentllearing bearen--beanii based "Other-Plas Play" agent designed to maximize zero-shot coordination. Despite achieving similar scores between these teams, human players strongly preferred working with the rule-based agent, and view the Other-Play agent quite negatively, citing their bilateral understanding, trust, comfort, and perceived performance as reasons. The ability of Al agents team with humans is important determinant of whether they be deployed in many real-world situations. These results show that even tfte-off-the-rn RL agents largely fail to convince humans that they are good teammates, and suggest that human perception of AI needs greater consideration in future Al design and development if it is to achieve real-world adoption.
10

---

Acknowledgments and Disclosure of Funding
DISTRIBUTION STATEMENT A. Approved for public release. Distribution unlimited. This ma- terial is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommen- dations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering. @ 2021 Massachusetts Institute of Technology. Delivered the U.S. Government with Unlimited Rights, defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.
The authors would like to thank our experiment participants for their time. We thank Hengyuan Hu for providing the Other-Play model. We thank Peter Morales for his guidance during the early phase of this work.
References
[1] Trapit Bansal, Jakub Pachocki, Szymon Sidor, lya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. arXiv preprint 171.0377,, 2017.
[2] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
[3] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-22 279, 2013.
[4] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint
2019.
[5] Bruno Bouzy. Playing hanabi near-optimally. In Advances in Computer Games, pages 51-62. Springer, 2017.
[6] Murray Campbell, A Joseph Hoane Jr. and Feng-hsiung Hsu. Deep blue. Artificial intelligence, 134(1-2):57-63) 83, 2002.
[7] Rodrigo Canaan, Haotian Shen, Ruben Torrado, Julian Togelius, Andy Nealen, and Stefan Menzel. Evolving agents for the hanabi 2018 cig competition. In 2018 IEEE Conference on Computational Intelligence and Games (CIG), pages 1-8. IEEE, 2018.
[8] Rodrigo Canaan, Xianbo Gao, Julian Togelius, Andy Nealen, and Stefan Menzel. Generating and adapting to diverse ad-hoc cooperation agents in hanabi. arXiv preprint 3711, 2020.
[9] Micah Carroll, Rohin Shah, Mark K Ho, Thomas L. Griffiths, Sanjit A Seshia, Pieter Abbeel, and Anea Dragan. On the utility of learning about humans for human-ai coordination. arXiv preprint 190.05579, 2019.
[10] Jacob Cohen. Statistical power analysis for the behavioral sciences. Academic Press, New York, 1977. ISBN 0-12-179060-6.
[11] DARPA. AlphaDogfight Trials Foreshadow Future of Human-Machine Symbiosis. https: //www. darpa www.dda.......e.eeeeeeeeeeeeeeee http://ewsueeeeetseeeeeeeeeeeeee 08000008-220000 26, 2020.
[12] Anca D Dragan, Kenton CT Lee, and Siddhartha S Srinivasa. Legibility and predictability of robot motion. 2013 8th ACM/FEE International Conference on Human Roobtt Interaction (HRI), pages 301-308. IEEE, 2013.
11

---

[13] Markus Eger, Chris Martens, and Marcela Alfaro Córdoba. An intentional ai for hanabi. In 2017 IEEE Conference on Computational Intelligence and Games (CIG), pages 68-75. IEEE, 2017.
[14] Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi agent reinforcement learning. In International Conference Machine Learning, pages 1942-1951. PMLR, 2019.
[15] James Goodman. Re-determinizing information set monte carlo tree search in hanabi. arXiv preprint 1rXiv: 2019.
[16] Jonathan Gray, Adam Lerer, Anton Bakhtin, and Noam Brown. Human-level performance in no-press diplomacy via equilibrium search. arXiv preprint (rxiv:20000029233 2020.
[17] Herbert P Grice. Logic and conversation. In Speech acts, pages 41-58. Brill, 1975.
[18] Sandra Hart and Lowell E Staveland. Development of nasa-tix (task load index): Results of empirical and theoretical research. In Advances in psychology, volume 52, pages 139-183. Elsevier, 1988.
[19] Guy Hoffman. Evaluating fluency in huban-robot collaboration. IEEE Transactions on Human-Machine Systems, 49(3):209-216 2019.
[20] Hengyuan Hu and Jakob N Foerster. Simplified action decoder for deep multi-agent reinforce- ment learning. arXiv preprint 1922v009 2019.
[21] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-plas paatt-pla"" for zero-shot coordination. In International Conference on Machine Learning, pages 4399 4410. PMLR, 2020.
[22] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pages 3040-3049. PMLR, 2019.
[23] Matthew Johnson and Alonso Vera. No ai is an island: the case for teaming intelligence. AI Magazine, (0(())16-28) 2019.
[24] Zoe Juozapaitis, Anurag Koul, Alan Fern, M. Erwig, and Finale Doshi-Veles Explainable reinforcement learning via reward decomposition. 2019.
[25] Gary A Klein. A sccognition-primet decision (rpd) model of rapid decision making. Decision making in action: Models and methods, 5(4):188-147 1993.
[26] John Lee and Katrina See. Trust in automation: Designing for appropriate reliance. Human f 46(1):50-88) 50, 2004.
[27] Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7187-7194, 2020.
[28] Claire Liang, Julia Proft, Erik Andersen, and Ross A Knepper. Implicit communication of actionable information human-ai teams. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1-13, 2019.
[29] Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin. On the pitfalls of measuring emergent communication. arXiv preprint 4rX0::1903.05168, 2019.
[30] Prashan Madumal, Miller, Sonenberg, and Vetere. Explainable reinforcement learning through a causal lens. In AAAI, 2020.
[31] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1022v.331..5002, 2013.
12

---

[32] Geoff Norman. Likert scales, levels of measurement and the "laws" of statistics, Advances in health sciences education, 15(5):625-60 632, 2010.
[33] Arthur O'Dwyer. quuxplusonechaanabi framework for writing bots that play hanabi.
http://hhhtbbooommmuuuuuuuuuua/a 2019.
[34] OpenAL. Openai five defeats dota 2 world champions, Apr 2019. URL https:/t:///ppaaa.ccoo blog/open hogg/ppeaai--o citimna--iiv--eee
[35] OpenAL. Openai five, Dec 2019. URL UtC.htttp:///pppaat.com///////
[36] Hirotaka Osawa. Solving hanabi: Estimating hands by opponent's actions in cooperative game with incomplete information. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
[37] Raja Parasuraman and Victor Riley. Humans and automation: Use, misuse, disuse, abuse. Human factors, 39(2):330-255 1997.
[38] Aron Sarmasi, Timothy Zhang. Chu-Hung Cheng, Huyen Pham, Xuanchen Zhou, Duong Nguyen, Soumil Shekdar, and Joshua McCoy. Hoad: The hanabi open agent dataset. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 1646-16660 1648, 2021.
[39] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1 111/64499:1110-1111 140-11444 2018
[40] P. Stone, G. Kaminka, S. Kraus, and J. Rosenschein. Ad hoc autonomous agent teams: Collabo- ration without pre-coordination. In AAAI, 2010.
[41] Pradyumna Tambwekar, Andrew Silva, Nakul Gopalan, and Matthew Gombolay. Inter- pretable policy specification and synthesis through natural language and rl. arXiv preprini arXiv:2101.07/ 101.071110..... 2021.
[42] Mark JH van den Bergh, Anne Hommelberg, Walter A Kosters, and Flora M Spieksma. Aspects of the cooperative card game hanabi. In Benelux Conference on Artificial Intelligence, pages 93-105. Springer, 2016.
[43] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint 17Xiv:1708.04788. 2017.
[44] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in stareraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-3 335-354, 2019.
[45] Joseph Walton-Rivers, Piers R Williams, Richard Bartle, Diego Perez-Liebana, and Simon M Lu- cas. Evaluating and modelling hanabi- banabi-playing agents. In 2017 IEEE Congress on Evolutionary Computation (CEC), pages 1382-12 1389. IEEE, 2017.
[46] Joseph Walton-Rivers, Piers R Williams, and Richard Bartle. The 2018 hanabi competition. In 2019 IEEE Conference Games (CoG), pages 1-8. IEEE, 2019.
[47] D. Wu. lighttectorr/iieeloower https///ttta...ommiiiivveooorrrt 2018.
rewrite of hanabi-bot in scala.
[48] J. Wu.
wuthefiaashaa/hanaaii.ss Thahhaaabissa 2018.
Hanabi
simulation
in rust.
[49] Chih-Kuan Ych, Cheng-Yu Yu Hsich, and Hsuan- Assan-Tiin Lin. Automatic bridge bidding using deep reinforcement learning. IEEE Transactions on Games, 10:365-377, 2018.
13


Reviews:
[{"review_id": "OQhwV-6wL4", "reviewer": "Reviewer_PtgZ", "summary": "This paper conducts a user study in which humans play alongside a rule-based AI bot (SmartBot) and a learning-based AI bot (Other Play) in the cooperative card game Hanabi, and then are asked to rate the bots along several axes. The key finding of interest is that while both bots achieve similar performance in play with humans (about 14 or 15 on average), human players significantly preferred the rule-based AI bot, and this preference increased in strength with additional expertise (i.e. the most expert humans most strongly preferred the rule-based bot.)", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like this paper a lot: much work has been done in multiagent RL, and recently it has started to look into human-AI teaming as well, but so far the focus has primarily been on optimizing the reward obtained, with at best qualitative evaluations of other factors. These other factors seem very important for the deployment of human-AI teaming in the real world, so it is good to see a paper explicitly aim to quantify and analyze these factors. The survey design and analysis seem sensible (though I do not know much about these topics, and might be missing something). The paper is very clear; I especially appreciated the abstract, which laid out exactly what I should expect from the paper (in contrast to most abstracts, which seem to be vague and hiding information about the proposed method).\n\nAmbition\n--------\n\nMy one critique of the paper is that it could have been more ambitious and had a broader scope. For example, it could have studied multiple domains, to check whether the detected effects generalize to these other domains. As the authors mention, alternatives include StarCraft and Overcooked; the authors could adapt the [existing website](https://humancompatibleai.github.io/overcooked-demo/) for Overcooked in which humans can play against trained agents.\n\nGiven the results, it would also be nice to test learning agents that are specifically trained to play with humans. It sounds like participants disliked Other Play because of its alien play style and because it “ignores” information provided by humans. An agent that is trained to play with humans is less likely to fall into such a trap. (That being said, I am not aware of any learning agents that are trained to play with humans on Hanabi, so this may not have been feasible.)\n\nIt would also be interesting to learn more about qualitative reasons for participant’s responses -- it seems there were multiple free response questions. Did participants mostly not answer these? If they did, what did they tend to say?\n\nFinally, given that this is (to my knowledge) the first paper studying other aspects besides reward in human-AI teaming for agents based on deep reinforcement learning, it might have been nice to mention other qualitative results that have been informally reported, such as in the cooperative matches for OpenAI Five and Capture the Flag, in which some bots and humans play together on a team. I know that there are some public comments on at least OpenAI Five’s cooperative games. This could capture a broader range of qualitative effects that we may expect to see in human-AI teaming.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "OKQa1dxlgYh", "reviewer": "Reviewer_WHgS", "summary": "This paper presents the results of a user study that enlisted 29 participants to play the cooperative game of Hanabi. The results suggest that humans prefer to play with rule-based agents than with reinforcement learning (RL) agents despite the average game score being nearly identical with either type of agent.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and deals with an important problem, which is to understand how humans perceive their interaction with RL-based agents in cooperative settings. The game of Hanabi is a great choice for studying this problem as it is one of the standard benchmark problems for evaluating RL algorithms in cooperative settings. The choice of the agents used in the study is also good: Smart Bot (SB) and Other-Play (OP), which are the current state-of-the-art rule-based and RL-based algorithms for playing Hanabi. \n\nThe results of the study show an interesting trend: objective metric of number of points scored is similar for both agents, but subjective metrics such as whether the agent \"understands\" the human player's intention mostly favor the rule-based agent. I have played Hanabi with older RL agents such as policies trained with DQN Rainbow and I know that it can be frustrating to play with RL agents. However, I have not played with agents trained with the newer OP algorithm. When I read the OP paper I had thought that it would be fun to play with OP agents because humans were shown to score much higher with OP agents (while I can score around 5 or 6 with DQN Rainbow, the OP paper showed that experienced players can score ~15). This submission is telling me that \"hey, you might score ~15 with OP agents, but you will still prefer to play the SB agent.\" I think this is an interesting result that needs to be investigate more carefully. The research community needs to get to the bottom of this result and understand what is going on. \n\nAfter my initial excitement with the result, I started to be concerned with its validity for the authors enlisted a small number of people to perform their study. A careful analysis show that the results might be too preliminary and we would need a larger study to confirm the trend observed in the current submission (see details below). \n\nParticipants of the study played 3 games with each agent (either SB or OP and then with the other agent). The results suggest (according to the paper) that there is a learning effect, which is expected: participants score higher as they play the second block of games. The experimental design deals with learning effects by controlling the number of people that gets to play each agent first. It would be helpful to know exactly how many of the 29 participants played OP first and how many played SB first. Small variations in the distribution could affect the results due to the strong learning effect of the design. A cleaner approach is to use a between-subjects design, where each participant plays either SB or OP. Then the learning effect won't be carried across different treatments of the study. \n\nI am also concerned with the incentive design of the study. Each participant received a 10 USD gift card and the highest-scoring participant received an additional 50 USD gift card. I wonder if a different compensation model (e.g., participants receive more if score higher in the game, independently of the other participants' scores) would change the results. Figure 1 shows that the distributions of scores of OP and SB are different. In particular, there is a peak for the OP results for the score of 24 (maximum is 25). I wonder how people who played those matches ranked OP compared to SB. Would the participants who scored higher with OP than with SB still prefer to play with SB -- even if OP gives them a higher monetary reward? \n\nA careful analysis of the distributions shown in Figure 1 also shows that the number of participants enlisted in the study is probably too small to support significant conclusions. For example, no participant scored 15 with SB and I don't see why 15 would be too different from 14 or 16. The lack of scores 15 makes the distribution look bimodal, but I suspect this is just the effect of a small number of participants. \n\nThe distinction between experienced and inexperienced players seems to play an important role in these user studies (e.g., the one presented in the OP paper). However, no one has proposed an effective metric of experience for Hanabi that allows others to reproduce the results. There could even be a cultural effect of how people regard their own experience in the game. Perhaps one can formulate a Hanabi test to allow for a concrete evaluation of one's experience?\n\nIn short, the results presented in this paper are interesting, but perhaps too preliminary for publication. I suggest that the study is repeated with a between-subject design, a more lenient reward function to make people prefer higher scores, and with many more participants. It would also be helpful to report demographic data about the participants and also the effect size of the comparisons.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation of the paper is the pool of participants used in the study. The number of participants enlisted is very small and we have no information about their diversity. Which part of the world's population is represented in the pool of participants? The paper doesn't present any demographic data and this is certainly a limitation. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "DP-pLuI8tG", "reviewer": "Reviewer_8ptp", "summary": "The paper evaluates a human behavior with and perception of Hanabi agents when paired with human players, comparing a top-performing rule-based and learned agent. Results show no significant differences in team performance when comparing the two agents. But humans show a preference for the rule-based agent in terms of subjective ratings of trust, perception of agent skill, understanding of the agent, and preference as a partner. Some results suggest high skill (by self rating) players more strongly distrust the learned agent when compared to novices. Together the results describe the current state of how Hanabi AI agents are perceived as teammates, opening future investigations of ways to bridge this trust/preference gap.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has several key strengths:\n- Topic relevance: The paper investigates an important (at least in near term) question around how human-AI collaboration is experienced by humans.\n- Rigor: Overall the empirical analysis is well-described and uses statistical tests where appropriate.\n\nThe paper has some modest weaknesses:\n- Limited generality: Any human study will have necessarily limited generalization to other problem domains. The value in this work is the insight into the problem and as a good demonstration of appropriate methodology for others to apply to other domains.\n- Scale: The study is limited to 29 participants, which is modest for the variety of questions to be addressed. Regardless of statistical power it would be more compelling to see a larger scale replication of this work, particularly including a more balanced sample of participant skill levels.\n- Representativeness: Results reporting do not give a sense as to whether the sample should be considered \"representative\". Or what a representative population might be for something like Hanabi. This makes it hard to assess the whether the outcomes would transfer to the general population of human-AI teaming for Hanabi.\n- Missing results: Many negative results are not included in the main text. That is fine, but it would help to include these in the appendices. The narrative of the results may change depending on what other measures of cooperation and task load (TLX) were *not* significantly different.\n\nBelow are comments specific to the review criteria prompts.\n\n# Originality\n\nThe work extends previous contributions by doing a comparison among agent authoring methods. The task/methods are not new and that is a good thing for this sort of empirical evaluation. \n\nThe topic of human-AI teaming in terms of human perception has not been adequately answered in the past for this domain in specific or more generally. Related work is adequately cited and highlights this point. Related work also appropriately cites relevant materials for the algorithms studied.\n\n# Quality\n\nThe empirical analysis uses appropriate statistical tests and shows significant effects. This is clearly a complete piece of work (aside my desire to see a larger population used in the experiment, which is beyond the scope of this paper per se).\n\nI would ask the authors to include the non-significant effects in a supplement or appendix to understand how they fit with the narrative presented. The focus on only positive results is reasonable, but means it is unclear whether contradictory results on related questions are being omitted.\n\nThe authors are clear on the ways the work may not generalize due to the test domain (Hanabi), while still addressing what aspects of the work may generalize (expert distrust of AI).\n\nOne comment: Figure 1 makes note that \"no player achieved a perfect score with OP\". This comment is true, but overlooks that seemingly many more players achieved a near-perfect score with OP compared to SB. So it's not clear what point this note is intending to make.\n\n\n# Clarity\n\nThe submission is clearly written and generally well organized.\n\nOne clarification (if possible without violating IRB and privacy concerns): how was the experiment population selected? Or what is it's composition? This would help understand the limitations to the generalization of the results to other populations.\n\nGiven many NeurIPS readers are not necessarily well-versed in experiment design and analysis it would help to inform the reader more about the choice of statistical tests. For example, why use a one-sample t-test? \n\nOthers could readily reproduce this work, aside from the choice of sample population.\n\n# Significance\n\nThe results are important empirically and drive the need for others to extend this work and incorporate this type of evaluation in future efforts with human-AI teams. Others *should* use these results to inform how human-AI agent teams are assessed; the paper reveals a clear gap in prior approaches to evaluation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper addresses the potential societal impacts of human-AI teaming should the paper results generalize. The work itself is an evaluation and thus is not necessarily likely to have negative societal impacts by sharing these results.\n", "review_text": "", "extra_scores": {"ethical_concerns": "The paper appropriately addressed IRB and ethical concerns.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "-Z4ae1cE8as", "reviewer": "Reviewer_tDFg", "summary": "This paper presents a comparative study on the objective and subject performance of a rule-based agent and a RL-learned agent for playing the card game Hanabi with human teammates. The authors selected each of the best-performing existing Hanabi AI from the rule-based methods and learning-based methods, and conducted a user study where human participants are invited to play the game with both types of AI agents and the team performances as well as the human’s subjective feedback are collected. The results show that, despite no significant differences in performance, the rule-based AI agent is preferred by human participants over the learning-based agent.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: This work is the first to study the human teammate’s subjective evaluation of different types of AI agents for the card game Hanabi. The user study is carefully designed and controlled to offer a comprehensive analysis of the two selected candidate AI agents. The results of the user study are presented in detail with clarity and the claims are supported with statistical analysis. The findings imply that human subjective evaluations are also important metrics for designing AI agents since the task performance is not indicative of subjective evaluations.\n\nCons: One concern for this paper is that the selection of candidate AI agents for testing is too limited (one from each category) and therefore the results may not generalize across different agents within the same game. Especially that RL training is an unstable process and the paper only tested one policy trained by RL to represent the full range of learning-based agents.\nOn the other hand, the findings do not seem to be too surprising since the rule-based AI agent is designed with hard-coded human knowledge and therefore are expected to play smoothly with human teammates while the RL-based policy was trained to optimize for zero-short coordination but the test scenarios in this user study is few-shot coordination. Moreover, the RL agent was not explicitly designed to perform well with experienced human teammates who have developed their conventions for communication. \n\nWhile I agree with the implication from the findings of this paper that human experiences/evaluations are important metrics on their own, I do not think this directly indicates that subjective metrics should be incorporated into RL training to achieve good human evaluations. Experiments and comparisons with RL agents trained or fine-tuned with human teammates are needed to show whether it is necessary to directly optimize for human subjective evaluations. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors commented on how this work could influence future research on AI agents and how this can have a larger impact on the users. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers were unanimous that the paper addresses an increasingly important area of research (human-AI collaborations), focuses on an understudied but crucial aspect of the research (i.e., how do humans perceive and assess their AI teammates?), and offers insightful results. While they found the paper well-written overall, reviewers suggested several avenues for further clarification and improvement, in particular, they would like the authors to further elaborate on their choice of the learning agent, the number and representativeness of participants in the study, and their arguments around ecological validity.  Given the authors’ comprehensive response to the reviews, I am confident they can revise their paper accordingly, so I recommend acceptance.
Decision: Accept (Poster)