Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi
Authors: Ho Chit Siu
Jaime Daniel Pena
Edenna Chen
Yutai Zhou
Victor Lopez
Kyle Palko
Kimberlee Chestnut Chang
Ross Emerson Allen
Venue: NeurIPS 2021 Poster (2021)
Keywords: Human-AI Teaming
Deep Learning
Reinforcement Learning

Abstract: Deep reinforcement learning has generated superhuman AI in competitive games such as Go and StarCraft. Can similar learning techniques create a superior AI teammate for human-machine collaborative games? Will humans prefer AI teammates that improve objective team performance or those that improve subjective metrics of trust? In this study, we perform a single-blind evaluation of teams of humans and AI agents in the cooperative card game Hanabi, with both rule-based and learning-based agents. In addition to the game score, used as an objective metric of the human-AI team performance, we also quantify subjective measures of the human's perceived performance, teamwork, interpretability, trust, and overall preference of AI teammate. We find that humans have a clear preference toward a rule-based AI teammate (SmartBot) over a state-of-the-art learning-based AI teammate (Other-Play) across nearly all subjective metrics, and generally view the learning-based agent negatively, despite no statistical difference in the game score. This result has implications for future AI design and reinforcement learning benchmarking, highlighting the need to incorporate subjective metrics of human-AI teaming rather than a singular focus on objective task performance.

Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi
Ho Chit Siu
Jaime D. Peña
Yutai Zhou
Edenna Chen
Victor J. Lopez
Kyle Palko
Kimberlee C. Chang
Ross E. Allen
Abstract
Deep reinforcement learning has generated superhuman Al in competitive games such as Go and StarCraft. Can similar learning techniques create a superior Al teammate for human machine collaborative games? Will humans prefer Al team- mates that improve objective team performance or those that improve subjective metrics of trust? In this study, we perform single- evaluation of teams of humans and AI agents in the cooperative card game Hanabi, with both rule-based objective
and learning- based agents. In addition to the game score, used metric of the human-A Al team performance, we also quantify subjective measures the human's 's perceived performance, teamwork, interpretability, trust, and overall preference of AI teammate. We find that humans have clear preference toward rule-basee AI teammate (SmartBot) over state-of-the-art learning- based Al teammate (Other-Play) across nearly all subjective metrics, and generally view the learning- bearning-based agent negatively, despite statistical difference in the game score. This result has implications for future Al design and reinforcement learning benchmarking, highlighting the need to incorporate subjective metrics of human-Al teaming rather than singular focus on objective task performance.
Introduction
Advances in artificial intelligence (AI) have resulted in agents that perform at superhuman levels within domains previously thought to be solely the purview of human intelligence. Such performance has been demonstrated through application of reinforcement learning (RL) environments such as board games [6. 39], arcade games [31], real-time strategy games [44], multiplayer online battle arenas [4], and simulated aerial dogfights [11].
In nearly all cases, these demonstrations of Al superiority are in purely adversarial, one- or two-player games. However, order to achieve wal-world applicability and adoption, AI must be able to demonstrate teaming intelligence, particularly with human teammates [23]. Due to the focus on adversarial games, teaming intelligence has been understudied RL research. Al teammates must also exhibit behavior that engenders an appropriate level of certain human reactions, such as trust, mental workload, and risk perception [26, 37]. Failure do so risks the same kind of misuse, disuse, and abuse that Parasuraman and Riley illustrated with traditional automation systems [37]. These issues are distinct from much of current multi-agent AI work, as AI that is able to team effectively with other AI agents has failed to work effectively with humans [9].
MIIT Lincoln Laboratory, {hochit. siu,, siniiten,,,jjpaaa,,,,a, yutai zhou, chestnut, ross allen)-011 mit.edu 1 MIT Department Electrical Engineering and Computer Science, edennadmit. edu U.S. Air Force Artificial Intelligence Accelerator, fvictor. lopez. 10, kyle.yplllll......e 11sus. af mil DISTRIBUTION STATEMENT Approved public release. Distribution is unlimited.
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

The objective of this paper is to evaluate human-Al teaming in the cooperative, cuperfect-informatne game of Hanabi. We consider not only the objective performance of a human-Al team, but also the subjective human reactions and preferences when working with different kinds of Al teammates. Based on the success of applying deep reinforcement learning to create superhuman AT adversarial games, we hypothesize that similar RI. techniques can render collaborative AI that outperform and are preferred over rule-based agents in human-Al Hanabi teams. Our results show that this hypothesis is not supported given the current state of the art of collaborative RL agents. Human participants show a clear preference toward rule-based AI even though the learning- Al perform no worse and are specifically optimized for teaming with previously unknown partners (e.g. humans) [21]. To the best of our knowledge, this is the first comparative study of objective performance of rule-based and learning- basedd Hanabi AI in human-teame -tamiing experiments, as well as the first quantified study of subjective human preferences toward such AI.
2 Background
2.1 Hanabi
Hanabi is a cooperative card game in which two to five players attempt to stack twenty-five cards into five different fireworks (piles), one for each suit (color) and by ascending rank (number). We consider only the standard, two-player version, where the deck is composed of 50 cards, five suits, each suit having three 1s, two 2s, two 3s, two 4s, and one 5, and each player is dealt hand of five cards. Hanabi's 's difficulty lies in the fact that players can only see their teammate's 's hand and never their own and communication about cards is strictly limited.
Games start with eight hint tokens and three bomb tokens. Each turn, a player may discard a card from their hand, play a card from their hand, or give a hint to teammate. Whenever card is played or discarded, it revealed and a new card is drawn from the deck. A correctly-playe card is discarded and the team loses one bomb.
card is placed
in the appropriate firework; an incorrectly-playy Hinting costs one hint token and allows a player to reveal either all cards of a certain suit or rank in their teammate's hand. Hint tokens are earned back when a card is discarded, or when 5 card is successfully played. The final score is the sum of the top card in each firework, for a maximum of 25 points. The game ends when all fireworks have been completed, the deck is empty (and each player has one additional turn), or the team loses 3 bombs.
Hanabi is purely cooperative game with imperfect information, and limited, codified communication. These properties make it an interesting challenge for teaming since players must consider the reasons for their teammates' actions and any implied information, while avoiding misinterpretations. Bard et al. [2] presents a more complete treatment of Hanabi and its properties as an Al problem.
2.2 Training and Evaluating AI Teams
Hanabi's uniquely collaborative nature has made the subject of several AI challenges in recent years [2, 46). In these challenges AI agents are paired with other teammate agents, and their performance measured based on the paradigms of self-play and cross-play [21].
Self-play (SP) is when an Al plays a game with a copy of itself as the opponent (adversarial games) or teammate (cooperative games like Hanabi). Self-play can be used as a form of RL training [1, 20],
and/or as a form of evaluation [2].
Cross-play (XP) is an evaluation-only paradigm where an agent is teamed with other agents (AI or humans) that were not encountered during training, measuring how well an agent can cooperate with previouslyunnsenn teammates. SP-trained agents achieve high SP-scores by developing "secretive conventions" that are not understood by agents not present during training, thus completely failing in the XP setting [21]. Cross-play with humans (human-play) is of particular importance as it measures human-machine teaming (HMT) and is the foundation for the experiments in our paper.
Cross-plan can be evaluated under the zero-shot coordination (ZSC) [21] or ad-hoc teaming [40] settings. ZSC assumes that teams are formed between agents that have no prior knowledge of each other; therefore it is impossible to train an agent to bias towards teammate idiosyncrasies, though the agents may agree to a common training strategy beforehand. On the other hand, ad-hoe teaming attempts to achieve coordination by having agents update their policy while interacting with other

---

Table Survey of Rule-based and Learning Based Hanabi Al Performance in 2- 2-Player Games
Hanabi AI
Original Author
References
Self-Play
Cross-Play
Human-Play
Van den Bergh
Van den Bergh 2016
[42, 45. 15]
13.8
10.8
Self- Self-Recognition
Osawa 2015
[36. 15]
15.9
Piers
Walton-RRvers 2017
[45, 38]
17.3
11.2
Intentional
Eger 2017
[13, 8]
12.6
13.8
15.0
Expectimax
Bouzy 2017
[5, 155
19.0
MirrorSituationa
Canaan 2018
[5]
20.1
12,4
RIS-MCTS
Goodman 2019
[15, 8]
20.6
13.3
WhFWThat
Wu 2018
[48, 2, 15]
19.5
FireFlower
D. Wu 2018
[47, 2, 14]
22.7
Implicature AI
Liang 2019
[28, 8]
18.9
15
SmartBot (SB)
O'D O'Dwyer 2019
[33, 2, 15]
23.0
IL- Valuebot
Sarmasi 2021
[38, 33]
18.0
ACHA
Bard 2020
[2, 20]
22.7
1.0
BAD
Foerster 2019
[14, 2, 20)
23.9
SAD
Hu 2019
[20, 21]
24.0
3,0
SAD+AUX
Hu 2019
[20, 21]
24.0
21.1
9.2
SPARTA
Lerer 2020
[27, 8]
24.6
Other-Play (OP)
Hu 2020
[21]
24.1
22.5
15.8
We provide Hanabi AI agents' scores as they reported in existing literature. This gives notional perspective of the current state of the art; however, we note that evaluation conditions have not been standardized throughout previous literature (i.e. number of game seeds for self-play, pool of teammates for cross-play etc.). Therefore some caution is needed when making direct comparisons between reported scores. Please respective literature for details evaluation conditions.
agents, and the the pool of partner agents can be known in advance (though it can be of arbitrarily large size, making such knowledge practically impossible to exploit).
2.3 AI for Hanabi
Table summarizes the most salient published Hanabi Al agents in 2-player games with game score in self-play, cross-play, and human-play. The table is separated into rule-based (top) and learning- -based (bottom) agent types. Rule-based agents have policy composed of a predefined set of rules to follow given any particular game situation, and the rules often derived from human domain knowledge. Learning-base based agents, on the other hand, use statistical learning methods to adjust the parameters of their policy. The mechanism governing what action the policy will choose is learned via the agent's experience, without the need for human domain knowledge.
Most early work focused on rule-based AI [36, 42, 45]. Goodman Competition [46] with a rule-based Monte Carlo Tree Search agent. O'Dwyer's SmartBot (SB) is the highest performing rule- based agent created date, in terms of self-play [33, 2, 15). Due to its state-o satee-o-fae afethh-art SP score, as well as a readily available implementation [33], we select SB as our rule-based agent in the Auman-AI experiments presented in Sections and 4. We found no previous work that evaluated SB in the cross-play or human-play settings.
the 2018 CoG Hanabi
Recently developed learning-based agents have demonstrated breakthrough Hanabi performance. Sarmasi et al. provide a collection of agents trained with imitation learning that nearly match the performance of the rule-based agents from which they were trained [38]. A sequence of publications [14, 20, 27, 21] offered reinforcement learning agents that each advanced the state the art self-play and/or cross-play performance at the time of publication. This culminated with the SPARTA [27] and Other-Play (OP) [21] agents; which, respectively, represent the highest performance to date in self-play and cross-play of any agent type.
The OP agent selected as the learning-basse based agent for study our human-Al Al experiments described in Sections 3 and 4. We choose OP, not only for its state-of-the-art cross-play performance, but also because it uses learning objective function specifically designed optimize for zero-shot coordination settings. Due to this optimization, expect that humans would subjectively prefer and

---

objectively perform better with OP agent teammates over other Al teammates that were not designed for the zero-shot coordination setting, such SmartBot.
handful of works have conducted human-play experiments; however these works are uncommon due the significant time and effort required to conduct such experiments. Both Eger et al. [13] and Liang et al. [28] propose rule-based AI derived from Gricean maxims [17] and achieved an average human-play score of approximately 15.0 over set of experiments that included hundreds of human participants. The learning-based Other-Play agent is arguably the highest performer in terms of human- play with an average score of 15.8, however the experiments were run with a much smaller pool of human participants [21]. No prior works were found that provide a comparative study of rule-based and learning- basedd--bdddd Hanabi Al in human experiments.
2.4 Human-Al Teaming and Metrics
interaction evaluations typically consider two broad categories of metrics: objective performance metrics (raw score, error rates, accuracy, time required, etc.), and subjective team- or human-focused metrics (situation awareness, trust, workload, etc.). Measurement of the former is heavily task-dependent, and are often the primary evaluation metries of Al systems. The latter, however, can also involve metrics for probing systems and teams in a quantitative way, with important implications for how technology is used and adopted [37].
One key metric of teaming is trust, which is defined by Lee and See as "the attitude that an agent will help achieve individual's goals in situation characterized by uncertainty and vulnerability" [26]. Potential difficulties with trust include trust calibration (whether one's trust of an agent is commensurate with its capabilities) and trust resolution (whether the range of situations where human trusts system is commensurate with its range of capabilities).
Closely related to trust are the notions of legibility (being expressive of one's intent) and predictability (matching one's expectations) [12]. measuring teaming, one might directly ask humans about their teammates' intent and their [12] argue that these two ideas trade off in the context of robot motion, but that legibility is the more important factor when working in close collaboration with human. Similar arguments may also be made in the context of Hanabi, where maximum legibility is key to guiding a team's actions in the tightly-coupled, imperfect- imperfect-informaatie information scenario.
expectations, particularly in relation to shared goals. Dragan et al.
Hoffman [19] proposes set of subjective and objective methods (overlapping with the aforemen- tioned ideas) to measure fluency in human-robot interaction, defined as the "coordinated meshing of joint activities between members of well-synchronized team." " Fluency encompasses subjective elements such as teamwork, trust, and positive perception; and objective measures of timing and concurrency.
Much of the contemporary work on learning based agents has not evaluated such teaming metrics, due to their focus on single-player [3] or adversarial games [39, 43]. In this work, we consider the use of both objective performance and subjective teaming based measurement of human-Al pairs in the context of Hanabi. We aim to provide a more thorough analysis of the human-AL -AI teaming data than previously-dic discusssd Hanabi HMT studies [13, 28, 21]. Rather than proposing new Al with better raw performance, we are interested in the subjective aspects of human-Al teaming with state-of-the-ard rule- and learning-based AI, and how they might drive future AI development.
3 Methods
3.1 Human-Al Teaming Experiment
Experiments consisted of two-player Hanabi games played by teams of one human participant and one Al agent. Experiments aimed to measure the objective team performance and subjective human reactions to different types of AI. The AI agents used in experiments were the Other-Play RL agent [21] (specifically the OP+ agent, hereafter referred to as the OP bot) and the "SmartBot" agent [33] (hereafter referred to as the SB bot), both of which have MIT Licenses. These agents were chosen because they were the top-performing learning-basse and rule-based Hanabi AI, respectively at the time of the experiment.

---

Participants were first introduced to the experiment and the rules of Hanabi as defined in Section 2.1. The experiment followed with brief familiarization game so participants could acquaint themselves with the game interface. Then, each participant played two sets (blocks) of three games, with each set using a different Al teammate. The participant was not informed which AT teammate played each block of games and the order of the Al teammates was counterbalanced over the course of the study. Participants answered Likert scale surveys after each game (Table 2, left), a NASA Task Load Index (TLX) survey after each block [18]. and a Likert scale survey after both blocks that directly compared their experience with both agents (Table 2. right), Likert scale survey questions were largely derived from compilation of similar questions in Hoffman et al. [19]. Unless otherwise noted, Likert scales were arranged with 1 corresponding to "strongly disagree" "
and 7 corresponding to "strongly agree." "
A total of 29 adult participants completed an experiment and each provided written, informed consent. The protocol was approved by the MIT Committee on the Use of Humans as Experimental Subjects (protocol E-2520) and the United States Department of Defense Human Research Protection Office (protocol M0T1.20200003). Participants received $10 USD gift card the end of their experiment, and the highest-scoring participant received an additional $50 gift card. Each experiment took approximately 1.5 to 2.5 hours. The total amount spent on participant payment was $350, with one participant being paid twice due to technical difficulties ending their initial session early. Experiments were conducted virtually, with all interactions occurring through eddo-cconeerecce online surveys, and the Hanabi game interface, adapted from [27]. No personallyiddentffaaln information was collected, and no significant risk to participants was expected.
Liang et al. [28] conducted human experiments with subjective survey questions; however the only significant result reported was that humans were more likely to mistake Implicature Al as another human. Although Hu et al. [21] conducted some human experiments with the OP bot and a self-play RL bot, their experiments only focused on game score, involved only one game with each agent, and did not consider participant expertise in Hanabi.
Table 2: Post-Game (Left) and Post-Experiment (Right) Evaluation Statements
7-point Likert Scale statement
G1
am playing well.
G2
The agent is playing poorly.
G3
The team is playing well.
7-point Likert Scale statement
G4
This game went well.
E1
Which agent did you prefer playing with? Which agent did you trust more? [trust] Which agent did you understand more? [legibility]
G5
The agent and have good teamwork.
E2
[fuency]
E3
G6
The agent is contributing to the success of the team.
E4
Which agent understood you better? [legibility] Which agent was the better Hanabi player? Which agent was more reliable? [predictabilit]] Which agent had better understanding of the game on average?
G7
understand the agent's 's intentions. |legibility]
E5
E6
G8
The agent does not understand my intentions. [legibility]
E7
G9
I feel comfortable playing with this agent.
E8
Which agent caused you have greater mental workload? [mental workload]
G10
do not trust the agent. [trust] The agent is not a reliable teammate.
G11
[predictability]
G12
not confident in my gameplay.
Left: Statement order was randomized when presented to participants. Right: Rating is "strongly prefer first agent" and rating is "strongly prefer second agent." Items associated with particular human factors constructs have the construct name in brackets (not shown to participants).
3.2 Statistical Analysis
We evaluated both objective and subjective metrics of human-Al teaming, with the overall hypothesis that the Other-Play RL agent (OP) is preferred over- and would outperform the rule-based SmartBot (SB) agent. As an objective measure, we consider the team score during each game. For subjective measures, we consider outcomes related to perceived performance, teamwork, legibility, comfort, trust, and workload as measured by the Likert surveys. Linear mixed-effects regression models were used for both objective and subjective variables, following the recommendation of Norman [32].
5

---

Due to space constraints, we do not present the results of the TLX here, but include one question on mental workload from the post-experiment survey (Table 2, E8).
Both objective and subjective omnibus tests for post-game results used second-order mixed-effects models with fixed factors of (1) Al agent, (2) self-rated Hanabi experience, (3) block (the first second set of three games), and (4) game number (first, second, or third game within a block), and a random factor of participant number. Al agent and participant number were considered categorical variables. Post-experiment surveys were evaluated separately with one-sample (-tests, and a Holm -onffrorrro multiple comparison correction, since these tests were looking for differences from a neutral value on the Likert scale, and there were multiple hypotheses being tested. Additionally, correlations between self-rated experience, and post-game responses were checked against game score.
4 Results
We present the results of the teaming experiment. Our results indicate that despite no significant difference in objective performance between teaming with the two agents (Section 4.1), human sentiment results (Sections 4.2 and 4.3) show a clear preference toward rule-based agent over learning-based agent.
For context, we have colored most of the plots to show the self-rated Hanabi experience level of the participant with which the data associated, and we note that our participant pool is skewed towards -eter--sel-------eee-eeee the demographic survey statement "I am experienced in Hanabi" (see supplemental materials for demographic survey results).
players. Self-rated Hanabi experience was based on ratings of
4.1 Game Score
The mixed-effects regression did not support score differences due to agent type (1(158) 0.374), 0.709) or self-rated Hanabi experience (I(158) 1.228, p = 0.221) (Figure 1). This means that the data do not support OP under-performing relative to SB.
The dependent variable of game score only has significant effect of block (b 5.282, t(158) 2. 2.636, p = 0.009), and a hear-significant effect of game number (b 2.957, t f(158) = 1.887, p = 0.061). The positive regression slope in both cases indicates that this is likely learning effect over time; i.e. the human participant adapts his/her play over time.
OP Games
5B Games
16
25
Mezn standerd
deviation: 14.39
Mezn standard deviatio: 14.07
14
Median
Median 14.0
12
Dersity
10
10
20
25
0
35
20
Scores
Hanabi Experience
Figure 1: Game scores by agent type (left) and self-rated player experience (right). No significant differences were found when teaming with either agent, and correlation with self-rated experience was significant only for SmartBot games.
Since score is the primary performance metric of interest in Hanabi, and acts as the reward function for RL agents in this domain, we examined a few additional correlations with score. Note that since these are bivariate correlations, they not complete the aforementioned statistical regression.
Self-rated Hanabi experience and score had a small, but significant positive correlation when pooling both agents' games (p 0.0053, r 0.214). Correlation remains significant for the subset of SB games (p- = 0.0023, r 0.247), but not significant for OP games (p = 0.0867, r 0.18)), indicating that for this bivariate analysis, experience only correlates with score for SB, not with OP.

---

A small, but sigmificant positive correlation was found between subjective team performance (G3, G4) and score (p 0.0003, r 0.275 and p 0.0002, 0.280). However, subjective ratings of self- and agent-performance (G1, G2) were not significantly correlated to score.
When participants were ordered by the absolute differences in their total scores with each agent, the ten subjects with the greatest differences (max/min point differences of 57 and 17), six of the had better performance with OP, but when considering only the top five subjects in terms of score difference (max/min differences of 57 and 28), only the participant with the greatest difference performed better with SB, and all four others performed better with OP. full plot of scores by subject and agent type is in the supplemental materials.
4.2 Post-Game Sentiments
Significant effects in post-game subjective measures are summarized in Table Although the statistical model considers factors of block and game, which were sometimes significant, we do report these main effects or their interactions in the table, in order focus on the independent variables of greater interest.
In all cases where the interaction of agent type and Hanabi experience found to be significant, more experienced Hanabi players rated the Other-Play agent much more negatively than the SmartBot agent, while novices rated the two similarly (Figure 2). However, there was significant difference between novice and expert ratings of the SmartBot agent. Cases where neither agent nor experience significant factors are shown in Figure 3.
Table 3: Post-Game Sentiment Statistics (Statistically-Signiie Factors Only)
Dependent Variable
Factor
P
I am playing well (G1)
experience
2.698
0.008
The agent playing poorly (G2)
experience
1.44
< 0.0001
The team is playing well (G3)
agent:experience
-2.082
0.039
The agent and I have good teamwork (GS)
agent
2.578
0.011
agenteexpriiecce
-3.021
0.003
T understand the agent's intentions (G7)
agentxexperience
-2.73
0.021
The agent does not understand my intentions (G8)
experience
2.098
0.037
agent:experience
-3.166
0.002
I feel comfortable playing with this agent (G9)
agent
3.302
0.001
agentrexperience
-3.561
< 0.0001
The agent is not reliable teammate (G11)
experience
3.159
0.002
Degrees of freedom for t-tests 164 in all cases. Agent is agent type (SmartBot or Other- and experience self-rated Hanabi experience. Agent:experience is the interaction effect agent and experience
To specifically examine the effect of self-rated player experience, we performed post-hoc pairwise comparisons cases where experience was significant. Participants were pooled into "novice" (n = 10, self-rated experience of < 5) and "expert" (n = 19, self-rated experience of > 5) groups, and comparisons were made on ratings of each agent in the cases where an interaction effect was significant (G3, G5, G7, G8, G9). The groups did not rate SB significantly differently, but experts always rated OP worse than novices did. The difference in G3 "the team is playing well" (L(85) 3.551, < 0.001, effect size d = 0.752) was not stark the others (1(85) 5.068 to 5.855, p < 0.0001, [d] > 1.0), but all were still clearly significant, and all but one case had large effect sizes.
4.3 Post-Experiment Sentiments
Participants' direct comparisons of the agents (Table 2) are shown in Figure 4. The statements were presented to participants as comparing the "first" and "second" agents (without reference to agent type, which was not disclosed to participants). For ease of interpretation, we matched the responses to the agent type, and flipped the scale as appropriate. All positively-framed questions (E1 to E8) showed a strong preference SB over OP (corrected p < 0.05), while mental workload (E8) was split.
7

---

G3: The team 8 playing well (OP)
G3: The team playing well (5B)
G5: The egent and have good teamwork (OP)
G5: The egent and have good teamwork (58)
25
25
25
25
20
20
20
15
15
15
10
10
understand agent's
understand agent's
The agent
agent does understand my
(OP)
25
25
25
retf-rated Hanabi Experience
Frequencies
15
10
5-
5-
G9: feel comfortable playing with this agant (OP)
G9: feel comfortable
playing with this agent (SB)
20
15
15
10
10
5
Rating
Figure 2: Participant rating for post-game questions by self-rated Hanabi experience (SB vs OP), where statistically significant differences related to factors of agent and/or experience were found (Table 3). The scale ranges from 1-7, corresponding to "strongly disagree" to "strongly agree".
agent not Seitrated Experience
playing agent playing poorly
40-
Frequencies
36-
10
Rating
Figure Participant rating for post-game questions by self-rated Hanabi experience where statistically significant differences related to factors of agent and/or experience were found. The scale ranges from 1-7, corresponding to "strongly disagree" to "strongly agree".
We note there were three participants who obtained a score of 24 with OP, with one of these participants obtaining 24 twice (no player achieved a 25 with OP). All three replied at the extreme end of our Likert scale (i.e., or 7) for question E1 with preference for SB. Interestingly, their cumulative scores for OP and SB, respectively, were: Participant 6 (played with OP first, self-rated experience of 7): 57 and 28; Participant 19 (SB first, experience 7): 68 and 48; Participant 20 (OP first, experience 6): 70 and 35. The participant with the highest cumulative score (Participant 2, OP first, experience 7) had cumulative scores of 68 (OP) and 54 (SB) and preferred SB with Likert rating of All participant scores are provided in the Appendix (Figure 7).
Participant commentary indicated that low mental workload when working with OP was often caused by frustration with the agent and giving up on teaming with For example, after OP bot failed to act several hints from the human ("I gave him information and he just throws it away") participant commented that " At this point, don't It know what the point is, regarding working with the agent.
8

---

E1: Which agent did you prefer playing with?
E2: Which agent did you trust more?
E3: Which agent did you understand more?
E4: Which agent understood you better?
12
10-
Self-ratd Hamaai Experience
Frequencles
E7: Which agent hed better understanding game average?
EB: Which agent caused you have a greater mental workload?
ES: Which agent was Hanaloi player?
EG: Which agent was more
12
10.
Preference
Figure 4: Post-experiment questions. All showed statistically significant preference for the rule- based SmartBot (p < 0.05), except the question on workload (E8). The scale ranges from ("strongly prefer OP") 7 ("strongly prefer SB").
Discussion
5.1 Other-Play Agent Performance
Other-Play (OP) training is designed to avoid the creation of "secretive" conventions that can result from self-play training. However, even though OP agents can pair well with previously-unseen teammates, there are still assumptions placed on the types of teammates OP agents will encounter. Notably, OP assumes that teammates are also optimized for zero-shot coordination [21]. In contrast, human Hanabi players typically do not learn with this assumption. Pre- gam- convention-setiing and post-gand game reviews are common practices for human Hanabi players, making human learning more akin few-shot coordination. Furthermore, Hu et al. note that, due practical implementation details, OP does not always avoid secretive conventions. For example, some OP agents use color hints to indicate discarding of the 5th card [21]. Such conventions not legible human teammate without extensive observation.
5.2 Implications for Human-Al Teaming
The results shown here have important implications for the development of (earning-basede human- teaming agents. Regardless of the objective performance of Al systems, "teaming intelligence" is ultimately required for real-world deployment. The difference between rule-based and learning- based systems is particularly poignant here, where humans strongly favored rule-based agent over an RL agent many subjective metrics (and never significantly favored the RL agent) despite achieving objective scores that were not significantly different across the two agent types. Caution should be exercised when generalizing our results beyond Hanabi, though there is some indication of cross-domain consistency. The negative post-experiment sentiments expressed towards OP by some participants are similar to those expressed by participants during the DotA2 OpenaAl Five games where RL players exhibited playstyles that were not immediately understood by human players, though they were successful [34, 35].
An additional consideration is the user base of future learning learning-based systems. We note that between post-gams and post-experiment surveys, nearly statements/questions associated with human factors constructs (except workload) had differences related to the agent and/or the participant's experience level. Experienced players rated the OP agent particularly negatively (Figure 2, Table 3), despite similarity in scores. As domain experts are likely to be the first users of Al technology in operational settings, their perception of Al is key factor to its adoption. Part of the experts' negativity toward OP may come from the way human experts make decisions in uncertain situations.

---

Klein's 's recognition eeoogition-primed decision massing model [25] indicates that experts typically rely on base of knowledge to recognize "prototypical" situations and alleviate much of the mental burden of decision-makiin making. Working with an unpredictable and illegible teammate breaks experts' ability to rely on much of their knowledge base. This notion also supported by the scorefexperience correlations (Figure 1) which were only significant for SB games.
This work indicates a need for RL methods that produce legible and predictable policies. Methods in this direction generally involve either building inherently transparent models, or procedures for post- hoc analysis. The former includes human languagggguggdrrr driven policy specification [41], or explanation learning through construction of causal diagrams [30]. The latter includes saliency maps for visual domains, and action preference explanation through reward decomposition [24].
5.3 Limitations and Future Work
The tight coupling of coordination and performance. and the codification of communication, make Hanabi unique test bed. Games without these properties may elicit different responses from human-Al teams, so generalization of this experiment beyond Hanabi must be treated carefully.
well as the lack of readily available Due to the manual and dime-intensive experimental procedure implementations for many pre-existing Hanabi agents such as Intentional [13] and Implicature AI [28]-we we were limited in our pool of participants and agents. Future work could increase both by using an online, operator-free game-play and survey platform.
While we made an effort to recruit participant with diverse level of experience, the participant pool was still skewed towards those with higher self-rating (though is likely less biased than the pool from [28], who recruited exclusively from online gaming communities). Since Hanabi performance heavily dependent on the team, it is difficult to obtain an objective single-player skill rating. Still, the relatively tight clustering of many (self-rated) experts' responses to OP in particular is notable.
Game outcomes and the feasibility of perfect games (i.e. score 25) are dependent on the starting deck. We did not control for deck configuration to avoid the issue encountered in Hu et al. [21], who explored relatively narrow portion of the game space due to using only two deck seeds.
Interesting extensions to this Hanabi experiment include varying participants' knowledge of their teammates, increasing the number of games played, and adding additional metrics of teaming, such as positive listening and positive signalling [22, 29]. Modifications to the agents may include adding logic filters to prevent learning-based agents from making "obvious" mistakes, modifying the reward function to more heavily penalize "trust-breaking" moves. Beyond the domain of Hanabi, other candidates for Auman-Al teaming experiments include StarCraft (which date has only shown adversarial RL agent performance with humans [44]), Overcooked [9], and mixed cooperative- competitive environments like Bridge [49] and Diplomacy [16].
Conclusion
This study measured the game performance and human reactions in mixed human-Al teams in the cooperative card game Hanabi, comparing outcomes when humans were paired with a rule-based agent and when paired with reinforcemeentllearing bearen--beanii based "Other-Plas Play" agent designed to maximize zero-shot coordination. Despite achieving similar scores between these teams, human players strongly preferred working with the rule-based agent, and view the Other-Play agent quite negatively, citing their bilateral understanding, trust, comfort, and perceived performance as reasons. The ability of Al agents team with humans is important determinant of whether they be deployed in many real-world situations. These results show that even tfte-off-the-rn RL agents largely fail to convince humans that they are good teammates, and suggest that human perception of AI needs greater consideration in future Al design and development if it is to achieve real-world adoption.
10

---

Acknowledgments and Disclosure of Funding
DISTRIBUTION STATEMENT A. Approved for public release. Distribution unlimited. This ma- terial is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommen- dations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering. @ 2021 Massachusetts Institute of Technology. Delivered the U.S. Government with Unlimited Rights, defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.
The authors would like to thank our experiment participants for their time. We thank Hengyuan Hu for providing the Other-Play model. We thank Peter Morales for his guidance during the early phase of this work.
References
[1] Trapit Bansal, Jakub Pachocki, Szymon Sidor, lya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. arXiv preprint 171.0377,, 2017.
[2] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
[3] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-22 279, 2013.
[4] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint
2019.
[5] Bruno Bouzy. Playing hanabi near-optimally. In Advances in Computer Games, pages 51-62. Springer, 2017.
[6] Murray Campbell, A Joseph Hoane Jr. and Feng-hsiung Hsu. Deep blue. Artificial intelligence, 134(1-2):57-63) 83, 2002.
[7] Rodrigo Canaan, Haotian Shen, Ruben Torrado, Julian Togelius, Andy Nealen, and Stefan Menzel. Evolving agents for the hanabi 2018 cig competition. In 2018 IEEE Conference on Computational Intelligence and Games (CIG), pages 1-8. IEEE, 2018.
[8] Rodrigo Canaan, Xianbo Gao, Julian Togelius, Andy Nealen, and Stefan Menzel. Generating and adapting to diverse ad-hoc cooperation agents in hanabi. arXiv preprint 3711, 2020.
[9] Micah Carroll, Rohin Shah, Mark K Ho, Thomas L. Griffiths, Sanjit A Seshia, Pieter Abbeel, and Anea Dragan. On the utility of learning about humans for human-ai coordination. arXiv preprint 190.05579, 2019.
[10] Jacob Cohen. Statistical power analysis for the behavioral sciences. Academic Press, New York, 1977. ISBN 0-12-179060-6.
[11] DARPA. AlphaDogfight Trials Foreshadow Future of Human-Machine Symbiosis. https: //www. darpa www.dda.......e.eeeeeeeeeeeeeeee http://eweueeeeetseeeeeeeeeeeeee 08000008-220000 26, 2020.
[12] Anca D Dragan, Kenton CT Lee, and Siddhartha S Srinivasa. Legibility and predictability of robot motion. 2013 8th ACM/FEE International Conference on Human Roobtt Interaction (HRI), pages 301-308. IEEE, 2013.
11

---

[13] Markus Eger, Chris Martens, and Marcela Alfaro Córdoba. An intentional ai for hanabi. In 2017 IEEE Conference on Computational Intelligence and Games (CIG), pages 68-75. IEEE, 2017.
[14] Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi agent reinforcement learning. In International Conference Machine Learning, pages 1942-1951. PMLR, 2019.
[15] James Goodman. Re-determinizing information set monte carlo tree search in hanabi. arXiv preprint 1rXiv: 2019.
[16] Jonathan Gray, Adam Lerer, Anton Bakhtin, and Noam Brown. Human-level performance in no-press diplomacy via equilibrium search. arXiv preprint (rxiv:20000029233 2020.
[17] Herbert P Grice. Logic and conversation. In Speech acts, pages 41-58. Brill, 1975.
[18] Sandra Hart and Lowell E Staveland. Development of nasa-tix (task load index): Results of empirical and theoretical research. In Advances in psychology, volume 52, pages 139-183. Elsevier, 1988.
[19] Guy Hoffman. Evaluating fluency in huban-robot collaboration. IEEE Transactions on Human-Machine Systems, 49(3):209-21 2019.
[20] Hengyuan Hu and Jakob N Foerster. Simplified action decoder for deep multi-agent reinforce- ment learning. arXiv preprint 1912 2019.
[21] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-plas paatt-pla"" for zero-shot coordination. In International Conference on Machine Learning, pages 4399 4410. PMLR, 2020.
[22] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pages 3040-3049. PMLR, 2019.
[23] Matthew Johnson and Alonso Vera. No ai is an island: the case for teaming intelligence. AI Magazine, (0(())16-28) 2019.
[24] Zoe Juozapaitis, Anurag Koul, Alan Fern, M. Erwig, and Finale Doshi-Veles Explainable reinforcement learning via reward decomposition. 2019.
[25] Gary A Klein. A sccognition-primet decision (rpd) model of rapid decision making. Decision making in action: Models and methods, 5(4):188-147 1993.
[26] John Lee and Katrina See. Trust in automation: Designing for appropriate reliance. Human f 46(1):50-88) 50, 2004.
[27] Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7187-7194, 2020.
[28] Claire Liang, Julia Proft, Erik Andersen, and Ross A Knepper. Implicit communication of actionable information human-ai teams. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1-13, 2019.
[29] Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin. On the pitfalls of measuring emergent communication. arXiv preprint 4rX0::1903.05168, 2019.
[30] Prashan Madumal, Miller, Sonenberg, and Vetere. Explainable reinforcement learning through a causal lens. In AAAI, 2020.
[31] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 102.v.331..5002, 2013.
12

---

[32] Geoff Norman. Likert scales, levels of measurement and the "laws" of statistics, Advances in health sciences education, 15(5):625-60 632, 2010.
[33] Arthur O'Dwyer. quuxplusonechaanabi framework for writing bots that play hanabi.
http://hhhtbbooommmuuuuuuuuuua/a 2019.
[34] OpenAL. Openai five defeats dota 2 world champions, Apr 2019. URL https:/t////ppaaa.ccoo blog/open hogg/ppeaii--o citimna--iii--eee
[35] OpenAL. Openai five, Dec 2019. URL UtC.htttp:///pppaat.com////////
[36] Hirotaka Osawa. Solving hanabi: Estimating hands by opponent's actions in cooperative game with incomplete information. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
[37] Raja Parasuraman and Victor Riley. Humans and automation: Use, misuse, disuse, abuse. Human factors, 39(2):330-255 1997.
[38] Aron Sarmasi, Timothy Zhang. Chu-Hung Cheng, Huyen Pham, Xuanchen Zhou, Duong Nguyen, Soumil Shekdar, and Joshua McCoy. Hoad: The hanabi open agent dataset. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 1646-16660 1648, 2021.
[39] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1 111/64499:1110-1111 140-114444 2018
[40] P. Stone, G. Kaminka, S. Kraus, and J. Rosenschein. Ad hoc autonomous agent teams: Collabo- ration without pre-coordination. In AAAI, 2010.
[41] Pradyumna Tambwekar, Andrew Silva, Nakul Gopalan, and Matthew Gombolay. Inter- pretable policy specification and synthesis through natural language and rl. arXiv preprini arXiv:2101.07/ 101.071100..... 2021.
[42] Mark JH van den Bergh, Anne Hommelberg, Walter A Kosters, and Flora M Spieksma. Aspects of the cooperative card game hanabi. In Benelux Conference on Artificial Intelligence, pages 93-105. Springer, 2016.
[43] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint 17Xiv:1708.04788. 2017.
[44] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in stareraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-33 335-354, 2019.
[45] Joseph Walton-Rivers, Piers R Williams, Richard Bartle, Diego Perez-Liebana, and Simon M Lu- cas. Evaluating and modelling hanabi- banabi-playing agents. In 2017 IEEE Congress on Evolutionary Computation (CEC), pages 1382-12 1389. IEEE, 2017.
[46] Joseph Walton-Rivers, Piers R Williams, and Richard Bartle. The 2018 hanabi competition. In 2019 IEEE Conference Games (CoG), pages 1-8. IEEE, 2019.
[47] D. Wu. lighttectorr/iieeloower https///ttta...ommiiiivveooorrrt 2018.
rewrite of hanabi-bot in scala.
[48] J. Wu.
wuthefiaashaa/hanaaii.ss Thanhaaabissa 2018.
Hanabi
simulation
in rust.
[49] Chih-Kuan Ych, Cheng-Yu Yu Hsich, and Hsuan- Assan-Tiin Lin. Automatic bridge bidding using deep reinforcement learning. IEEE Transactions on Games, 10:365-377, 2018.
13


Reviews:
[{"review_id": "VbsVFkeayld", "reviewer": "Reviewer_zPDw", "summary": "The paper describes an experiment where humans were playing against bots in the game of Hanabi, a cooperative card game. There were two types of bots: rule based, and reinforcement learning based. Even though the two types played equally strong, the humans, when asked, strongly preferred the rule based bots.\nThis result holds implications for design of future human-computer interaction bots.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written. The English is perfect, the text is very clear.\nThe originality of the paper is high: this is a social science experiment with 29 human participants to see if they like a rule based bot or a reinforcement learning based bot. No algorithms were developed for this paper by the authors. The outcome may be surprising, and is certainly interesting.\nThe qulaity of the work is good, the statistical analysis is elaborate, and enough details are provided about setup and analysis.\nThe paper is significant, as so far as results based on a single card game can be considered significant. The results are certainly refreshing.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Ethical and limitations are dealt with in an appropriate manner in this paper. The limitations are used to show future work.", "review_text": "", "extra_scores": {"ethical_concerns": "There appear to be no ethical concerns. Human subjects have given their consent.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ME_Qqzteay", "reviewer": "Reviewer_aZxX", "summary": "This paper examines human-AI collaboration (“teaming”) in Hanabi. It seeks to expand current research on Hanabi in two ways: (1) comparing performance of a rule-based policy and a learned policy (for the AI teammate), and (2) measuring agent performance through both objective and subjective metrics. The two policies achieved comparable performance in games with human participants; however, participants tended to subjectively evaluate the rule-based policy more positively than the learned policy.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality.* This paper helps pioneer an important topic of research for both human-AI collaboration and Hanabi researchers: subjective perceptions of and preferences over agent partners (in the context of human-AI co-play). As the authors indicate, relatively few Hanabi studies have tested how agent performance generalizes to human partners. Critically, though the paper does not develop any new algorithms or improve upon existing algorithms, it breaks new ground by drawing attention to the importance of human preferences and pioneering the subjective measurement of agent effectiveness. Related work is adequately cited.\n\n*Quality.* I think the submission has the potential to be a solid paper, but currently its quality falls into an intermediate zone of sorts. Four areas for improvement occur to me as I read the current draft (and could revise my rating if addressed):\n\n1) Disclarity in the paper’s background. The information communicated in Section 2.2 is central to the design decisions underlying the paper, but in practice I found the explanation(s) muddled. In particular...\n\n   - The discussion of self play from lines 67-73 seems overly convoluted. Isn’t it straightforward to say (and understand) self play refers to an agent playing with a copy of itself, either during training or for an evaluation? The training-evaluation distinction is frequently made in RL research, so it seems unnecessary to spend (limited) space explaining that self play can be used in two “closely related but [...] not synonymous” contexts. (Also, in contrast to the expectations set by lines 72-73, I didn’t see any need for disambiguation throughout the paper.)\n    - Does “human play” refer to any evaluation-time play with humans (e.g., even if human play was part of the training loop), or specifically evaluation-time play with humans not encountered during training? If the former, human play should be defined in a paragraph separate from the cross-play paragraph. If the latter, I would recommend clarifying by changing “referred to as human-play” to “which we refer to as human-play”.\n    - The paragraph from lines 81-87 is very difficult to follow. What are the core characteristics that define zero-shot coordination and ad-hoc teaming, and how are they mutually exclusive? It’s absolutely fine to choose one paradigm over the other for the experiments at hand, but I was left wondering whether there are any practical differences (especially given the parenthetical statement on lines 87-87).\n\n- Section 2.3 presented similar questions:\n    - What were the criteria used to select the studies included in Table 1? Section 2.3 offers a surface-level description (“most salient”), but doesn’t clarify what metrics (or prior paper) gives this selection weight. (I think it's also fine to just say \"We surveyed the literature and analyzed agent performance across the cited papers\".)\n    - What does “exhaustive numerical comparison” (in the Table 1 notes) mean? The idea that we shouldn’t draw any inferences from these numbers is in tension with the bolding of certain (high) values in the table. To add to the confusion, two cross-play values that are separated by 0.5 are both bolded, whereas a self-play value within 0.3 of the top self-play score for rule-based agents is not bolded.\n    - What are the actual defining characteristics for “rule-based” and “learning-based” agents? Lines 91-93 offer “tend” definitions (“Rule-based agents tend…”), without clarifying what defines or differentiates these two categories.\n - In Section 2.4, many of the “metrics” introduced appear to be concepts rather than metrics. Indeed, they are never operationalized or connected to the actual variables measured in Section 3.1 (and Table 2). How do these map to the measures chosen for the study?\n - Finally, lines 168-172 appear to concern findings from a prior, related study, and should likely be relocated to the background section.\n\n2) Lack of method transparency. The authors take several steps to support transparency and reproducibility (e.g., including the tutorial materials and demographic questionnaire in their supplementary files). However, there are several critical pieces missing for method transparency, particularly from a behavioral research perspective.\n\n    - Where were participants recruited from? The generalizability of findings from the paper will be different depending on participant representativeness (which differs between sources like Mechanical Turk or department colleagues).\n    - How much information were participants provided about the experiment? Were participants told they would play with two different players? Were they informed of the pay structure in advance? (This relates to how strongly participant performance was incentivized; i.e., whether they cared at all how they performed.)\n    - How were the statements for the Likert-type scales chosen? In the paragraph on lines 155-158, it would be helpful if the authors either cited prior studies or stated their intention with selecting or generating these scale items (e.g., “To measure the perceived legibility of the agent teammates, …”).\n    - What part of the experiment was administered by video conference? Many (if not most) online experiments are conducted without need for experimenter intervention via video call, so this was a bit unexpected.\n    - Where does the verbal participant commentary described on lines 240-244 come from, in terms of the experimental procedure? It doesn’t appear to be described anywhere in Section 3.1.\n\n3) Adequate (but not stellar) analysis. I agree with the authors’ overall conclusion -- that objective performance does not differ with the two agents, but subjective preferences do -- based on the analysis that they present. However, there is a good amount of room to improve their analysis and interpretative work.\n\n    - The factorial study design (2 [agent type] x 2 [game block]) indicates that an interaction term should be included between agent type and game block within the mixed-effect models. When interaction terms are excluded in these situations, it is possible for underlying interaction effects to incorrectly register as main effects (even if potentially unlikely with the authors’ current findings).\n    - Given the large number of Likert items completed by participants, I was surprised the authors did not try to generate any combined measures. For example, Likert scales (properly defined) are the result of summing responses for multiple, *a priori* selected items. Another option would be to conduct factor analysis and use the first *N* factors. Either approach reduces the total number of statistical tests run in a way that accounts for the potential correlation between participants’ responses. These approaches may also encourage us to think about the underlying constructs we are attempting to study (e.g., legibility or predictability).\n    - I would discourage the authors from discussing “near-significance” (lines 203-204). Adding a new interpretative threshold for marginal significance inflates false positive report probabilities. (I also notice that a *p*-value of 0.061 is interpreted as “near-significant” but elsewhere a value of 0.0867 is \"non-significant\").\n    - I encourage the authors to report full results regression tables in their supplement for better results transparency and reproducibility.\n    - (As one final incidental comment on the results, doesn’t the paragraph on lines 222-225 say the exact same thing as the paragraph on lines 226-233? Does having both paragraphs add any particular value? It took me a read or two to realize they might be discussing the same finding.)\n\n4) (Slightly) overstretched discussion claims. In general, I enjoyed the writing and argumentation in this paper. However, I feel that there were a few small areas in the discussion and conclusion where the evidence at hand is stretched a bit further than is plausible. The paper makes several claims about human approaches to Hanabi (“Human Hanabi players typically do not learn with this assumption”, “common practices”, “human learning [is] more akin to few-shot coordination”, lines 250-253) without citing any sources or offering evidence. The conclusion also claims that participants “cit[e] their bilateral understanding, trust, comfort, and perceived performance as reasons” (lines 307-308). Was this substantiated at any point in the paper (or in the data gathered in the experiment)? As I understand it, the design could offer at best correlational support for a link between preferences and perceptions of performance, etc.\n\n*Clarity.* The paper is mostly well written, though the “unclear explanations” and “lack of method transparency” concerns elaborated previously impinge on clarity.\n\n*Significance.* Multiple research teams have noted that Hanabi presents an interesting challenge domain for AI research. This paper substantially advances the state of knowledge around Hanabi agents, given the unique research approach, data collected, and questions posed. Though the authors note that the costs of human evaluation are high relative to AI research, I hope this paper will encourage others to engage in more human-AI collaboration research (and within that space, collect data on human perceptions and preferences).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would encourage the authors to strengthen their discussion of two limitations. First, their findings are specific to these agents -- without further empirical or conceptual evidence, the results can’t be immediately translated to rule-based and learned policies in general. Second, I would broaden the disavowal of generalization beyond Hanabi. The “tight coupling of coordination and performance, and the codification of communication” are not the only reasons to hesitate extending these conclusions to other domains. Would the authors expect these findings to generalize to a spatially extended game like Overcooked? What about a domain in which rule-based policies and learned policies did not perform comparably (vis a vis the lack of correlation between objective performance and subjective preferences)?\n\nOtherwise, I thought the authors adequately addressed limitations of their work. (They did not address negative societal impact.)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "KWycJbjfTuC", "reviewer": "Reviewer_mR2Y", "summary": "This paper experimentally evaluates player perception and subjective experience when playing Hanabi in a team with a learning or rule-based agent. A quantitative analysis of player survey responses indicates that while human-agent teams performed similarly in terms of score for either type of agent, players preferred to play with the rule-based agent. \n\nThe paper is very well written, and Hanabi is particularly well explained. The subject of the paper, and its results, are also quite interesting. My main criticism of the paper is that there is not enough qualitative analysis of player responses. The quantitative analysis shows a preference for the rule-based agent, and I can make assumptions as to why, but I wanted to read some analysis of the likely reasons based on player responses. I think this is important as this kind of study should give us some insight into what human players expect or desire in an agent teammate. I assume that player feedback was elicited as some responses are discussed for the survey question on mental workload.  \n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Page 6: How should we interpret the games scores by self-rate player experience plot in Figure 1? When I look at it, I see a correlation between player score and self-rated experience for OP games, but then the commentary says the correlation is not significant for OP games. Could the authors elaborate on this more, and provide more commentary for Figure 1? Given that the result for OP games -- that there is no significant correlation between self-rated player experience and score -- is a bit counter-intuitive, there needs to be more discussion here as to the likely reasons why this is the case. Why do you think there was correlation for SB games and not OP? \n\nIn the results where self-rated Hanabi experience is referred to, I would add the \"self-rated\" prefix so that a reader doesn't confuse \"Hanabi experience\" with \"subjective game-play experience\" (i.e., how they rated their experience in playing with a particular agent). I did this on my first read through the paper. \n\nI did find the use of OP and SB to denote the two agents used in the experiments confusing sometimes, as I had to keep reminding myself that OP was the RL agent and SB was the rule-based agent. The acronyms do make sense given the agents they are referring to, but it would have made my life easier if they were something like LB-OP and RB-SB. This is a minor point, however. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Ai-DTUP1kE", "reviewer": "Reviewer_Tigt", "summary": "The work investigates tradeoffs between objective metrics of performance and subjective metrics of partner collaborative-ness in human-AI cooperation scenarios, taking the Hanabi domain as a case study. After reviewing previous work in this domain, the authors then perform a detailed user study with two state-of-the-art human-AI bots (one rule-based, one learned), and demonstrate that interestingly, users tend to prefer rule-based rather than learned Hanabi agents (even though they have similar objective performance metrics). They argue that this has important implications for the field of AI, as we want AI agents to be considered trustworthy teammates in addition to obtaining high reward.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\n\nWhile there is no new method presented in the paper, the main contribution would consist of the in-depth analysis of a user study performed to assess 1) the performance of two difference categories of human-AI Hanabi agents and 2) how they are perceived by the users. While minor user studies were performed for individual agents (focusing on reward), I am not aware of any studies comparing different Hanabi AI bots or focusing on the qualitative aspects of the behavior. Generally, this result is interesting and provides further an in depth account of the phenomenon that by default, learned agents for collaboration will not be human-interpretable even if they perform very well with each other (and perform ok with humans). In particular, I don't know of any work showing that this holds even when comparing to rule based agents, which might tend to have more baked-in inductive biases which lead them to be more human-interpretable.\n\n## Quality / Clarity\n\nThe claim that people tend to prefer the SmartBot Hanabi AI agent relative to the Other-Play one is well supported by the statistical analysis, which seems thorough and well executed. The difference between the perceptions of novice and experienced players was particularly interesting – this seems like something worth spending more time on explaining. It would also be interesting to think about whether one should expect similar types of trends in other domains than Hanabi. \n\nGenerally, the paper is very clearly written, well organized, and a pleasure to read. The bar charts in particular were well made and very informative.\n\n## Significance\n\nWhile the analysis was clear, I found the takeaway of this work to be less so: are the authors simply showing that some agents are perceived to be less cooperative than others, even under similar levels of objective performance (311-312)? If so, it would be interesting to head what preliminary ideas for including these subjective judgements into the AI generation process would be. Alternatively, if the intention of the authors was to claim that rule-based agents in Hanabi tend to be perceived as better agents relative to learned ones (as seems to be suggested in various parts of the paper), doing a comparison with just one rule-based and one learned agent seems a bit limiting (more of this in the limitations below). Another potential takeaway might be that the authors are suggesting that one should use rule-based agents because they tend considered better partners (although this is not explicitly said anywhere in the paper)? \n\nGenerally, it would be helpful to spell out more obviously what the intended contribution is.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "If the intention of the authors was to claim that rule-based agents in Hanabi tend to be perceived as better agents relative to learned ones (as it seems in various parts of the paper), doing a comparison with just one rule-based and one learned agent seems a bit limiting. While it seems plausible that learned approaches tend to systematically be less interpretable to human partners, doing a comparison with just one member of each group (rule vs. learned) seems potentially insufficient: there might be other factors that are method-specific which influence the perception of the user. \n\nIn particular, I would not expect all rule-based systems to be better than all learned systems. If the rules are not human-like or not very good, trivially such a system would not be well-perceived by users. On the other hand, the untrustworthiness (and convergence to obscure equilibria) of learned AI agents doesn't seem to necessarily be an intrinsic characteristic of learned systems: if one trained AI agents which leverage human data for learning policies which complement users' policies, that would likely lead to much better perception of such agents (this phenomenon has been shown at least when comparing to pure self-play trained collaboration partners, e.g. in [9] from your paper).\n\nFurther qualifying the claim, reducing the generality of it, or addressing why comparing just one rule-based and learned agent is enough would strengthen the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers found the problem setting and focus on subjective agent preferences interesting and promising, but ultimately I don't believe the paper is ready for publication at NeurIPS. Reviewers raised concerns about 1) the generalizability of the findings given the focus on one learning agent and one bot in one environment, 2) the human evaluation protocol (especially recruitment and the single-blind setup), and 3) a lack of qualitative analysis backing up the quantitative findings. Although the author response to #2 was reassuring and #3 could potentially be addressed before publication, #1 ultimately limits the interest of the current work to the community. As suggested by the reviewers, the authors could increase the scope of their findings by e.g. including a behaviorally cloned agent, including another collaborative environment such as Overcooked, or including enough qualitative analysis that it becomes more clear where the results should and should not generalize.

Decision: Accept (Poster)