The Utility of Explainable AI in Ad Hoc Human-Machine Teaming
Authors: Rohan R Paleja
Muyleng Ghuy
Nadun Ranawaka Arachchige
Reed Jensen
Matthew Gombolay
Venue: NeurIPS 2021 Poster (2021)
Keywords: Explainable AI
Human-Machine Teaming
Situational Awareness
Ad Hoc Teaming

Abstract: Recent advances in machine learning have led to growing interest in Explainable AI (xAI) to enable humans to gain insight into the decision-making of machine learning models. Despite this recent interest, the utility of xAI techniques has not yet been characterized in human-machine teaming. Importantly, xAI offers the promise of enhancing team situational awareness (SA) and shared mental model development, which are the key characteristics of effective human-machine teams. Rapidly developing such mental models is especially critical in ad hoc human-machine teaming, where agents do not have a priori knowledge of others' decision-making strategies. In this paper, we present two novel human-subject experiments quantifying the benefits of deploying xAI techniques within a human-machine teaming scenario. First, we show that xAI techniques can support SA ($p<0.05)$. Second, we examine how different SA levels induced via a collaborative AI policy abstraction affect ad hoc human-machine teaming performance. Importantly, we find that the benefits of xAI are not universal, as there is a strong dependence on the composition of the human-machine team. Novices benefit from xAI providing increased SA ($p<0.05$) but are susceptible to cognitive overhead ($p<0.05$). On the other hand, expert performance degrades with the addition of xAI-based support ($p<0.05$), indicating that the cost of paying attention to the xAI outweighs the benefits obtained from being provided additional information to enhance SA. Our results demonstrate that researchers must deliberately design and deploy the right xAI techniques in the right scenario by carefully considering human-machine team composition and how the xAI method augments SA.

The Utility of Explainable AI in Ad Hoc Human-Machine Teaming
Rohan Paleja', Muyleng Ghuy', Nadun R. Arachchige', Reed Jensen', Matthew Gombolay'
'Georgia Institute of Technology, 'MIT Lincoln Laboratory
Atllnta, GA 30332, Lexiingonn MA 02420 (rpaleja3, mghuy3, nkra3) n @gatech edu, rjensen@l1. mit. edu mgombolay30gatecc.cem
Abstract
Recent advances in machine learning have led to growing interest in Explainable AI (xAI) to enable humans to gain insight into the decision-making of machine learning models. Despite this recent interest, the utility of xAI techniques has not yet been characterized in promise of enhancing team situational awareness (SA) and shared mental model development, which are the key characteristies of effective human machine teams. Rapidly developing such mental models especially critical in ad hoc human- machine teaming, where agents do not have priori knowledge of others' decision- making strategies. this paper, we present two novel human-subject experiments quantifying the benefits of deploying xAI techniques within teaming scenario. First, we show that xAI techniques can support SA (p 0.05). Second, we examine how different SA levels induced via a collaborative Al policy abstraction affect ad hoc human-machine teaming performance. Importantly, we find that the benefits of xAI are not universal, there is strong dependence the composition of the human-mas machine team. Novices benefit from xAI providing increased SA (p 0.05) but are susceptible to cognitive overhead (p 0.05). On the other hand, expert performance degrades with the addition XAI based support (p < 0.05), indicating that the cost of paying attention the xAI outweighs the benefits obtained from being provided additional information to enhance SA. Our results demonstrate that researchers must deliberately design and deploy the right xAI techniques in the right scenario by carefully considering human-machine team composition and how the xAI method augments SA.
teaming. Importantly xAI offers the
Introduction
Collaborative robots (i.c., "cobots") and machine fcarning-based virtual agents are increasingly entering the human workspace with the aim of increasing productivity, enhaneing safety, and im- proving the quality of our lives [16 22]. In the envisage of ubiquitous cobots, these agents will dynamically interact with a wide variety of people dynamic and novel contexts. Ad hoc teaming characterizes this type of scenario, where multiple unacquainted agents (in this case, humans and cobots) with varying capabiliitees must collectively collaborate accomplish a shared goal [33 52 Ad hoc teaming presents significant challenge in that agents are unaware of the capabilities and behaviors of other agents, and lack the opportunity to develop team identity, shared mental models, and trust (9 10 57]. For example, the human may not be aware of the cobot's possible actions
DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material based upon work supported by the United States Air Force under Air Force Contract No. FA8702-15-D 13702-155--000 15-0-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Loop
Decision-Making
Situational Analysis
Q1. Can Explainable techniques support Situational Awareness?
Level Perception
Level 2:
Level 3:
Robot Policy Explanation
Comprehension
Projection
Collaborativ Task
xAl-based support benefit human-
machine teaming performance?
Coliaboralive Robol
Human
No sAl-aaed Support
Exporty
Human
Action
Cobot Action
Stalus xA-ooseed Support
Decision xAU based
Expertise
Noview
Figure I: This figure displays an overview of our experimentation in relation to the Observe-Orient Decide-Act (OODA) loop. On the left, we display the human- machnn teaming interaction with both agents taking actions and the cobot outputting policy explanation to the human teammate. On the right-hand side, display the two questions assessed by human-subjects experiments.
and proclivities towards specific activities, limiting her ability to coordinate and plan effectively Furthermore, this lack of understanding negatively impacts the user's ability to perform (OODA) loop 77. For a human teammate to maintain situational awareness (SA) and effectively make decisions teaming, the human must maintain an internal model of the cobot's behavior. However, to develop such understanding of the cobot, the human may have constantly observe or monitor the cobot's behavior, a costly and tedious process.
35
48
situational analysis, impeding key component of the Diserve-Orient-Deccdee
Effective collaboration in teaming arises from the ability for team members to coordinate their actions by understanding both the capabilities and recision-making criterion of their teammates [37 47). For humman-human teams, Sebanz et al. [45] states that successful joint coordination among agents depends on the abilities to share representations, the effects of these action predictions. Similarly. we believe these findings should correlate in human-machin teaming. Explainable AI (xAI) techniques, utilizing abstractions or explanations that provide the user insight into the Al's rationale, strengths and weaknesses, and expected behavior 21 can supply the human teammate representation of the cobot's behavior policy and may assist in the human teammate's ability to predict and develop collaboration plan. Furthermore, effective human-machine teaming requires the ability for team members to develop a shared mental model (i.e., team members share common expectations about the team coordination strategy, the outcomes of individual strategies, and the individual roles in achieving the team's ' objective (34]) 33 xAI techniques offer the promise of enhancing team situational awareness, shared mental model development, and human-machine teaming performance.
predict other agents' actions, and to integrate
Recent work in the machine learning community on XAI has emphasized the importance of in- terpretability, and post-hoc explainability in enhancing the prevalence of machine learning- based approaches 43 Other work has even explored simulatability [40 49], assessing human's 's ability to observe a model (e.g., decision tree, which lends itself to interpretability (4) and be able to produce the correct output given an input feature. Augmenting machine learning-basee based systems with some form of intepretability or simulatability can enable these systems to gain human trust B 25 38), an essential quality in performance teaming (28). While prior work has provided approaches for explaining machine behavior through natural language [42], interpretable decision trees 440, and attention-based focusing(55) the utility of collaborative agents augmented with explainable AI techniques in naman-machinn teaming has not been explored.
In this work, we present two novel human-subjects studies to quantify the utility of xAI in human- machine teaming. We assess the ability for human teammates to gain improved SA through the augmentation of xAI techniques and quantify the subjective and objective impact of xAl-sppprteed SA on numan-machine team fluency. We first assess xAI can support the different levels of SA 15 by assessing how different abstractions of the AI's policy support SA within a

---

teaming scenario. Second, we study the effect of augmenting cobots with online xAI and assess how different abstractions of the Al's policy affect ad hoc numan-machinn teaming performance. In Figure] D we present an overview of our experimentation in relation to the Observe-Orient-Deeiidee (OODA) loop. We provide the following contributions:
1. We design and conduct study relating different abstractions of the cobot's policy their induced situational awareness levels, measuring how different explanations can help human perceive the current environment (Level 1), comprehend the Al's recision-making model (Level 2), and project into the future to develop a collaboration plan (Level 3). Our results show that xAI techniques can support situational awareness (p 0.05).
2. We design and conduct ad hoc human- teaming study assessing how online xAI-based support, generated via cobot abstractions, and the human's 's ability to process higher levels of information affect teaming performance. We find novices benefit from xAI- based support (p < 0.05) but are susceptible to information overload from more involved xAI abstractions (p < 0.05). Expert performance, on the other hand, degrades with the addition of xAI-based support (p < 0.05), indicating that the cost of paying attention to the explanation outweighs the benefits obtained from generating an accurate mental model of the cobot's 's behavior.
2
Related Work
In this section, we discuss relevant prior literature in human- machine teaming and explainable AI.
Human-Machine Teaming - The field of designing, and evaluating machines for use by or with humans [1]]. Prior work has emphasized the importance of a shared mental model, noting the benefits of maintaining understanding and
teaming is concerned with understanding.
predictability within a mamann team [7 [8 (36). While these methods have been successful in helping humans identify robot intent, these approaches require modifying the action primitives to make the behaviors more expressive, which can degrade performance, or prior cross-training to enhance understanding. which limits the deployability of cobots.
Explainable AI (xAI) Explainable AI (xAI) is concerned with understanding and interpreting the behavior of AI systems [30]. Prior work has explored utilizing a clear visualization of a policy to help a human form an accurate representation of its capabilities [3] extracting meaning from deep networks through explanations of each layer [38]. or generating explanations through separate networks (5). While the varying types of explanations have been shown to be helpful in limited classification tasks, the utility of these explanations has not assessed in multiagent sequential decision- making problems, such as long-term interactions with cobots.
XAI in Human-Machine Teaming Explainable AI in direction as automation with the ability to explain will allow users better understand the behavior of their AI teammates. Prior work attempts to induce transparency in human-robot team by the explanation of failure modes
teaming is a promising
[4] (53, synthesis of policy descriptions [24 54). and the verbalization of experiences [1O] 42). While these approaches are successful are instilling a sense of understanding of robot behavior within the human teammate, the level of collaboration is limited, and these works do not assess the preoccupation cost of online explanations in human- machine teaming. More recently, Bansal al. [S) Bussone al. R González et al. 20). Lai and Tan 29). Zhang et al. (59), and Zhang et al.
(88 investigate the type and accuracy of an XAI explanation to a human's 's trust and reliance on the setting of Auman-Al teaming. However, while this prior work deploys -bI-based support in classification problems (utilizing the Al as a recommender system), our task and scenario are widely different in that we consider a complex sequential decision-making and planning problem where the AI is collaborative agent that actively shapes the world. Anderson et al. [2] provides insight into how different xAI explanations relate to a human player's mental model generation in simple real-time strategy (RTS) game. Our experiment differs from Anderson et al. 2 in that the human actively teams with the cobot while receiving xAl-based support (Section requiring users to build mental models the fly. In Anderson et al. (2), the human only acts as observer, receiving explanations while the AI solely makes decisions within a RTS game.
In our work, we present foundational set of studies that (1) assess whether xAI-based techniques can support situational awareness human-machine teaming and (2) quantify the costs and benefits of deploying online xAI techniques in an ad hoc
sequential decision decision-making problem.
3

---

We explore two types of xAI-based support, short, text-based explanation of the cobot's policy outputs and a decision-uus tree representation of the cobot's policy. We note that providing human teammates with the complete cobot policy in the form of an interpretable decision tree, as we do in our work, is type of xAI not investigated in the referenced prior work.
3 Human-Machine Teaming Domain
For our experiments investigating the deployment of xAI techniques in human- human-machine machine teaming, we utilize the Microsoft Malmo Minecraft AI Project [27]. Minecraft is open-world environment where players can build structures, craft tools, and play alongside other individuals. Crafting within Minecraft can be classified as a hierarchical task that requires obtaining base materials before generating a more complex object/tool and may require traveling to a crafting table. Building structures is also hierarchical task as lower layers must be (partially) constructed before the upper layers of a structure can be built. Collaboration among other individuals is highly complex as high- performance teaming requires intent recognition, task coordination, resource/tool sharing, among other personal factors such as game proficiency and trust among teammates.
We generate 61 x61 grid utilizing the Python API provided by Johnson et al. (77]. Within this grid two agents, the human and the cobot. The human plays with standard Minecraft controls, utilizing both keyboard and mouse to continuously move and change her field of view. This teaming scenario notably different from prior numan-maching teaming experiments [9] that restrict human motion to discrete cardinal movements. The cobot is also programmed in continuous space with several action primitives further described in the supplementary material. Planning within this domain is challenging as agents take macro-actions that can take an arbitrary number of time-steps to execute which induces asynchronicity, closely resembling the complexities associated with that of solving a Decentralized Partially Observable Meri-Markov Decision Process PDcMPP) M
Within this domain, the human unable explicitly communicate with the cobot (e.g., through chat), and the cobot is able to communicate with the human through policy abstraction depending on the experiment factor (cf. Section 5 The human also has access to a first-person view of the cobot's 's screen, providing partial knowledge of the cobot's current location, cobot's current resources, cobot's current action. The cobot has complete access to the human's global state, including current resources, location, and approximate action. The cobot and human are able to share resources through a chest located at a static position, with both agents able to store and take resources depending on need. We provide additional environment dynamics within the supplementary material.
As shown Figure 2 the human will see up to four elements on her screen. On the top-left, the participant will play Minecraft. On the top-right, the participant has display showing the first- person view of the cobot. On the bottom of the screen, we utilize the Pygame interface to display information regarding the cobot's policy. On the bottom-left, the participant has display that may show information regarding the cobot's 's inference of her action (i.e., the cobot's inference of the human's 's behavior). On the bottom-right, the participant has a display that may show information regarding the robot's policy (i.e., how the cobot chooses an action).
3.1 Human-Machine Collaborative Task
The human and cobot are assigned to build house with certain specifications. The 4-level house contains eight unique objects including two types of planks, two types of stone, doors, stairs, fence, and fence gates totaling 89 objects. The latter four items are crafted materials that require combining base materials. Each agent has unique capabilities. The cobot has the ability to collect resources and craft certain items. The human is made aware of all possible behaviors of the cobot. However, the cobot cannot help the human build. The human player is able to collect resources and is in charge of building the house. The human is told a priori that her goal is to "Collaborate with the AI as best as possible to complete the house as quickly as possible." " However, the human is not strictly specified to build collect resources in any order.
Cobot Policy - The cobot maintains a hierarchical policy. low-level policy determines the cobot's action, and high-level policy determines the cobot's inference of the human's is action (which the low-level policy is conditioned upon). Both components of the hierarchical policy are decision tree-based policies of depth two and with four leaf nodes. The cobot's high-level policy stays fixed

---

Human
Robot Gameplay: First- Person Display
Gameplay: First-
Person Display
lobot Interence of Human Action
Robot Polly
Robot Inference Human Action Explanation
Dian Lonoe have
Tarman chepping Woue!
Robot Inference Human Action Explanation
Pirnen
Chif
Clegg Strck
Cruh Potuso
Figure 2: This figure displays a sample gameplay image where the cobot is augmented with the decision-tree explanation. Note this shows IV1:SA1-2-3 condition and IV2:Display Cobot Inference of Human Policy and Cobot Policy Section
throughout gameplay and is displayed in the bottom-left of Figure The cobot's 's low-level policy changes throughout gameplay across five phases of play, correlating with different periods of the task (i.e., building different portions of the house). A depiction of one low-level policy in displayed on the bottom-right in Figure 2
We conducted an initial IRB-approved pilot study with 30 participants to fine-tune the thresholds within the cobot's policy to enforce that the cobot is highly collaborative. We note that while the cobot policy models are hand-designed, the decision-tree based policies serve as surrogate for the interpretable models produced by interpretable machine barning-based approaches (40 50
4 Study 1: Relationship Between Explanations and Situational Awareness
Here, we present the details of our first human-subjens -unaess study relating different xAI techniques the three situational awareness levels specified by [15]. We explore the question:
Q1: Can xAI techniques be used support situational awareness (SA)? If so, which levels of SA are enhanced by xAI?
We review the different levels of situational awareness, study conditions, design, and procedures, and describe the measures employed.
4.1 Situational Awareness
Situation awareness (SA) represents user's internal model of the world around her at any point in time [15), representing the user's ability to perceive the elements of the environment, comprehend their meaning, and project their status in the near future. Situational awareness plays significant role in mental model alignment among teams and can largely impact performance [46). Following 11 44), the three levels of situational awareness induced through xAI techniques are follows:
Level 1: Perception Explanation of the current state of the world.
Level Comprehension Explanation of agent's current decision-
Level 3: Projection Explanation enabling user to predict future behavior.
The degree to which SA maintained typically assessed through the Situation Awareness Global Assessment Technique (SAGAT). The SAGAT prompts user random point within simulation with series of fact-based questions to determine her knowledge while blanking out the simulation display.
Situational Awareness Assessment We conduct a SAGAT questionnaire following the recommen- dations of 44 [1] that provide considerations for adapting the SAGAT towards evaluating situational
5

---

awareness in human machine teaming. Questions for Level 1,2 and 3 consisted of those that deter- mine the current state and action of the human and the cobot, determine the human's understanding of the cobot's s current capabilities and features considered in the cobot's 8 decision-making policy, and assess the user's ability to predict the cobot's next action, cobot action given modified inputs, and input requirements to produce a favorable action. Similar to Endsley [15], we maintain sample of questions for each level of situational awareness. We provide the user three questions per situational awareness level, restricting each question to multiple choice with 2-4 answer choices. We provide the complete list of questions for each SA level within the supplementary material.
While the SAGAT can provide insight into user's SA, this test is highly intrusive as it requires task interruption. These interruptions can be distracting, interfering with performance [32], increasing stress [3I], and likelihood of failure [56]. Accordingly, we choose to assess user SA and the effect of SA separately to reduce intrusiveness in the team flueney study and contribute a novel, targeted SA study across xAI techniques.
4.2 Experiment Conditions and Procedures
In this study, seek to determine how different abstractions of the Al's policy induce different levels of situational awareness. As such, we utilize a 1 1x3 within-subjects design varying across three abstractions: 1) No explanation of the robot's hierarchical policy, 2) A status explanation of the cobot's hierarchical policy, and 3) A decision-tree explanation of cobot's 's hierarchical policy.
No Explanation In this condition, the human is given no information about the cobot's policy, We note that even within the explanation condition, the human still has access to the first-person displays of both the robot and human.
Status Explanation - In the status explanation, the human is given information about the cobot's s policy relating the cobot's hicrarchical policy outputs. The information is given in the form of short phrase representing the information within the leaf node of the decision-tree explanation.
Decision- Tree Explanation In the decision-tree explanation, the human is given decision tree representing the cobot's 's policy with active edges and decision nodes highlighted. The decision trees displays complete information about the state features that the cobot uses make decisions and the choice of action. We display the sample game display of this condition in Figure
The experiment was conducted through an online platform where the first page provided the partic- ipants with introductory information regarding the numan-machinn teaming task. Users will then conduct three episodes in which each episode corresponds to a different factor. Each episode consists of 10 minutes of gameplay that users must view interrupted by the SAGAT at two-minute intervals (five trials within each episode). The factor/episode ordering were randomized to mitigate confounds.
4.3 Results
We recruited 48 participants in an IRB-approved experiment, whose ages range from 18 to 58 (Mean age: 21.69: Standard deviation: 5.61; 18.75% Female). We analyze our data using a repeated- measures ANOVA for the omnibus significance and Tukey HSD for pairwise comparisons, which includes Bonferroni corrections. Assumptions of normality and homoscedasticity are tested with Shapiro-Wilke and Levene's 's test respectively. We found that only our Level 2 data failed to meet these parametric assumptions, and thus Friedman's test with Nemenyi post-hoe (with corrections) was employed instead. We display our findings in Figure 3a within the supplementary material.
We provide additional analysis details
Q1: Overall, we find that xAI techniques support higher levels of situational awareness. No significant difference was found in the ability for users to perceive their environment (Level 1) with and without xAI-based support. This indicates that access to a first-person view of teammate gameplay and one's s own gameplay sufficient for perception of the environment. For Level 2 SA, a Friedman's 's test found significance (2) 34.2:p < 0.001) with the pairwise results showing both status xAI-based support (p < 0.05) and decision-tree xAI-based support (p < 0.001) significantly increases the ability for the user to comprehend her environment compared to cobots without xAI-based support. Lastly, we also observe an omnibus significance in the Level (F(2, 94) = 4.01: < 0.05), and find
that the Recision-tree xAI-based support enhances the user's ability to project cobot's 's behavior into

---

100
100
change
Change
sule lianed
75
Desitann Tree Boseo
SAGAT Score
25
25
Time
Time
-50
-50
-75
-75
Loudl Situational Awareness
Level
Levd
Explonation
Exponation Policy Information Level
Explonation
Explanation Policy Information Level
(a) Study Findings
(b) Study 2 Novice Findings
(c) Study 2 Expert Findings
Figure 3: This figure represents the findings of Study 1 (a) and Study (b-c). Figure Ba displays the
SAGAT scores across SA levels and xAI abstractions. Figure 3b and 3c display the performance residuals (inverse scale: lower is better) with xAI-based support across policy information levels with respect the no-explanation condition for novices (Figure 3b) and experts (Figure 3cb.
the near future (p < 0.05). These results support the hypothesis that xAI techniques enhance SA in human-machine teaming.
Study 2: Situational Awareness in Ad Hoc Human-Machine Teaming
In this section, we present the details of our second human--ubjern experiment exploring:
Q2: How does xAI-based support affect ad hoc human-machine teaming performance?
Q3: How does the amount information regarding the cobot's policy affect ad hoc human- machine teaming performance?
In Study 2, participant and a cobot are asked to complete the human-machine collaborative task defined in Section 3.1 We lifetime experience in Minecraft to participate. The exclusion criteria removes participants who are unfamiliar with the gameplay controls and ensures that participants are actively learning to play Minecraft while attempting to collaborate with cobot they are unfamiliar with. We note that the collaboration between the human and cobot is highly unstructured, and the human's 's behavior can range from choosing to ignore the cobot to actively perturbing the cobot to understand its behavior more clearly. Below, we review the experimental conditions, design, procedure, and describe the measures employed.
exclusion criteria allow only those with minimum of 20 hour of
5.1 Experiment Conditions
We conduct 2x3 mixed between-/vithin--uujee design with two factors: IV1: Induced SA (Section 5.1 and IV2: Policy Information (Section 5.11
IV1: Induced Situational Awareness Levels - The situational awareness levels are aligned via our first study detailed in Section We vary across 1) SA1: No Explanation, in which the human is able to perceive the environment, 2) SA1-2: Status Explanation. in which the human is able to perceive and comprehend the environment, and 3) SA1-2-3: Decision Tree Explanation, in which the human is able to perceive, comprehend, and project the environment.
IV2: Policy Information Level Here, describe the policy information levels that vary between subjects. Specifically, complete information provides users assistance with Theory of Mind (ToM) 19), reasoning strategically reason about one's actions in the context of the cobot's decision- making. However, the additional information provided for supporting ToM reasoning may come at the cost of increased complexity and risk of information overload.
Condition 1: Display Cobot Inference of Human Policy and Cobot Policy The cobot displays both its low-level and high-level policy, providing the user with increased information regarding its decision decision-making strategy.

---

Condition 2: Display Cobot Policy- The cobot displays only its low-level cobot policy. This condition provides partial information of how the cobot makes decisions, leaving out the cobot's inference of the human policy.
5.2 Procedure
The participant is first briefed on the objective of the experiment, to investigate the impact of online explanations human-machine teaming. The participant told that she will be playing with three different autonomous agents (use of deception). The participant starts with a pre-experiment survey collecting demographic information, gaming background (experience with video games and Minecraft), and the Big Five Personality Questionnaire [13] Afterwards, the participant is handed instructional document regarding specifications for the house and some notes regarding the cobot's s behavior. Next, the participant conducts brief hand-crafted tutorial in Minecraft. Once completed, the participant is tasked with individually building the entire house. The individual house build helps the human gain familiarity with the house specifications, which benefits in the user's task understanding. Once completed, the primary experimentation begins. Users will conduct three episodes in which she will play with three cobots, each of which are programmed the same but with varying XAI-based support and randomly selected level of policy information. The ordering across xAI-based support levels is randomized to mitigate potential experimental confounds, such as learning effect, fatigue, etc. In each episode, the participant will first be given a sample #:30-econd video that describes the upcoming visualization (Figure 2). Following the video, the participant is given a written tutorial describing the xAl-based support. Once completed, the participant will build the specified house in Minecraft with the cobot, a timed task. To conclude the episode, we administer several post-study scales to support our quantitative findings including the Human-Robot Collaborative Fluency Assessment (26), Inclusion of Other in the Self scale 41. Godspeed Questionnaire [6]. NASA-TLX Workload Survey [23]. Each scale has been verified for validity in prior work and is used to assess the quality of the numan--achinn teaming interaction. The Sluman-Robo Collaborative Fluency Assessment (2 measures team fluency, cobot contribution, trust, positive teammate traits, and perceived improvement through several sub-scales. The Inclusion of Other the Self 4 scale measures the perceived closeness between teammates. The Godspeed Questionnaire [ is used to measure perceived likability and perceived intelligence. The NASA-TLX (23) measures the task
workload. We provide additional details regarding our procedure in the supplementary material.
5.3 Results
We recruited 30 participants under an IRB- app-appd protocol, whose ages range 18 to 23 (Mean age: 19.97; Standard deviation: 1.30; 16.67% female), with participants randomly assigned to each of the factor levels of IV2 (the between-subjects variable) with 15 subjects per level. We evaluate participant performance by using the time taken for the human-machine team to finish building the house. Our data modeled as a mixed-effects ANOVA to capture any possible relation between independent variables across IV1 and IV2. We test for normality and homoschedascity in our data, and employed a corresponding non- pormmaracctiic test if the data failed to meet these assumptions. We provide details regarding our analysis within the supplementary. We display our objective findings in Figure and Figure 3c
Q2: We conduct a preliminary anaa-snalysis on our participant data and find two distinctive clusters in participants' individual build times (the separate, pre-test calibration task), implying two distinct categories of participants that vary in gameplay speed. We dichotomize our subject data with an additional categorical variable within our analysis representing each cluster, the first referring to "cxperts" (i.c., those with higher proficiency in the game of Minecraft) and "novices" " (i.c., those with lower proficiency in the game of Minecraft). We find that there are 17 experts and 13 novices within this dataset. While 13 are characterized as novice, it should be noted that our exclusion criteria serves to ensure that all participants are familiar with the gameplay controls of Minecraft and should thus be able to accomplish all key components of the teaming task. With respect to our separate calibration task, we find significant effect between participant's teaming ability with the cobot and the participant's 's individual build speed (F(1, 26) = 23.5: < 0.001).
Novice performance significantly differs across levels of xAI-based support (F (2))22) = 4.67: p < 0.05). We find that novices working with cobots augmented with status xAI-based support (SA/-2) able to complete the
teaming task significantly quicker than users that did not
8

---

Without xAl-Based Support
Status xAI Based Support
Decision-Tres Treee xAl-Based Support
Average Rating
0.8
0.6
0.4
0.0
Positive Traits
Perceived
Working
Closeness
Inte
Alliance
Subjective Measures
Figure 4: This figure represents the normalized subjective findings of Study 2. We that all users find cobots with decision-trea xAI-based support to maintain more positive teammate traits, maintain better working alliance, and are perceived as more intelligent than cobots without xAI-based support. Users also perceive both cobots with status xAl-based support and those with decision-tree xAI-based support as more close than cobots without xAI-based support.
have xAI-based support (i.e., the baseline condition) (p < 0.05). While we that xAl-based support the form of short phrase beneficial, more cognitively intense xAI-based support (decision-tree based support SAI-2-3) does not provide novices with benefits in performance. Status xAI-based support SAI-2 led to significantly improved performance compared to decision-tres xAI- based support (SAI-2-3) for novices (p 0.05). Conversely, have significantly increased time to build compared to those without xAI-based support, indicating a performance decrease (t(33) 2.09: 0.05). We provide further discussion in Section
find experts given xAI-based support
Q3: Expert performance significantly differs across policy information levels (F,1, 15) = 5.27:
0.05). We find that experts working with cobots augmented with partial information (only the cobot's s low-level policy) were able to complete the human-machine teaming task significantly quicker than users with complete information of the cobot's 's hierarchical policy (p < 0.05). We find no significant difference novice performance across information levels (IV2).
Subjective Findings:
While experts and novices interacted differently with cobots, the subjective findings were similar and are presented an aggregate. Here, we report on subjective measures that yielded statistical significance. We provide a full analysis within the supplementary material.
Q2: We find that and perceived intelligence significantly differently across xAI-based support (F(2, 56) [3.16, 4.08, 7.29, 5.31]; [p 0.05, 0.05, 0.01, 0.01)). Users rate a cobot with decision- tree xAl-based support as significantly more positive than a cobot with no explanation (p < 0.05) and rate muman-machine teams with decision-tree xAl-based support with higher working alliance scores than human-machine teams without xAl-based support (p < 0.05). We also find that users perceive cobots with decision-tree xAI-based support as significantly more close than cobots without xAI-based support (p < 0.01) and cobots with status xAl-based support as significantly more close than cobots with no xAI-based support (p < 0.05). In assessing the perceived intelligence of the cobot, we conduct Friedman's 's test and find that users perceive cobots with decision-tree xAI-based support as significantly more intelligent than cobots with no explanation (p 0.01). We provide depiction of our results for Q2 in Figure
assess positive teammate traits, team working alliance, closeness,
Q3: When displaying partial information IV2: Condition 2, we find that trust cobots and rate cobots with higher improvement scores across xAI-based support (F(2, 28) [3.55, 5.14]; Ip < 0.05, p < 0.05). We find users trust cobots augmented with decision-tree xAI-based support in comparison to a cobot augmented with no xAI-based support (p < 0.05) and numan-machin teams consisting cobot with decision-tre tree xAI-based support are rated with higher improvement scores than
teams consisting of cobots with no xAl-based support (p < 0.05).
Discussion
In Study (Section 4. we start by exploring xAI techniques can support situational awareness
in mananne teaming. We find objectively that xAI techniques can support SA. Specifically,

---

we see that providing users with status explanations supports the ability to comprehend AI behavior (Level 2. p < 0.05). More so, decision-tree based support provides users with the ability to both comprehend AT behavior (Level 2, p < 0.001) and project the cobot's s behavior into the near future (Level 3, p < 0.05). These results provide promising evidence supporting the deployment of xAI-based support in human-machine teaming.
In Study 2 (Section 5), we assess how cobots augmented with xAI-based support subjectively and objectively affect ad hoc human-machine teamning performance. Novices benefit from a cobot augmented with status xAI-based support (p < 0.05) but do not benefit similarly from a cobot augmented with decision-tree xAl-based support. The benefit achieved through status explanations (SAI-2) indicates that novices are able to use the support to develop a shared mental model that benefits them. However, the more cogmitively intense decision- tree xAI-based support does not provide any benefit, noting that such an XAI technique is not suitable for novices. Expert performance, on the other hand, degrades with the addition of xAI-based support, suggesting that the support serves as distraction. As experts are more experienced with the game of Minecraft, we hypothesize that the cost of paying attention to the xAI-based support outweighs the benefits obtained from generating an accurate shared mental model. As the cobot is programmed to be collaborative, the cobot behavior may already fall reasonably within the expert's 's shared mental model, and the support, while providing an accurate description of the cobot's 's policy, may reduce the user performance. This hypothesis is supported by the further reduction in performance when experts are presented with complete xAI-based support as opposed to partial (IV2. < 0.05). Thus, xAl-based support may not be universally beneficial and depends on the composition of the team.
While the performance benefits vary across experts and non-experts, we see that all users find cobots with decision-tree xAI-based support to be more trustworthy, capable of learning, maintain more positive teammate traits, maintain a better working alliance, and be perceived as more intelligent than cobots without xAI-based support. Given the results of our study, we wish to provide some key takeaways for future research in XAI and Machnne Teaming (HMT): 1) The addition of xAI techniques can induce SA, an important element of the OODA loop, 2) It is important to account for heterogeneity within both the cobot design [2] and the xAI-based support, and 3) Information overload can exist impedance in mamann-accine teaming and support strategies should consider the cognitive bandwidth of humans.
Limitations - In our teaming scenario, the cobot has specific capabilities and does not have equal capability to the human. Our findings are limited to teaming scenarios with similar heterogeneous agents. Our experiment is limited to two broad classes of interpretable models, representing xAI approaches that can generate decision trees and those that can generate a short phrase representing policy information. These abstractions can be generated via number of methods, including [0) We also did not explicitly vary the explanation size (e.g., depth of the decision tree, specificity of cobot status, etc.) in our experiment. Increasing the size/complexity of xAl-based support and accordingly modifying the training time is an important future direction to explore, as there is a tradeoff between the utility of xAl-based support and its complexity.
7 Conclusion
In this paper, we present findings relating xAI and situational awareness in numan-machine teaming. Within our first study, we find that cobots with xAI-based support can provide human teammates a higher level of SA, benefiting a human teammate's ability to perform situational analysis and understand the human-machin teaming scenario. Within our second study, we find that novices working with cobots augmented with status xAI-based support gain significant performance benefits (p < 0.05) within the ad hoc human- human-machine teaming scenario compared to cobots without xAI- based support. Expert performance, on the other hand, degrades with the addition of xAl-based support (p < 0.05), indicating that the cost of paying attention to the xAI-based support outweighs the benefits obtained from generating an accurate shared mental model. These results display the benefits and drawbacks of deploying xAI in ad hoc human-machine teaming and provide interesting future directions for xAI-based support in numan--aching teaming.
Broader Impact - Our work has the potential to benefit future research in deploying xAI in human- machine teaming and advance the envisage of democratizing cobots, focusing on developing higher levels of understanding between humans and cobots. We note that it is important to account for the differences across humans, such as expertise. In other scenarios, may be important to take into account other demographic information.
10

---

Acknowledgments and Disclosure of Funding
This work was sponsored by MIT Lincoln Laboratory grant 7000437192, NASA Early Career Fellowship grant 80HQTR19NON00-199CFFB and gift to the Georgia Tech Foundation from Konica Minolta, Inc.
References
[1] Chris Amato, G. Chowdhary, A. Geramifard, N. K. Ure, and Mykel J. Kochenderfer. Decentralized control of partially observable markov decision processes using belief space macro-actions. 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 5962-5969, 2015.
[2] Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, E. Newman, Jed Irvine, Souti Chattopadhyay, M. Olson, Alan Fem, and M. Burnett. Mental models of mere mortals with explanations of reinforcement learning. ACM Transactions on Interactive Intelligent Systems (TūS), 10:1 37, 2020.
[3] Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata, Grounding visual explanations. In Proceedings of the European Conference on Computer Vision (ECCV), pages 264-279, 2018
A. Aron, F. Aron, Michael Tudor, and Greg Nelson. Close relationships including other in the self. Journal of Personality and Social Psychology, 60:241-253, 1991.
[5] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Túlio Ribciro, and Daniel S. Weld, Does the whole exceed its parts? the effect of ai explanations on comple- mentary team performance. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021.
[6] Christoph Bartneck, Dana Kulic, E. Croft, and Susana Zoghbi. Godspeed questionnaire series. 2019.
[7] J. Boyd. The essence of winning and losing. 2012.
[8] Adrian Bussone, S. Stumpf, and D. O' O'Sulliv.n. The role of explanations on trust and reliance in clinical decision support systems. 2015 International Conference on Healthcare Informatics, pages 160-169, 2015.
[9] Micah Carroll, Rohin Shah. Mark K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan, On the utility of learning about humans for human-ai coordination. In NeurlPS, 2019.
[10] T. Chakraborti, Kartik Talamadupula, Yu Zhang, and S. Kambhampati. Interaction in human-robot societies.
2015.
[11] Jessie Y.C. Chen and M. Barnes, Human-agent teaming for multirohot control: A review of human factors issues. IEEE Transactions on Human-Machine Systems, 44:13-29, 2014.
[12] Letian Chen, Rohan R. Paleja, Muyleng Ghuy, and Matthew Craig Gombolay. Joint goal and strategy inference across heterogeneous demonstrators via reward network distillation. 2020 15th ACM/IEEE International Conference on Human- Robot Interaction (HRI), pages 659 668, 2020.
[13] Michael S. Chmielewski and Theresa A. Morgan. Five-Factor Model of Personality, pages 803-804. Springer New York, New York, NY, 2013. ISBN 144-1-4441-1105--- doi: 10.10077978-1-4441 1005-9 1226. https://://// org/ 10. 1007/978- 1-4419- 1005-9.0000 9_1226
[14] Finale Doshi- and Been Kim. Towards a rigorous science of interpretable machine learning, arXiv preprint /702 08608, 2017.
[15] M. Endsley, Situation awareness misconceptions and misunderstandings. Journal of Cognitive Engineering and Decision Making, 9:32 4, 2015.
[16] David Ferguson, Aaron Morris, Dirk Hähnel, Christopher Baker. Zachary Omohundro, Car- los Reverte, Scott Thayer, Charles Whittaker, William Whittaker, Wolfram Burgard, and Se- bastian Thrun.
autonomous robotic system for mapping abandoned mines In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems, vol- ume 16. MIT Press, 2004. URL Paper .ppper.pdf ppf
iewpeeeddinnnnnnuuppppeeeee neurips.ccmm http:/pssccppppeee/////////////e
[17] Michael J. Gielniak and A. Thomaz. Generating anticipation in robot motion. 2011 RO- RO-MAN, pages 449 454, 2011.
11

---

[18] Michael J. Gielniak, C. Liu, and A. Thomaz. Generating human-like motion for robots. The International Journal of Robotics Research, 32:1275 1301. 2013.
[19] A. Goldman. Theory of mind. 2012.
[20] Ana Valeria González, Gagan Bansal, Angela Fan, Robin Jia, Yashar Mchdad, and Srini Iyer. Human evaluation of spoken vs. visual explanations for open-domain qa. ArXiv, abs/2012.15075, 2020.
[21] David Gunning, Darpa's explainable artificial intelligence (xai) program. Proceedings of the 24th International Conference on Intelligent User Interfaces, 2019.
[22] Abhinav Gupta, Adithyavairavan Murali, Dhiraj Prakashchand Gandhi, and Lerrel Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL patt:p::/pproceeeeee wwwoeeedinga.nuuriss..... wwwprripa.cpppper///00/ htll://bbeeecccceee77/eee00bbb60
Paper. pdf
[23] S. Hart and L. Staveland. Development nasa-tlx (lask load index): Results of empirical and theoretical research. Advances in psychology, 52:139-183, 1988.
[24] Bradley Hayes and Shah. Improving robot controller transparency through autonomous policy explana- tion. 2017 12th ACM/IEEE International Conference on Human- RtamannRRoot Interaction (HRI, pages 303-312, 2017.
[25] Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Generating counterfactual explanations with natural language. arXiv preprint arXiv: (006:08800090000 09809, 2018.
[26] Guy Hoffman. Evaluating fluency in human-robot collaboration. IEEE Transactions on Human-Machite Systems, 49:209-218, 2019.
[27] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In IJCAI, 2016.
[28] Been Kim. Interactive and interpretable machine learning models for human machine collaboration. 2015.
[29] Vivian Lai and Chenhao Tan. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. Proceedings of the Conference on Fairness, Accountability; and Transparency, 2019.
[30] Pantelis Linardatos, Vasilis Papastefanopoulos, and S. Kotsiantis. Explainable ai: A review of machine learning interpretability methods. Entropy, 23, 2021.
[31] G. Mark, Daniela Gudith, and Ulrich Klocke. The cost of interrupted work: more speed and stress. In CHI, 2008.
[32] D. McFarlane and K. Latorella. The scope and importance of human interruption in human- compuecr interaction design. Claman-Compute Interaction. 17:1 61, 2002.
[33] Reuth Mirsky, William Macke, A. Wang, Harel Yedidsion, and P. Stone. A penny for your thoughts: The value of communication in ad hoe teamwork. In IJCAI, 2020.
[34] Susan Mohammed, Katherine Hamilton, Miriam Sanchez-Manzanenn and Ramón Rico. Team Cognition, chapter 16. pages 369-392. John Wiley & Sons, Ltd, 2017. ISBN 97811 18999977. doi: https://doi. 100/10.0002/978111 11w11111119999cchl111 URR.httppe://onnneliirre wiley. wwt.cooooooooo. htt////o/abb//..../ 1002/ 9781 1781118999997.. ch16
[35] S. Nikolaidis, Przemyslaw A. Lasota, Ramya Ramakrishnan, and J. Shah. Improved numan-robor team performance through cross-training, an approach inspired by human team training practices. The International Journal of Robotics Research, 34:1711 1730, 2015.
[36] S. Nikolaidis, Ramya Ramakrishnan, K. Gu, and J. Shah. Efficient model learning from joint-action demonstrations for human robot collaborative tasks. 2015 10th ACM//EEE International Conference on Robman-Rooot Interaction (HRI), pages 189-196, 2015.
[37] Yaru Niu, Rohan R. Palcja, and Matthew Craig Gombolay. Multi-agent graph-attention communication and tcaming. In AAMAS, 2021.
[38] Chris Olah, Arvind Satyanarayan, lan Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 3(3):e10, 2018.
12

---

[39] Steffi Paepeke and Leila Takayama. Judging a bot by its cover: An experiment expectation setting for personal robots. In Proceedings of the 5th ACM/IEEE International Conference on Human-robot Interaction, HRI 10, pages 45-52, Piscataway, NI, USA, 2010. IEEE Press. ISBN 978-1-4244-48337.
[40] Rohan R. Paleja, A. Silva, Letian Chen, and M. Gombolay. Interpretable and personalized apprenticeship scheduling: Learning interpretable scheduling policies from heterogeneous user demonstrations. arXiv: Learning, 2020.
[41] V. Raman and H. Kress-Gazit. Explaining impossible high-level robot behaviors. JEEE Transactions on Robotics, 29:94 104, 2013.
[42] S. Rosenthal, Sai P. Selvaraj, and M. Veloso. Verhalization: Narration of autonomous robot experience. In ICCAI, 2016.
[43] W. Samek, Grégoire Montavon, A. Vedaldi, L. K. Hansen, and K. Müller. Explainable ai: Interpreting. explaining and visualizing deep learning. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, 2019.
[44] Lindsay Sanneman and J. Shah. A situation awareness based framework for design and evaluation of explainable ai. Explainable, Transparent Autonomous Agents and Multi-Agent Systems, 12175:94 - 110, 2020.
[45] Natalie Sebanz, Harold Bekkering, and Gunther Knoblich. Joint action: Bodies and minds moving together. Trends in Cognitive Sciences, 10:70-76, 03 2006. doi: 10. 1e1.1//...i.s.200000 200..2005......00
[46] Sangwon Sco, Lauren R. Kennedy- Metz, M. Zenati, J. Shah, R. Dias, and Vaibhav Unhelkar. Towards an ai coach to infer team mental model alignment in healthcare. ArXiv, abs/2102.08500 2021.
[47] Esmaeil Seraj, Zheyuan Wang. Rohan R. Paleja, Matt B. Sklar, Anirudh Patel, and Matthew Craig Gombo- lay. Heterogeneous graph attention networks for learning diverse communication. ArXiv, abs/2 108.09568. 2021.
[48] J. Shah. Human-robot tcaming using shared mental models. 2012.
[49] Andrew Silva and Matthew Craig Gomholay. Encoding human domain knowledge to warm reinforce- ment learning. In AAAI, 2021.
[50] Andrew Silva, M. Gombolay, Taylor W. Killian, Ivan Dario Jimenez Jimenez, and Sung- Hyun Son. Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In AISTATS, 2020.
[51] David Sirkin, Nikolas Martelaro, Mishel Johns, and W. Ju. Toward measurement of situation awareness in autonomous vehicles. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2017.
[52] P. Stone, G. Kaminka, S. Kraus, and J. Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In AAAI, 2010.
[53] Stefanie Tellex, Ross A. Knepper, Adrian Li, D. Rus, and N. Roy. Asking for help using inverse semantics. In Robotics: Science and Systems, 2014.
[54] Vaibhav Unhelkar, Shen Li, and J. Shah. Decision-making for bidirectional communication in sequential human-robot collaborative tasks. 2020 15th ACM/IEEE International Conference on Human-Rond Robot Interaction (HRI), pages 329-341, 2020.
[55] Yequan Wang. Minlie Huang. X. Zhu, and L. Zhao. Attention-based Istm for aspect-level sentiment classification. In EMNLP, 2016.
[56] Westbrook, Raban, S. Walter, and Heather E. Douglas. Task errors by emergency physicians are associated with interruptions, multitasking, fatigue and working memory capacity: a prospective, direct observation study. BMJ Quality & Safery, 27:655 663, 2018.
[57] Bobbie Ann Adair White, Angela Eklund, T. Mcneal, A. Hochhalter, and A. Arroliga. Facilitators and barriers hoc team performance. Baylor University Medical Center Proceedings. 31:380 384, 2018.
[58] Yunfeng Zhang, Q Liao, and R. Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. Proceedings of the 2020 Conference Fairness, Accountability, and Transparency, 2020.
[59] Zhan Zhang, Y. Gene, Dakuo Wang. M. Ahsen, and Xiangmin Fan. Effect of ai explanations on human perceptions of patient-facing ai-powered healthcare systems. Journal of Medical Systems, 45, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contri- butions and scope? [Yes] The claims provided the introduction are supported through two human-subjects experiments, presented in Section Section S and thoroughly discussed in Section
(b) Did you describe the limitations of your work? [Yes] We descrihe the limitations of our work in Section G
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We note that the deployment of XAI may lead over-compliance in Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material CORR--Roooiiss-aaaiiiiiiii experiment domain NeurIPS2021. Furthermore, we provide details regarding our procedure both the main paper and the supplementary material to allow for the reproduction of our experimental findings.
URL)? [Yss] We provide codebase with
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Ys] All statistical tests and test statistics are reported within the main paper and supplementary. (c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes] Error bars for our objective results are reported in Figure
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
If you are using existing assets (e-g., code, data, models) or curatingfreleasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 'r us- ing/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
If you used crowdsoureing or conducted research human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicablle? [Yes] Yes, these have been attached in the supplementary material.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] Both of IRB-approved human-subject studies minimal risk, involving minor physical and mental stressors typical those encountered in an office staring at a screen and using keyboard and mouse controls.
(c) Did you include the estimated hourly wage paid to participants and total amount spent on participant compensation? [Yes] This information is included in the supplementary material.
14


Reviews:
[{"review_id": "xC6nN42EVhu", "reviewer": "Reviewer_jmSk", "summary": "The paper presents two user studies to quantify the impact of explanations on Human-AI teaming. The first study experiments with varying cobot (AI) policies to understand how well humans can create a mental model and situational awareness. Results from this study show that explanations can assist mental model creation. The second study instead also explores how human expertise and ability impacts this process. Results from this study show that for novice users there is a clear benefit from XAI but for expert users such benefits degrade due to the extra cost that humans have to pay to process the AI/XAI information.\n\nThe formalization of situational awareness in 3 levels (perception, comprehension, projection) is interesting and potentially reusable for other RL applications. \n\nResults are clearly presented and sufficient information is included for clarity and reproducibility.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The questions posed by the paper are important and relevant to the XAI field. \n\nThe paper and its novelty however is not very well position with respected work. The question of whether explanations help Human-AI teams has been recently explored in several works which are not discussed in this submission (see list below for examples). A lot of this related work also suggests that benefits from AI and XAI are higher when the human performance without AI is lower than AI alone. A lot of other work has also shown that it is difficult to improve human-ai team performance for experts. Thus, results in the paper need to be contextualized and clarified for novelty. Perhaps some aspect of novelty to consider is the RL application in this case. However, there is still some work in this area as well.\n\n\n\n\nRelated Work to consider:\nOn human predictions with explanations and predictions of machine learning models: A case study on deception detection\n\nBansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., ... & Weld, D. (2021, May). Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-16).\n\nGonzalez, A. V., Bansal, G., Fan, A., Jia, R., Mehdad, Y., & Iyer, S. (2020). Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA. arXiv preprint arXiv:2012.15075.\n\nAdrian Bussone, Simone Stumpf, and Dympna O'Sullivan. 2015. The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems. In Proceedings of the 2015 International Conference on Healthcare Informatics (ICHI '15). IEEE Computer Society, USA, 160–169. DOI:https://doi.org/10.1109/ICHI.2015.26\n\nEffect of AI Explanations on Human Perceptions of Patient-Facing AI-Powered Healthcare Systems.\n\nZhang, Yunfeng, Q. Vera Liao, and Rachel K. E. Bellamy. “Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making.” Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (2020): n. pag. Crossref. Web.\n\nMental Models of Mere Mortals with Explanations\nof Reinforcement Learning", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have adequately addressed the limitations and potential negative societal impact of their work.\n\nFurther suggestions beyond clarifying the work for novelty, would be to describe how the same type of approach or study style could apply to other RL applications.", "review_text": "", "extra_scores": {"ethical_concerns": "Authors have adequately touched upon the importance of considering a diverse set of human expertise in such studies and also the potential impact of automation bias and over reliance.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "uP0rK8zjCC7", "reviewer": "Reviewer_d8ML", "summary": "The authors present two user studies assessing the effect of decision tree-based explanations on human teammates in a human-cobot collaborative task - and specifically a minecraft house building task.  The results show that, while explanations can help situational awareness of the human subject, they can also overwhelm novices and hinder experts.", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is a pleasure to read.  Measuring the effects of explanations in human-machine collaborative settings is extremely important for all interactive AI/ML applications.  The idea of doing so using a modified Minecraft platform is quite clever.  I could not find any major issues with the experimental design.  The evaluation is quite thorough.\n\nTo the best of my knowledge, the related work has good coverage with one exception that I could find:  the potentially harmful effects of machine explanations have been thoroughly explored in\n\n  Ai et al., \"Beneficial and harmful explanatory machine learning\". MLJ 2021. \n  https://link.springer.com/article/10.1007/s10994-020-05941-0\n\nwithin the context of logic-based explanations.  Their results seem to anticipate the findings about the negative effects of explanations on novice players (although not the ones on expert players, as far as I could see).  It would be fair to mention and briefly discuss the link to this very relevant prior work.\n\nThis is a solid and welcome contribution and I do not have any majore complaints about the paper, just one remark/question.\n\n1. The explanations used in the experiments are quite small, essentially conjunctions of two literals.  Did you attempt to asses the impact of increasing the size/complexity of the explanations?  This question is quite central in the current XAI context, in which researchers do not shy away from presenting users with very detailed explanations.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors explicitly address some limitations of their study in Section 6.  Perhaps one could add that the effect of explanation size was not explicitly tested, and that explanations were assumed to be faithful to the policy, which is often not the case in current research.\n\nThe broader impact paragraph is just fine.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "UDhsY99-wgb", "reviewer": "Reviewer_jxhE", "summary": "This paper presents a study on the effect of two types of interpretable/explainable model in human-machine teaming. By varying the type of explanation (showing the model for human intention recognition used by the robot and showing the robot's action-selection model), the authors conclude that showing both models increased participants' situation awareness, but that only the human intention recognition model decreased the overall task completion time. For 'expert' participants, the performance degraded with\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Related work\n============\n\nDespite the claim in the paper, there is a bit of work looking at XAI in human-robot teaming; although it is not directly called this. Some articles I know of:\n\n- Some of Tatagatha Chakraborti's work looks at explanation and explicability for cobots -- there are a number of papers, and some of the tasks are collaborative.\n\n- Neerincx et al. Using perceptual and cognitive explanations for enhanced human-agent team performance. In International Conference on Engineering Psychology and Cognitive Ergonomics (pp. 204-214). Springer, Cham., 2018 -- this paper looks to build collaborative teaming \n\n- Madumal et al. Explainable reinforcement learning through a causal lens. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 03, pp. 2493-2500), 2020 -- this paper does evaluations with collaborative tasks.\n\nFurther, the `status explanation' appears to be the same as the idea of communicating intent. If the leaf nodes of the decision trees are explained, then this is simply explaining what the robot will do next. This has been studied in prior work in human-agent teaming; e.g., there are the articles that I know of, presumably there are others:\n\n- Harbers et al.: Explanation and coordination in humanagent teams: a study in the BW4T testbed. In Proceedings of the 2011 IEEE International Conferences on Web Intelligence and Intelligent Agent Technology, pages 17–20. IEEE Computer Society, 2011.\n\n- Harbers et al. Explanation in human-agent teamwork. In Coordination, Organizations, Institutions, and Norms in Agent System VII, pages 21–37. Springer, 2012.\n\n- Li et al. Communication in human-agent teams for tasks with joint action. In International Workshop on Coordination, Organizations, Institutions, and Norms in Agent Systems (pp. 224-241). Springer, Cham, 2015.\n\n- Unhelkar et al. Decision-making for bidirectional communication in sequential human-robot collaborative tasks. In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (pp. 329-341), 2020.\n\n\nNone of these touch on the same issues outlined in the paper, but they are perhaps more closely related than other references in the related work section.\n\nStudies\n=======\n\nThe studies using Malmo and the tasks are excellent. The task itself is reasonably rich and do run an interactive colalborative task is difficult to setup. This is the type of evaluation I'd love to see more of, and would love to do more of. I hope the authors share the setup and code for this once this paper is published. The actual policies are reasonably simple; perhaps simpler than anything I've seen in a truly collaborative task, but I do understand the limitations of working with participants recruited for studies and the time limits for training etc.\n\nThe study is well designed and separating studies between situation awareness and effects of SA effectively is a sound decision.\n\nThe major concern about the study is the data analysis. I *think* the analysis of the expert vs novice is unsound. The issue is that the split of the data into two sets is done on the same data that is used to analyse the results; that is, the performance of the participants is used to split the two cohorts, but then the performance is what is used to measure effectiveness of SA on the different conditions. This is circular analysis (see https://en.wikipedia.org/wiki/Circular_analysis) and is not valid. As such, the results on the split between novice and expert are not valid and should be removed. If, on the other hand, the expert vs novice was measured in some other way (e.g. number of hours played on a particular server previously), then this split would be fine.\n\nSome comments/questions:\n\n- Did you validate the 20 hours Minecraft minimum? Or is this just self-rated? If you did, it would be great to explain how you did this (always a challenge for researchers!)\n\n- The two polic-information levels in the 2nd study are interesting. Is there a reason you didn't do this in the first study? (Not a criticism, just a question).\n\n- It would be good to get some idea of what the different surveys (mentioned at the bottom of page 7) and why they are important/useful. In particular, one page 8 (under Q2), there is a link between \"teaming ability\" and build speed. What is the teaming ability? Is this the measures from these surveys?\n\n- It would be great if the results from Q2 (including subject findings) could be graphed if possible.\n\n- How was trust measured (Q3). I assume this is perceived trust measured via a survey/questionnire? I couldn't find any details about this.\n\nSome minor typos, etc:\n\n- page 5: \"find-tuned\" --> \"fine-tuned\"\n\n- page 6 and beyond: the use pf \"p < 0.05\" and similar -- a lot of people (including me) really like to see the p-values themselves, rather than just significant or not significant. I realise you also do \"p < 0.001\" (which is probably fine), but really a p=0.049 and p=0.051 are almost the same result and the arbitrary boundary of 0.05 is not so important.\n\n- page 9: \"HMT\" is used for the first time and is not introduced. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not particularly. In the 'broader impact' section, the authors mostly just summarise the general benefits of explainability. Looking into some of the broader issues with explainability, particularly in a paper that questions the usefulness for experts in particularly scenarios, would be useful.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "Ehs21GqaqFx", "reviewer": "Reviewer_VynQ", "summary": "This paper makes an initial study on how explanations from an AI agent impact human productivity on Minecraft tasks.  \n", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem studied in this paper is an important one, given the extensive amount of explainability work in the field right now.  The paper is also well-written and uses sound statistical analysis methods.  \n\nI am concerned about whether the narrow scope of the experiments makes this paper a poor fit to this conference.  Their results are understandable that different explanation models will have different effects, particularly if information overload might slow an expert on certain tasks.  However, it seems reasonable that other explanation approaches might work much better, improving productivity.  E.g., what of an image-based explanation (with an easily read color map), which could be read at a glance, for an expert?  Or what of a natural language-based explanation (more tutorial-like) for a novice?  Or a model that adapts its explanations based on human performance?  It seems that this question is better studied in tandem with HCI.  \n\nAnother question I have is how much does the quality of the agent's policy matter?   E.g., what if the cobot is dumb, or at least less optimal?   What effect do the different types of explanations have then?\n\nDetailed comments;\n- Line 78: \"a ad hoc\" -> \"an ad hoc\" \n- Line 164: \"find-tune\"\n- In Section 4.1, it would be good to have a citation for SAGAT. \n- In Section 5.2, why the use of deception? \n- Line 387: \"findings is limited\"\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "The only concern I thought of was the use of human subjects, but it was IRB approved. \n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "0PgLP_k8QM7", "reviewer": "Reviewer_j9dy", "summary": "This work investigates the impact of different levels of robot explanations in the context of human-machine collaboration, with respect to the impact on 1) the human’s situational awareness and 2) the time it takes to complete the task. It studies this in two separate human user studies.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Overall:**\n\nThis paper is well-written and clear. The experiments are well-designed and the metrics capture the outcomes of interest. The experiment task itself, collaboratively building a house in Minecraft, is an interesting and non-trivial task. I also appreciate that the human-robot collaboration is unstructured, in that the human is free to choose the extent to which they collaborate with the robot; this makes the studies more ecologically valid.\n\nThe results of the study on time completion are surprising and useful: experts perform the task most efficiently with no robot explanations at all, and novices perform best with simple robot explanations (i.e., text) rather than more detailed explanations (i.e., decision trees). This surprising result is of potential interest to the human-robot interaction community.\n\nThe main limitation of the work is that the policy is hand-crafted and very simple: a high-level and low-level policy, each of which is a two-level binary decision tree with a total of four leaf nodes. It is unclear whether the results of this study could be generalized for more complex policies and explanations, that cannot be summarized so concisely.\n\nI am also not sure whether this work is a good fit for NeurIPS, given that there is no algorithmic contribution. I think this would be a strong submission for HRI, since the main contribution is the user studies.\n\n**Novelty:**\n\nThe main novelty of this work is its thorough investigation of the utility of explainable AI for human-machine collaboration, in the form of two user studies on a non-trivial collaboration task in Minecraft. There is no algorithmic or theoretical novelty.\n\n**Quality:**\n\nThe experiments are well-designed. The analysis of the experiments is thorough and statistically sound. \n\n**Clarity:**\n\nThis paper is well-written, and overall the motivation and user study design are explained clearly. I just have one clarification question about the first user study—were the status and decision-tree explanations for both the high and low level, or just for the low level?\n\n**Significance:**\n\nI found it interesting that users had the most positive feelings toward the robot with decision-tree explanations, based on their answers to questions about trustworthiness, intelligence, positive teammate traits, etc. But they actually performed worse with it, compared to robots with simpler explanations (that they didn’t feel as positively about). This is a valuable insight for the human-robot interaction community, although it is unclear whether this would generalize to more complex policies and tasks.\n\nIn terms of future work, something I am quite curious about is whether there is an explanation technique that would obtain similar positive user perception as the decision-tree explanations, without hurting performance on the task.\n\n**Minor suggestions:**\n- In Figures 3b and 3c, it would be good to state what the performance residuals are with respect to. I assume it’s with respect to the no-explanation condition.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work addresses some limitations, but does not adequately acknowledge the limitation of whether these results are applicable for more complex policies.\n\nThis work has no direct negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a study on the effect of two types of interpretable/explainable models in human-machine teaming. By varying the type of explanation (showing the model for human intention recognition used by the robot and showing the robot's action-selection model), the authors conclude that showing both models increased participants' situation awareness, but that only the human intention recognition model decreased the overall task completion time. Furthermore, for 'expert' participants, the performance degraded with.
The paper has strengths in several aspects: The problem is significant; The study is well designed; and the paper is well-written. However, there are a few reservations: It is unclear if results can generalize to more complex settings; Some important references are missing.  

After a thorough discussion among reviewers, we feel the paper contains significant contributions, and the strengths outweigh the weaknesses, assuming the authors should be clear and explicit about them. In particular, please, add a more thorough literature review to alleviate the feeling that the paper is out of context and overclaiming. Further, try to explain the scope of the setting and emphasize the importance of one of the first analyses of xAI under the sequential decision-making settings.
 
Lastly, the issue of fitness of the work to NeurIPS raised by one of the reviewers was discussed with the SAC and Program Chairs, and the committee was encouraged to take a broad view of the scope of NeurIPS. Our goal is also to encourage a more experimental and data-driven type of AI and Machine Learning research.  
Decision: Accept (Poster)