The Utility of Explainable AI in Ad Hoc Human-Machine Teaming

1 Introduction . Collaborative robots ( i.e. , `` cobots '' ) and machine learning-based virtual agents are increasingly entering the human workspace with the aim of increasing productivity , enhancing safety , and improving the quality of our lives [ 16 , 22 ] . In the envisage of ubiquitous cobots , these agents will dynamically interact with a wide variety of people in dynamic and novel contexts . Ad hoc teaming characterizes this type of scenario , where multiple unacquainted agents ( in this case , humans and cobots ) with varying capabilities must collectively collaborate to accomplish a shared goal [ 33 , 52 ] . Ad hoc teaming presents a significant challenge in that agents are unaware of the capabilities and behaviors of other agents , and lack the opportunity to develop a team identity , shared mental models , and trust [ 9 , 10 , 57 ] . For example , the human may not be aware of the cobot ’ s possible actions DISTRIBUTION STATEMENT A . Approved for public release . Distribution is unlimited . This material is based upon work supported by the United States Air Force under Air Force Contract No . FA8702-15-D-0001 . Any opinions , findings , conclusions or recommendations expressed in this material are those of the author ( s ) and do not necessarily reflect the views of the United States Air Force . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and proclivities towards specific activities , limiting her ability to coordinate and plan effectively [ 35 , 48 ] . Furthermore , this lack of understanding negatively impacts the user ’ s ability to perform situational analysis , impeding a key component of the Observe-Orient-Decide-Act ( OODA ) loop [ 7 ] . For a human teammate to maintain situational awareness ( SA ) and effectively make decisions in human-machine teaming , the human must maintain an internal model of the cobot ’ s behavior . However , to develop such an understanding of the cobot , the human may have to constantly observe or monitor the cobot ’ s behavior , a costly and tedious process . Effective collaboration in teaming arises from the ability for team members to coordinate their actions by understanding both the capabilities and decision-making criterion of their teammates [ 37 , 47 ] . For human-human teams , Sebanz et al . [ 45 ] states that successful joint coordination among agents depends on the abilities to share representations , to predict other agents ’ actions , and to integrate the effects of these action predictions . Similarly , we believe these findings should correlate in human-machine teaming . Explainable AI ( xAI ) techniques , utilizing abstractions or explanations that provide the user insight into the AI ’ s rationale , strengths and weaknesses , and expected behavior [ 21 ] , can supply the human teammate a representation of the cobot ’ s behavior policy and may assist in the human teammate ’ s ability to predict and develop a collaboration plan . Furthermore , effective human-machine teaming requires the ability for team members to develop a shared mental model ( i.e. , team members share common expectations about the team coordination strategy , the outcomes of individual strategies , and the individual roles in achieving the team ’ s objective [ 34 ] ) [ 35 ] . xAI techniques offer the promise of enhancing team situational awareness , shared mental model development , and human-machine teaming performance . Recent work in the machine learning community on xAI has emphasized the importance of interpretability , and post-hoc explainability in enhancing the prevalence of machine learning-based approaches [ 43 ] . Other work has even explored simulatability [ 40 , 49 ] , assessing a human ’ s ability to observe a model ( e.g. , a decision tree , which lends itself to interpretability [ 14 ] ) and be able to produce the correct output given an input feature . Augmenting machine learning-based systems with some form of intepretability or simulatability can enable these systems to gain human trust [ 3 , 25 , 38 ] , an essential quality in high-performance teaming [ 28 ] . While prior work has provided approaches for explaining machine behavior through natural language [ 42 ] , interpretable decision trees [ 40 ] , and attention-based focusing [ 55 ] , the utility of collaborative agents augmented with explainable AI techniques in human-machine teaming has not been explored . In this work , we present two novel human-subjects studies to quantify the utility of xAI in humanmachine teaming . We assess the ability for human teammates to gain improved SA through the augmentation of xAI techniques and quantify the subjective and objective impact of xAI-supported SA on human-machine team fluency . We first assess if xAI can support the different levels of SA [ 15 ] by assessing how different abstractions of the AI ’ s policy support SA within a human-machine teaming scenario . Second , we study the effect of augmenting cobots with online xAI and assess how different abstractions of the AI ’ s policy affect ad hoc human-machine teaming performance . In Figure 1 , we present an overview of our experimentation in relation to the Observe-Orient-Decide-Act ( OODA ) loop . We provide the following contributions : 1 . We design and conduct a study relating different abstractions of the cobot ’ s policy to their induced situational awareness levels , measuring how different explanations can help a human perceive the current environment ( Level 1 ) , comprehend the AI ’ s decision-making model ( Level 2 ) , and project into the future to develop a collaboration plan ( Level 3 ) . Our results show that xAI techniques can support situational awareness ( p < 0.05 ) . 2 . We design and conduct an ad hoc human-machine teaming study assessing how online xAI-based support , generated via cobot abstractions , and the human ’ s ability to process higher levels of information affect teaming performance . We find novices benefit from xAIbased support ( p < 0.05 ) but are susceptible to information overload from more involved xAI abstractions ( p < 0.05 ) . Expert performance , on the other hand , degrades with the addition of xAI-based support ( p < 0.05 ) , indicating that the cost of paying attention to the explanation outweighs the benefits obtained from generating an accurate mental model of the cobot ’ s behavior . 2 Related Work . In this section , we discuss relevant prior literature in human-machine teaming and explainable AI . Human-Machine Teaming – The field of human-machine teaming is concerned with understanding , designing , and evaluating machines for use by or with humans [ 11 ] . Prior work has emphasized the importance of a shared mental model , noting the benefits of maintaining understanding and predictability within a human-machine team [ 17 , 18 , 36 ] . While these methods have been successful in helping humans identify robot intent , these approaches require modifying the action primitives to make the behaviors more expressive , which can degrade performance , or prior cross-training to enhance understanding , which limits the deployability of cobots . Explainable AI ( xAI ) – Explainable AI ( xAI ) is concerned with understanding and interpreting the behavior of AI systems [ 30 ] . Prior work has explored utilizing a clear visualization of a policy to help a human form an accurate representation of its capabilities [ 39 ] , extracting meaning from deep networks through explanations of each layer [ 38 ] , or generating explanations through separate networks [ 3 , 25 ] . While the varying types of explanations have been shown to be helpful in limited classification tasks , the utility of these explanations has not assessed in multiagent sequential decisionmaking problems , such as long-term interactions with cobots . xAI in Human-Machine Teaming – Explainable AI in human-machine teaming is a promising direction as automation with the ability to explain will allow users to better understand the behavior of their AI teammates . Prior work attempts to induce transparency in a human-robot team by the explanation of failure modes [ 41 , 53 ] , synthesis of policy descriptions [ 24 , 54 ] , and the verbalization of experiences [ 10 , 42 ] . While these approaches are successful are instilling a sense of understanding of robot behavior within the human teammate , the level of collaboration is limited , and these works do not assess the preoccupation cost of online explanations in human-machine teaming . More recently , Bansal et al . [ 5 ] , Bussone et al . [ 8 ] , González et al . [ 20 ] , Lai and Tan [ 29 ] , Zhang et al . [ 59 ] , and Zhang et al . [ 58 ] investigate the type and accuracy of an xAI explanation to a human ’ s trust and reliance on the setting of human-AI teaming . However , while this prior work deploys xAI-based support in classification problems ( utilizing the AI as a recommender system ) , our task and scenario are widely different in that we consider a complex sequential decision-making and planning problem where the AI is a collaborative agent that actively shapes the world . Anderson et al . [ 2 ] provides insight into how different xAI explanations relate to a human player ’ s mental model generation in a simple real-time strategy ( RTS ) game . Our experiment differs from Anderson et al . [ 2 ] in that the human actively teams with the cobot while receiving xAI-based support ( Section 5 ) , requiring users to build mental models on the fly . In Anderson et al . [ 2 ] , the human only acts as an observer , receiving explanations while the AI solely makes decisions within a RTS game . In our work , we present a foundational set of studies that ( 1 ) assess whether xAI-based techniques can support situational awareness in human-machine teaming and ( 2 ) quantify the costs and benefits of deploying online xAI techniques in an ad hoc human-machine sequential decision-making problem . We explore two types of xAI-based support , a short , text-based explanation of the cobot ’ s policy outputs and a decision-tree representation of the cobot ’ s policy . We note that providing human teammates with the complete cobot policy in the form of an interpretable decision tree , as we do in our work , is a type of xAI not investigated in the referenced prior work . 3 Human-Machine Teaming Domain . For our experiments investigating the deployment of xAI techniques in human-machine teaming , we utilize the Microsoft Malmo Minecraft AI Project [ 27 ] . Minecraft is an open-world environment where players can build structures , craft tools , and play alongside other individuals . Crafting within Minecraft can be classified as a hierarchical task that requires obtaining base materials before generating a more complex object/tool and may require traveling to a crafting table . Building structures is also a hierarchical task as lower layers must be ( partially ) constructed before the upper layers of a structure can be built . Collaboration among other individuals is highly complex as highperformance teaming requires intent recognition , task coordination , resource/tool sharing , among other personal factors such as game proficiency and trust among teammates . We generate a 61×61 grid utilizing the Python API provided by Johnson et al . [ 27 ] . Within this grid is two agents , the human and the cobot . The human plays with standard Minecraft controls , utilizing both a keyboard and mouse to continuously move and change her field of view . This teaming scenario is notably different from prior human-machine teaming experiments [ 9 ] that restrict human motion to discrete cardinal movements . The cobot is also programmed in continuous space with several action primitives further described in the supplementary material . Planning within this domain is challenging as agents take macro-actions that can take an arbitrary number of time-steps to execute which induces asynchronicity , closely resembling the complexities associated with that of solving a Decentralized Partially Observable Semi-Markov Decision Process ( Dec-POSMDP ) [ 1 ] . Within this domain , the human is unable to explicitly communicate with the cobot ( e.g. , through chat ) , and the cobot is able to communicate with the human through a policy abstraction depending on the experiment factor ( cf . Section 5 ) . The human also has access to a first-person view of the cobot ’ s screen , providing partial knowledge of the cobot ’ s current location , cobot ’ s current resources , cobot ’ s current action . The cobot has complete access to the human ’ s global state , including current resources , location , and approximate action . The cobot and human are able to share resources through a chest located at a static position , with both agents able to store and take resources depending on need . We provide additional environment dynamics within the supplementary material . As shown in Figure 2 , the human will see up to four elements on her screen . On the top-left , the participant will play Minecraft . On the top-right , the participant has a display showing the firstperson view of the cobot . On the bottom of the screen , we utilize the Pygame interface to display information regarding the cobot ’ s policy . On the bottom-left , the participant has a display that may show information regarding the cobot ’ s inference of her action ( i.e. , the cobot ’ s inference of the human ’ s behavior ) . On the bottom-right , the participant has a display that may show information regarding the robot ’ s policy ( i.e. , how the cobot chooses an action ) .

Summary: The paper presents two user studies to quantify the impact of explanations on Human-AI teaming. The first study experiments with varying cobot (AI) policies to understand how well humans can create a mental model and situational awareness. Results from this study show that explanations can assist mental model creation. The second study instead also explores how human expertise and ability impacts this process. Results from this study show that for novice users there is a clear benefit from XAI but for expert users such benefits degrade due to the extra cost that humans have to pay to process the AI/XAI information.  The formalization of situational awareness in 3 levels (perception, comprehension, projection) is interesting and potentially reusable for other RL applications.   Results are clearly presented and sufficient information is included for clarity and reproducibility.