<scp>DARPA</scp>'s explainable <scp>AI</scp> (<scp>XAI</scp>) program: A retrospective

Applied AI LettersVolume 2, Issue 4 e61 EDITORIALOpen Access DARPA's explainable AI (XAI) program: A retrospective David Gunning, David Gunning orcid.org/0000-0001-5435-796X Information Innovation Office (I2O), Defense Advanced Research Projects Agency (DARPA), Arlington, Virginia, USASearch for more papers by this authorEric Vorm, Eric Vorm orcid.org/0000-0003-2844-1994 US Naval Research Laboratory, Navy Center for Applied Research in Artificial Intelligence, Washington, District of Columbia, USASearch for more papers by this authorJennifer Yunyan Wang, Corresponding Author Jennifer Yunyan Wang [email protected] orcid.org/0000-0002-9102-0482 Quantitative Scientific Solutions (QS-2), Arlington, Virginia, USA Correspondence Jennifer Yunyan Wang, Quantitative Scientific Solutions (QS-2), 4601 Fairfax Drive Suite 1200, Arlington, VA 22203, USA. Email: [email protected]Search for more papers by this authorMatt Turek, Matt Turek Information Innovation Office (I2O), Defense Advanced Research Projects Agency (DARPA), Arlington, Virginia, USASearch for more papers by this author David Gunning, David Gunning orcid.org/0000-0001-5435-796X Information Innovation Office (I2O), Defense Advanced Research Projects Agency (DARPA), Arlington, Virginia, USASearch for more papers by this authorEric Vorm, Eric Vorm orcid.org/0000-0003-2844-1994 US Naval Research Laboratory, Navy Center for Applied Research in Artificial Intelligence, Washington, District of Columbia, USASearch for more papers by this authorJennifer Yunyan Wang, Corresponding Author Jennifer Yunyan Wang [email protected] orcid.org/0000-0002-9102-0482 Quantitative Scientific Solutions (QS-2), Arlington, Virginia, USA Correspondence Jennifer Yunyan Wang, Quantitative Scientific Solutions (QS-2), 4601 Fairfax Drive Suite 1200, Arlington, VA 22203, USA. Email: [email protected]Search for more papers by this authorMatt Turek, Matt Turek Information Innovation Office (I2O), Defense Advanced Research Projects Agency (DARPA), Arlington, Virginia, USASearch for more papers by this author First published: 04 December 2021 https://doi.org/10.1002/ail2.61Citations: 1 Funding information: Defense Advanced Research Projects Agency, Grant/Award Number: Explainable AI (XAI) AboutSectionsPDF ToolsExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Graphical Abstract Summary of Defense Advanced Research Projects Agency's (DARPA) explainable artificial intelligence (XAI) program from the program managers' and evaluator's perspective. Defense Advanced Research Projects Agency (DARPA) formulated the explainable artificial intelligence (XAI) program in 2015 with the goal to enable end users to better understand, trust, and effectively manage artificially intelligent systems. In 2017, the 4-year XAI research program began. Now, as XAI comes to an end in 2021, it is time to reflect on what succeeded, what failed, and what was learned. This article summarizes the goals, organization, and research progress of the XAI program. 1 CREATION OF XAI Dramatic success in machine learning has created an explosion of new AI capabilities. Continued advances promise to produce autonomous systems that perceive, learn, decide, and act on their own. These systems offer tremendous benefits, but their effectiveness will be limited by the machine's inability to explain its decisions and actions to human users. This issue is especially important for the United States Department of Defense (DoD), which faces challenges that require the development of more intelligent, autonomous, and reliable systems. XAI will be essential for users to understand, appropriately trust, and effectively manage this emerging generation of artificially intelligent partners. The problem of explainability is, to some extent, the result of AI's success. In the early days of AI, the predominant reasoning methods were logical and symbolic. These early systems reasoned by performing some form of logical inference on (somewhat) human readable symbols. Early systems could generate a trace of their inference steps, which could then become the basis for explanation. As a result, there was significant work on how to make these systems explainable.1-5 Yet, these early AI systems were ineffective; they proved too expensive to build and too brittle against the complexities of the real world. Success in AI came as researchers developed new machine learning techniques that could construct models of the world using their own internal representations (eg, support vectors, random forests, probabilistic models, and neural networks). These new models were much more effective, but necessarily more opaque and less explainable. The year 2015 was an inflection point in the need for XAI. Data analytics and machine learning had just experienced a decade of rapid progress.6 The deep learning revolution had just begun, following the breakthrough ImageNet demonstration in 2012.6, 7 The popular press was alive with animated speculation about Superintelligence8 and the coming AI Apocalypse.9, 10 Everyone wanted to know how to understand, trust, and manage these mysterious, seemingly inscrutable, AI systems. 2015 also saw the emergence of initial ideas for providing explainability. Some researchers were exploring deep learning techniques, such as the use of deconvolutional networks to visualize the layers of convolutional networks.11 Other researchers were pursuing techniques to learn more interpretable models, such as Bayesian Rule Lists.12 Others were developing model-agnostic techniques that could experiment with a machine learning model—as a black box—to infer an approximate, explainable model, such as LIME.13 Yet, others were evaluating the psychological and human-computer interaction aspects of the explanation interface.13, 14 DARPA spent a year surveying researchers, analyzing possible research strategies, and formulating the goals and structure of the program. In August 2016, DARPA released DARPA-BAA-16-53 to call for proposals. XAI program goals The stated goal of explainable artificial intelligence (XAI) was to create a suite of new or modified machine learning techniques that produce explainable models that, when combined with effective explanation techniques, enable end users to understand, appropriately trust, and effectively manage the emerging generation of AI systems. The target of XAI was an end user who depends on decisions or recommendations produced by an AI system, or actions taken by it, and therefore needs to understand the system's rationale. For example, an intelligence analyst who receives recommendations from a big data analytics system needs to understand why it recommended certain activity for further investigation. Similarly, an operator who tasks an autonomous system needs to understand the system's decision-making model to appropriately use it in future missions. The XAI concept was to provide users with explanations that enable them to understand the system's overall strengths and weaknesses; convey an understanding of how it will behave in future/different situations; and perhaps permit users to correct the system's mistakes. The XAI program assumed an inherent tension between machine learning performance (eg, predictive accuracy) and explainability, a concern that was consistent with the research results at the time. Often the highest performing methods (eg, deep learning) were the least explainable and the most explainable (eg, decision trees) were the least accurate. The program hoped to create a portfolio of new machine learning and explanation techniques to provide future practitioners with a wider range of design options covering the performance-explainability trade space. If an application required higher performance, the XAI portfolio would include more explainable, high-performing, deep learning techniques. If an application required more explainability, XAI would include higher performing, interpretable models. XAI program structure The program was organized into three major technical areas (TAs), as illustrated in Figure 1: (a) the development of new XAI machine learning and explanation techniques for generating effective explanations; (b) understanding the psychology of explanation by summarizing, extending and applying psychological theories of explanation; and (c) evaluation of the new XAI techniques in two challenge problem areas: data analytics and autonomy. FIGURE 1Open in figure viewer Defense Advanced Research Projects Agency (DARPA) explainable artificial intelligence (XAI) Program Structure, including technical areas (TAs) and evaluation framework The original program schedule consisted of two phases: phase 1, Technology Demonstrations (18 months); and phase 2, Comparative Evaluations (30 months). During phase 1, developers were asked to demonstrate their technology against their own test problems. During phase 2, the original plan was to have developers test their technology against one of two common problems (Figure 2) defined by the government evaluator. At the end of phase 2, the developers were expected to contribute prototype software to an open source XAI toolkit. FIGURE 2Open in figure viewer Challenge problem areas 2 XAI PROGRAM DEVELOPMENT In May 2017, XAI development began. Eleven research teams were selected to develop the Explainable Learners (TA1) and one team was selected to develop the Psychological Models of Explanation. Evaluation was provided by the Naval Research Lab. The following summarizes those developments and the final state of this work at the end of the program. An interim summary of the XAI developments at the end of 2018 is given in Gunning and Aha.15 XAI explainable learner approaches The program anticipated that researchers would examine the training process, model representations, and, importantly, explanation interfaces. Three general approaches were envisioned for model representations. Interpretable model approaches would seek to develop ML models that were inherently more explainable and more introspectable for machine learning experts. Deep explanation approaches would leverage deep learning or hybrid deep learning approaches to produce explanations in addition to predictions. Finally, model induction techniques would create approximate explainable models from more opaque, black-box models. Explanation interfaces were expected to be a critical element of XAI, connecting a user to the model to enable them to understand and interact with the decision making process. As the research progressed, 11 XAI teams explored a number of machine learning approaches, such as tractable probabilistic models16 and causal models and explanation techniques such as state machines generated by reinforcement learning algorithms,17 Bayesian teaching,18 visual saliency maps,19-24 and network and GAN dissection.24-26 Perhaps the most challenging and most unique contributions came from the combination of machine learning and explanation techniques27 to conduct well-designed psychological experiments to evaluate explanation effectiveness.28-31 As the program progressed, we also gained a more refined understanding of the spectrum of users and development timeline (Figure 3). FIGURE 3Open in figure viewer Explainable artificial intelligence (XAI) users and development timeline Psychological models of explanation The program structure anticipated the need for a grounded psychological understanding of explanation. One team was selected to summarize current psychological theories of explanation to assist the XAI developers and the evaluation team. This work began with an extensive literature survey on the psychology of explanation and previous work on explainability in AI.32 Originally, this team was asked to (a) produce a summary of current theories of explanation, (b) develop a computational model of explanation from those theories, and (c) validate the computational model against the evaluation results from the XAI developers. Developing computational models proved to be a bridge too far, but the team did gain a deep understanding of the area and successfully produced descriptive models. These descriptive models were critical to supporting the effective evaluation approaches, which involved carefully designed user studies, carried out in accordance with DoD human subject research guidelines. Figure 4 illustrates a top-level descriptive model of the XAI explanation process. FIGURE 4Open in figure viewer Psychological model of explanation. Yellow boxes illustrate the underlying process. The green boxes illustrate measurement opportunities. White boxes illustrate potential outcomes Evaluation Evaluation was originally envisioned to be based on a common set of problems, within the data analytics and autonomy domains. However, it quickly became clear that it would be more valuable to explore a variety of approaches across a breadth of problem domains. In order to evaluate the performance in the final year of the program, the evaluation team, led by Eric Vorm, PhD, of the US Naval Research Laboratory (NRL), developed an explanation scoring system (ESS). This scoring system provided a quantitative mechanism for assessing the designs of XAI user studies in terms of technical and methodological appropriateness and robustness. The ESS enabled the assessments of multiple elements of each user study, including the task, domain, explanations, explanation interface, users, hypothesis, data collection, and analysis to ensure that each study met the high standards of human subject research. XAI evaluation measures are shown in Figure 5, and include functional measures, learning performance measures, and explanation effectiveness measures. The DARPA XAI program demonstrated definitively the importance of carefully designing user studies in order to accurately evaluate the effectiveness of explanations in ways that directly enhance appropriate use and trust by human users, and appropriately support human-machine teaming. Often times, multiple types of measures (ie, performance, functionality, and explanation effectiveness) will be necessary to evaluate the performance of an XAI algorithm. XAI user study design can be tricky and the DARPA XAI program discovered that the most effective research teams were ones that featured diverse teams with cross-disciplinary expertise (ie, computer science combined with human-computer interaction and/or experimental psychology, etc.). FIGURE 5Open in figure viewer Evaluation measures for explainable artificial intelligence (XAI) algorithms XAI development approaches The XAI program explored many approaches, as shown in Table 1. TABLE 1. Technical approaches on Defense Advanced Research Projects Agency's (DARPA) explainable artificial intelligence (XAI) program Team Approach Application UC Berkeley Salience-map attention mechanisms implemented in DNNs19 Saliency maps for object detectors allow users to identify the detector which will be more accurate by reviewing sample detections and maps Transduction of DNN states into natural language explanations31 Explainable and advisable autonomous driving systems to fill in knowledge gaps. Humans can evaluate AI-generated explanations for navigation decisions21, 30 Charles River Analytics Causal models of deep reinforcement learning policies to enable explanation-enhanced training by answering counterfactual queries17 Human-machine teaming gameplay in StarCraft2 Developed a distilled version of a pedestrian detection model, which used convolutional auto encoders to condense the activations into user understandable “chunks” Carnegie Mellon University Robustified classifiers with salient gradients Interactive debugger interface for visualizing poisoned training datasets. Work is applied on the IARPA TrojAI dataset.33 Establishing objective/quantitative criteria to assess value of explanations for ML models34 Oregon State University iGOS++ visual saliency algorithm24 Debugging of COVID-19 diagnosis chest x-ray classifier Quantized bottleneck networks for deep RL algorithms Understanding recurrent policy networks through extracted state machines and key decision points in video games and control35 Explanation analysis process for reinforcement learning systems36 After-action review of AI decisions mirror the army's after action review system to understand why AI made its decisions to improve explainability and AI trust36, 37 Reinforcement learning model via embedded self-predictions Contrastive explanations of action choices in terms of human understandable properties of future outcomes38 Rutgers University Bayesian teaching to select examples and features from the training data to explain model inferences to a domain expert18 Interactive tool for analyzing a pneumothorax detector for chest X-rays. Targeted user study engaging ~10 radiologists demonstrated the effectiveness of the explanations39 UT Dallas Tractable probabilistic logic models where local explanations are queries over probabilistic models and global explanations are generated using logic, probability, and directed trees and graphs Activity recognition in videos using TACoS cooking tasks and WetLab scientific lab procedure datasets. Generates explanations about whether activities are present in the video data16 PARC Reinforcement learning implementing a hierarchical multifactor framework for decision problems Simulated drone flight mission planning task where users learned to predict each agent's behavior to choose the best flight plan. User study tested the usefulness of AI-generated local and global explanations in helping users predict AI behavior40 SRI Spatial attention VQA (SVQA) and spatial-object attention BERT VQA (SOBERT)20, 23 Attention-based (gradCAM) explanations for MRI brain-tumor segmentation. Visual salience models for video Q&A Raytheon BBN CNN-based one-shot detector, using network dissection to identify the most salient features41 Explanations produced by heat maps and text explanations42 Human-machine common ground modeling Indoor navigation with a robot (in collaboration with GA Tech) Video Q&A Human-assisted one-shot classification system by identifying the most salient features Texas A&M Mimic learning methodology to detect falsified text29 News claim truth classification UCLA CX-ToM framework: A new XAI framework using theory-of-mind where we pose explanation as an iterative communication process, that is, dialog, between the machine and human user. In addition, we replace the standard attention-based explanations with novel counterfactual explanations called fault-lines43, 44 Image classification, human body pose estimation A learning framework to acquire interpretable knowledge representation and an augmented reality system for explanation interface45, 46 Robot learning to open medicine bottles with locks and allows user interventions to correct wrong behaviors Theory of mind explanation network with multilevel belief updates from learning Minesweeper-like game to find optimal path for an agent IHMC Explanation scorecard Evaluate the utility of an explanation. Defines seven levels of capability, from the null case of no explanation, to surface features (eg, heat maps), to AI introspections such as choice logic, to diagnoses of the reasons for failures Cognitive tutorial A straightforward way to help users understand complex systems is to provide a tutorial up front but the tutorial should not be restricted to how the system works27 Stakeholder playbook Survey of stakeholder needs, including development team leaders, trainers, system developers, and user team leaders in industry and government AI evaluation guidebook Identifies methodological shortcomings for evaluating XAI techniques, spanning experimental design, control conditions, experimental tasks and procedures, and statistical methodologies 3 XAI RESULTS, LESSONS LEARNED Three major evaluations were conducted during the program: one during phase 1 and two during phase 2. In order to evaluate the effectiveness of XAI techniques, researchers on the program designed and executed user studies. User studies are still the gold standard for assessing explanations. There were approximately 12 700 participants in user studies carried out by XAI researchers, including approximately 1900 supervised participants, where the individual was guided through the experiment by the research team (eg, in person or on Zoom) and 10 800 unsupervised participants, where the individual self-guided through the experiment and was not actively guided by the research team (eg, Amazon Mechanical Turk). In accordance with policy for all US DoD funded human subjects research, each research protocol was reviewed by a local Institutional Review Board (IRB) and then a DoD human research protection office reviewed the protocol and the local IRB findings. In the course of those user studies, several key takeaways were identified. Users prefer systems that provide decisions with explanations over systems that provide only decisions. Tasks where explanations provide the most value are those where a user needs to understand the inner workings of how an AI system makes decisions. (Supported by 11 experiments across performer teams.) In order for explanations to improve user task performance, the task must be difficult enough that the AI explanation helps (PARC, UT Dallas). User cognitive load to interpret explanations can hinder user performance. Combined with the previous point, explanations and task difficulty need to be calibrated in order to improve user performance (UCLA, Oregon State). Explanations are more helpful when an AI is incorrect and are particularly valuable for edge cases (UCLA, Rutgers). Measures of explanation effectiveness can change over time (Raytheon, BBN). Advisability can improve user trust significantly over explanations alone (UC Berkeley). XAI is useful for measuring and aligning mental models for users and XAI systems (Rutgers, SRI). Lastly, since the last year of XAI took place during the unprecedented times of the COVID-19 pandemic, our performer teams developed best-practices for designing web interfaces to conduct XAI user studies when in-person studies were not possible (OSU, UCLA).47 As mentioned earlier, there seemed to be a natural tension between learning performance and explainability. However, throughout the course of the program, we found evidence that explainability can improve performance. From an intuitive perspective, training a system to produce explanations provides additional supervision, via additional loss functions, training data, or other mechanisms, that encourages a system to learn more effective representations of the world. While this may not be true in all cases and significant work remains to characterize when explainable techniques will be more performant, it provides hope that future XAI systems can be more performant than current systems while meeting user needs for explanations. 4 STATE OF THE WORLD, AI, AND XAI AFTER DARPA PROGRAM IN 2021 There currently is no universal solution to XAI. As discussed earlier, different user types require different types of explanations. This is no different from what we face interacting with other humans. Consider, for example, a doctor needing to explain a diagnosis to a fellow doctor, a patient, or a medical review board. Perhaps future XAI systems will be able to automatically calibrate and communicate explanations to a specific user within a large range of user types, but that is still significantly beyond the current state of the art. One of the challenges in developing XAI is measuring the effectiveness of an explanation. DARPA's XAI effort has helped develop foundational technology in this area, but much more needs to be done, including drawing more from the human factors and psychology communities. Measures of explanation effectiveness need to be well established, well understood, and easily implemented by the developer community in order for effective explanations to become a core capability of ML systems. UC Berkeley's result21 demonstrating that advisability, the ability for an AI system to take advice from a user, improves user trust beyond explanations is intriguing. Certainly, users will likely prefer systems where they can quickly correct the behavior of a system in the same ways that humans can provide feedback to each other. Such advisable AI systems that can both produce and consume explanations will be key to enabling closer collaborations between humans and AI systems. Close collaboration is required across multiple disciplines including computer science, machine learning, artificial intelligence, human factors, and psychology, among others, in order to effectively develop XAI techniques. This can be particularly challenging, as researchers tend to focus on a single domain and often need to be pushed to work across domains. Perhaps in the future a XAI-specific research discipline will be created at the intersection of multiple current disciplines. Toward this end, we have worked to create an XAI Toolkit, which collects the various program artifacts (eg, code, papers, reports, etc.) and lessons learned from the 4-year DARPA XAI program into a central, publicly accessible location (https://xaitk.org/).48 We believe the toolkit will be of broad interest to anyone who deploys AI capabilities in operational settings and needs to validate, characterize, and trust AI performance across a wide range of real-world conditions and application areas. Today, we have a more nuanced, less dramatic, and, perhaps, more accurate understanding of AI than we had in 2015. We certainly have a more accurate understanding of the possibilities and the limitations of deep learning. The AI apocalypse has faded from an imminent danger to a distant curiosity. Similarly, the XAI program has produced a more nuanced, less dramatic, and, perhaps, more accurate understanding of XAI. The program certainly acted as a catalyst to stimulate XAI research (both inside and outside of the program). The results have produced a more nuanced understanding of XAI uses and users, the psychology of XAI, the challenges of measuring explanation effectiveness, as well as producing a new portfolio of XAI ML and HCI techniques. There is certainly more work to be done, especially as new AI techniques are developed that will continue to need explanation. XAI will continue as an active research area for some time. The authors believe that the XAI program has made a significant contribution by providing the foundation to launch that endeavor. Biographies David Gunning (now retired) is a three-time DARPA program manager, who created and managed the XAI program from its inception in 2016 to its mid-point in 2019. His portfolio of DARPA research programs made significant contributions to the development of AI over the past 25 years. He led the Personalized Assistant that Learns (PAL) program, which produced the technologies behind Apple's Siri. His Command Post of the Future (CPoF) program was later adopted by the US Army as their Command and Control system for use during the Iraq and Afghanistan conflicts. Between DARPA tours, David served in senior positions at Facebook AI, Palo Alto Research Center, Vulcan Inc, Cycorp and co-founded SET Corp. Eric Vorm, PhD, is a cognitive systems engineer and serves as the Deputy Director for the Laboratory for Autonomous Systems Research at the US Naval Research Laboratory in Washington, DC. Dr Vorm led the evaluation team for the DARPA Explainable AI program, and led the development of the first comprehensive criteria for the evaluation of explanations generated by machine learning. Dr Vorm's research focuses on the design of intelligent systems to achieve ideal human-machine teaming, with special emphasis on the role of transparency and explainability in supporting appropriate trust, safety, and reliability in high-risk, time-sensitive operational domains. Jennifer Yunyan Wang, PhD, is a computational neuroscientist with a special focus on AI. As Systems, Engineering and technical Assistance (SETA) contractor to DARPA, she provided technical support and expertise to several programs including XAI, L2M, GARD, and AIE RED. After finishing postdoctoral fellowships in experimental neuroscience at Johns Hopkins University and the Food and Drug Administration, Jennifer joined Quantitative Scientific Solutions in 2018 as a consultant for government R&D and think tanks including IARPA and Center for Security and Emerging Technology. Matt Turek, PhD, joined DARPA's Information Innovation Office (I2O) as a program manager in July 2018 and took over as program manager of the XAI program in 2019. His portfolio also includes the Media Forensics (MediFor), Semantic Forensics (SemaFor), and Machine Common Sense (MCS) programs, as well as the Reverse Engineering of Deceptions (RED) AI Exploration. His research interests include computer vision, machine learning, artificial intelligence, and their application to problems with significant societal impact. Prior to his position at DARPA, Turek led a team at Kitware Inc developing computer vision technologies including large scale behavior recognition and modeling, object detection and tracking, activity recognition, normalcy modeling, and anomaly detection. Open