Explainable Artificial Intelligence (XAI)David Gunning
Information Innovation Office (I2O)
Defense Advanced Research Projects Agency (DARPA)

Approved for public release: distribution unlimited.

The Need for Explainable AI
User

AI System
http://listverse.com/©2007–2017 Listverse Ltd

Transportation

https://www.re-work.co/© 2017 RE-WORK X LTD

Finance

http://explainthatstuff.com• We are entering a new
age of AI applications• Machine learning is thecore technology• Machine learningmodels are opaque,non-intuitive, anddifficult for people tounderstand

http://listverse.com/©2007–2017 Listverse Ltd

Security

http://blog.soliant.com/©2004-2017 Soliant Health

Medicine

http://www.bbc.com/© 2017 BBC
Credit: Getty Images

Legalhttps://futureoflife.org/©FLI - Future of Life Institute

Military

Approved for public release: distribution unlimited.

• Why did you do that?
• Why not something else?
• When do you succeed?
• When do you fail?
• When can I trust you?
• How do I correct anerror?
2

Deep Learning Neural Networks

Training Data

Input
(unlabeled image)Low-levelfeatures to highlevel features

Neurons
respond tosimple shapes
Neurons respondto more complex
structures
Neurons respondto highly complex,
abstract concepts

1st Layer

2nd Layer

nth Layer

3What are we trying to do?
Today©Spin South WestThis is a cat(p = .93)

Learning
Process©University Of Toronto

Training
Data

Learned
Function

OutputUser

Tomorrow©University Of Toronto

Training
Data

This is a cat:New
Learning
Process•It has fur, whiskers,
and claws.
•It has this feature:

Explainable
Model

Explanation
Interface

Approved for public release: distribution unlimited.

User• Why did you do that?
• Why not something else?
• When do you succeed?
• When do you fail?
• When can I trust you?
• How do I correct an error?

• I understand why• I understand why not• I know when you’ll
succeed• I know when you’ll fail• I know when to trust you• I know why you erred
4• XAI will create a suite of machinelearning techniques that
• Produce more explainable models,
while maintaining a high level oflearning performance• Enable human users to understand,appropriately trust, and effectively
manage the emerging generation of
AI systems

Learning Performance

Goal: Performance and Explainability

Tomorrow
Today

Explainability

Approved for public release: distribution unlimited.

5

Challenge Problems
Learn a model

Data
Analytics
Classification
Learning Task

Explain decisions

Two trucks performing aloading activity

Perform a task

Explainable Explanation Recommend
ModelInterface Explanation©Getty Images©Air Force Research Lab

Multimedia DataClassifies items ofinterest in large data set

Autonomy

Explains why/why not
for recommended
items

Analyst decides which
items to report, pursue

Explainable Explanation Actions
Model
Interface Explanation

Reinforcement
Learning Task
©ArduPikot.org©US Army

ArduPilot & SITL Simulation

Learns decision
policies for missions

Explains behavior in anafter-action review

Operator decides
which future tasks todelegate

Approved for public release: distribution unlimited.

An analyst islooking for items
of interest in
massivemultimedia datasets
An operator is
directingautonomous
systems toaccomplish aseries of missions

6

Measuring Explanation Effectiveness
Explanation Framework

User Satisfaction• Clarity of the explanation (user rating)• Utility of the explanation (user rating)

Task

Mental Model

Recommendation,
Decision or
ActionExplainable Explanation
ModelInterface
XAI System
The system takes
input from thecurrent task andmakes arecommendation,
decision, or action

Explanation
The systemprovides an
explanation to theuser that justifies its
recommendation,
decision, or action

Decision
The usermakes adecisionbased onthe
explanation

Approved for public release: distribution unlimited.

• Understanding individual decisions• Understanding the overall model• Strength/weakness assessment• ‘What will it do’ prediction• ‘How do I intervene’ prediction

Task Performance• Does the explanation improve theuser’s decision, task performance?

Trust Assessment• Appropriate future use and trust

Correctability (Extra Credit)• Identifying errors• Correcting errors
7Learning Techniques (today)
Neural Nets
Deep
Learning

Graphical
Models
Bayesian
Belief Nets
SRL

Statistical
Models

AOGs
SVMs

CRFs

HBNs
MLNs

Markov
ModelsEnsemble
Methods
RandomForests

Decision
Trees

Approved for public release: distribution unlimited.

Learning Performance

Developing an Explainable Model

Today

Explainability

8

XAI Goal
Create a suite of machinelearning techniques that
produce more
explainable models, while
maintaining a high levelof learning performance

Learning Techniques (today)
Neural Nets
Deep
Learning

Modified deep learning techniques
to learn explainable features

Bayesian
Belief Nets
SRL

Statistical
ModelsAOGs
SVMs

Deep Explanation

Graphical
Models

CRFs

HBNs
MLNs

Markov
ModelsEnsemble
Methods
RandomForests

Decision
Trees

Interpretable Models

Techniques to learn more structured,interpretable, causal models
Approved for public release: distribution unlimited.

Learning Performance

Developing an Explainable Model

Tomorrow
Today

Explainability

Model Induction

Techniques to infer an explainablemodel from any model as a black box
9

XAI Developers and Technical Approaches
New
Learning
Process©University Of Toronto

Analytics

Autonomy

BothCP

Performer

Explainable
Model

Explainable Model

Explanation
Interface

Explanation Interface

UC Berkeley

Deep Learning

Reflexive and Rational

Charles River

Causal Modeling

Narrative Generation

UCLA

Pattern Theory+

3-level Explanation

Oregon State

Adaptive Programs

Acceptance Testing

PARC

Cognitive ModelingInteractive Training

CMU

Explainable RL (XRL)XRL Interaction

SRI International

Deep Learning

Show and Tell Explanation

Raytheon BBN

Deep Learning

Argumentation and Pedagogy

UT Dallas

Probabilistic Logic

Decision Diagrams

Texas A&M

Mimic Learning

Interactive Visualization

Rutgers

Model Induction

Bayesian Teaching

Approved for public release: distribution unlimited.

IHMC
Psychological Modelsof Explanation

10

Approaches to Deep Explanation
Modular Networks

Attention Mechanisms
Input Image

Salience Map

Learn to Explain

Feature Identification

CNN

RNN

This is a Downy Woodpecker because it is a black andwhite bird with a red spot on its crown.
Approved for public release: distribution unlimited.

11

Deeply Explainable Artificial Intelligence
UC Berkeley, Boston U., U. Amsterdam, Kitware
Explainable Model
Explanation Interface
Challenge Problem
Deep Learning

Reflexive and Rational

Autonomy• Post-hoc explanations bytraining additional DL
models• Explicit introspective
explanations (Neural
Module Networks)• Reinforcement Learning• Informative rollouts• Explicit modular agent• Reflexive explanations(arise from the model)• Rational explanations(come from reasoning
about user’s beliefs)• Evaluation criteria• Vehicle control (BDD-X,
CARLA)• Strategy games(StarCraft II)• Human interpretability• Predictive behavior• Appropriate trust

Data Analytics• Visual QA and filteringtasks (VQA-X, ACT-X,
xView, DiDeMo, etc.)• PI: Trevor Darrell (UC Berkeley)• Pieter Abbeel (UC Berkeley) • Dan Klein (UC Berkeley)• Anthony Hoogs (Kitware)• Tom Griffiths (UC Berkeley) • John Canny (UC Berkeley)• Kate Saenko (Boston U.)• Anca Dragan (UC Berkeley)• Zeynep Akata (U.
Amsterdam)
Approved for public release: distribution unlimited.

12

High Fidelity Visual Salience Model
UC Berkeley, Boston U., U. Amsterdam, Kitware
Probability of the ‘goldfish’class predicted by the
network vs. fraction ofremoved pixels
P[goldfish]

goldfishsalience

black
box

probe

RISE

Pixels Removed

black
box

goldfish

Salience isestimated byprobing the blackbox with randomly
masked images

Input Image

RISE

Grad-CAM

LIME

Petsiuk, Das and Saenko. RISE: Randomized Input Sampling for Explanation of Black-box Models, 2018
Approved for public release: distribution unlimited.

13

Textual justification with grounding and pragmatics
Given the multi-modal explanation generated by the model, do you think the system will answer correctly?
Question: Does thiselephant have tusks?

“because there are no bones
sticking out from its mouth”

Question: Is this aprofessional sporting event?

Incorrect! The system answered “no”
when the ground-truth answer is “yes”“because the players are
wearing official jerseys”

Correct! The system answered “yes”
when the ground-truth answer is “yes”

Explanation
EffectivenessAttention for
Explanation Used?

Accuracy of Users
Judgement

Without explanation (existing SOTA)

No

57.5%

UCB Model on descriptions

Yes

66.5%

UCB Model without attention

No

61.5%

UCB Model

Yes

70.0%

Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal
Explanations: Justifying Decisions and Pointing to the Evidence, 2018
Approved for public release: distribution unlimited.

14

Causal Grounded Driving
acceleration +courseInput image stream

Vehicle controller
Predicts vehicle control
commands with a visual
attention model

Attention heat map

control outputs(steering angle)

Explanation generator
Attention-based video-to-text
model to produce textual
justifications

Clustering analysis

descriptions +explanations

Visual saliency detection
and causality check

Kim and Canny, in ICCV, 2017
Kim, Rohrbach, Darrell, Canny, andAkata, in NIPS Interpretable ML
Symposium, 2017
Approved for public release: distribution unlimited.

15

CAMEL: Causal Models to Explain LearningCharles River Analytics (CRA), U. Mass, Brown
Explainable Model

Explanation Interface

Challenge Problem

Causal Modeling

Narrative Generation

Autonomy• Experiment with thelearned model (as agrey box) to learn an
explainable, causal,
probabilistic
programming model• Interactive visualization
based on the generationof temporal, spatial
narratives from thecausal, probabilistic
models• Atari• Starcraft

Data Analytics• Pedestrian Detection(INRIA)• Activity Recognition(ActivityNet)• PI: James Tittle (CRA)• Jeff Druce (CRA)• Avi Pfeffer (CRA)• David Jensen (U. Mass)• Michael Littman (Brown
U.)• James Niehaus (CRA)• Emilie Roth (Roth Cognitive Engineering)• Joe Gorman(CRA)• James Tittle (CRA)

Approved for public release: distribution unlimited.

16

Causal Models to Explain Learning (CAMEL)Charles River Analytics, U. Mass, Brown
Generate causal explanations of ML operation and present them to the user as intuitive narratives in aninteractive, easy-to-use interface grounded in cognitive engineering theories
Explanatory
Explanatory
Concepts
Concepts

Test
Test Data
Data

MachineMachineLearning
Learning
TechniqueTechnique

CausalCausalModel
Model
Template
Template

LearnedLearned
SystemSystemCausalCausalModel
Model
Learner
Learner

Explanatory
Explanatory
Dictionary
Dictionary
LearnedLearned
SystemSystemOutputOutputCausalCausalModel
Model

TRAINING
TRAINING

Explanation
Explanation
Interface
Interface
Explanatory
Explanatory
Concepts
Concepts

Narratives
Narratives

MachineMachineLearning
Learning
TechniqueTechniqueParameterization
Parameterization

Interactions
Interactions

CausalCausalProbabilistic
Probabilistic
Programming
Programming
Framework
Framework

Operational
Operational Data
Data

Operator
Operator

Narrative
Narrative
Generator
Generator

EXPLAINING
EXPLAINING
Approved for public release: distribution unlimited.

17

Learning and Communicating Explainable Representations
UCLA, Oregon State, Michigan State
Explainable Model

Explanation Interface

Challenge Problem

Pattern Theory+

3-Level Explanation

Autonomy• Concept compositions• Causal and counterfactual
reasoning• Utility explanations

Interpretable representations• STC-AOG: spatial, temporal,
and causal models• STC-PG: scene and event
interpretations in analytics• STC-PG+: task plans inautonomy

Explanation representations:• X-AOG: explanation model• X-PG: explanatory parsegraph as dialogue• X-Utility: priority and loss
for explanations

Theory of mind representations• User’s beliefs• User’s mental model of agent• Robot executing daily tasks
in physics-realistic VR
platform• Autonomous vehicle driving(GTA5 game engine)

Data Analytics• Network of video cameras
for scene understanding
and event analysis• PI: Song-Chun Zhu (UCLA)• Ying Nian Wu (UCLA)• Sinisa Todorovic (OSU)• Joyce Chai (Michigan State)

Approved for public release: distribution unlimited.

18

Learning and Communicating Explainable Representations
UCLA, Oregon State, Michigan State
System Architecture

Performer
Outputs interpretable
representations in a spatial,
temporal, and causal parsegraph (STC-PG)

Explainer
Outputs an explanatory
parse graph (X-PG) in a
dialogue process

STC Parse Graph

An attributed parse graph for a runningperson. Each node has 3 computing channels:• !: grounding the node on DNN features;• #: bottom-up;• $: top-down.
An explanation is represented as parse graph
X-pg

Approved for public release: distribution unlimited.

19

xACT: Explanation-Informed Acceptance Testing of
Deep Adaptive Programs
Oregon State University
Explainable Model

Explanation Interface

Challenge Problem

Adaptive Programs

Acceptance Testing

Autonomy• Explainable Deep
Adaptive Programs(xDAPs) – a new
combination of
Adaptive Programs,
Deep Learning, andexplainability• Provides a visual andNatural Language
explanation interface
for acceptance testingby test pilots based on
Information Foraging
Theory• Real-time Strategy
Games based on
custom designed gameengine designed tosupport explanation• Starcraft II• PI: Alan Fern (OSU)• Tom Dietterich (OSU)• Fuxin Li (OSU)• Prasad Tadepalli (OSU)• Weng-Keen Wong (OSU)• Margaret Burnett (OSU)• Martin Erwig (OSU)• Liang Huang (OSU)

Approved for public release: distribution unlimited.

20

xACT: Explanation-Informed Acceptance Testing of
Deep Adaptive Programs
Oregon State University
Decision Nets Explanation Nets
Explainable
Deep Adaptive
Program..
.

Explanation
Features..
.

Trained AI
Agent

AI System
Developer

Explanation
Translator

DomainOntology

Is it doing the rightthing, for the rightreasons?

Offline Learning
Reinforcement
Learning

Simulation
Manager

Explanation
User Interface

Explanation
Learning

SME

Interactive
NamingTest Pilot
Simulation
Environment

Launch simulations.
Explore rationale/explanations
for agent’s decisions.

Approved for public release: distribution unlimited.

21

COGLE: Common Ground Learning and Explanation
PARC, CMU, U. Edinburgh, U. Michigan, USMA, IHMC
Explainable Model

Explanation Interface

Challenge Problem

Cognitive Model

Interactive Training

Autonomy

3-layer architecture• Learning Layer (DNNs)• Cognitive Layer (ACT-R
Cognitive Model)• Explanation Layer(HCI)• Interactive visualizationof states, actions,policies, and values• Module for test pilots torefine and train thesystem• MAVSim wrapper over
ArduPilot simulation
environment• Value of Explanation
framework formeasuring explanation
effectiveness• Christian Lebiere (CMU)• John Anderson (CMU)• Robert Thomson (USMA)• Michael Youngblood(PARC)• PI: Mark Stefik (PARC)• Honglak Lee (U. Michigan)• SubramanianRamamoorthy (U.
Edinburgh)

Approved for public release: distribution unlimited.

22

COGLE: Common Ground Learning and Explanation
PARC, CMU, U. Edinburgh, U. Michigan, USMA, IHMC
Layered Cognitive Architecture
to Partition Explanation And
Learning

ACT-R

Hybrid Deep RL Agent/ ACT-R Architecture
Conv 1

Conv 2
Value
Prediction
ActionType

ScreenFeatures

Observation

Non-pixel
Information
FeatureConcatenation
Map
FeaturesApproved for public release: distribution unlimited.

Spatial Action
Policy23

XRL: Explainable Reinforcement Learning
Carnegie Mellon University
Explainable Model

Explanation Interface

Challenge Problem

Explainable RL (XRL)XRL Interaction

Autonomy• Create a new scientificdiscipline for Explainable
Reinforcement Learningwith work on newalgorithms andrepresentations• Interactive
explanations ofdynamic systems• Human-machineinteraction to improve
performance• Open AI Gym• Autonomy in theelectrical grid
• Mobile service robots• Self-improving
educational software• PI: Zico Kolter (CMU)• Geoff Gordon (CMU)• Pradeep Ravikumar (CMU)

Approved for public release: distribution unlimited.

24

XRL:
Explainable
Reinforcement
Learning
Aggrtate
a

.

.

if

mg

Ve

backupis

greedyas

goodenough

that

case

in

RC

an

.

.

it's

that'sbut

ago ,enough

to

ok

1

do

-

step

Carnegie Mellon University.

Create a new discipline of explainable RL to enablefdynamic human-machineinteraction and adaptation
for maximum team performanceisensili

e

n

-~

r

e

n

c

Sens

e

,

Sensor inputs

Image
segmentation.

M
EEEE.FI#eEeEEEEiiII*?.
of
%

.TO#Y

Imia

man

-

Model-based
Learn dynamics modelof environment, plan
actions in model, andexecute in real system
Model-free
Directly learn value and/or
policy for the environment

IN

Rigid bodysimulation

observations
rewards

Improve model learning/ creation
for RL agents to capture benefitsof model-based approach
For any type of RL approach,provide an explanation of why anagent acted in a certain way

iLQR-based
MPC

Control

Approved for public release: distribution unlimited.

Differentiable Physics - Applies implicit
differentiation to solutions of LCP toanalytically derive a backpropagation updateof next state with respect to previous state,control, and model parameters
25

DARE: Deep Attention-based Representations for ExplanationSRI International, U. Toronto, UCSD, U. Guelph
Explainable Model

Explanation Interface

Challenge Problem

Deep Learning

Show & Tell Explanation

Data Analytics

Multiple deep learningtechniques• Attention-based
mechanisms• Compositional NMNs
• GANs• DNN visualization• Query evidence that
explains DNN decisions• Generate natural
language justifications• VQA• Visual Gnome• Flickr30• MovieQA• PIs: Giedrius Burachas (SRI), Mohamed Amer (SRI)• Xiao Lin (SRI)• Ryan Villamil (SRI)• Dejan Jovanovic (SRI)• Avi Ziskind (SRI)• Michael Wessel (SRI)• Richard R. Zemel (U. Toronto)• Sanja Fidler (U. Toronto)• David Duvenaud (U. Toronto)• Graham Taylor (U. Guelph)

Approved for public release: distribution unlimited.

• Jürgen Schulze (UCSD)

26

DARE: Deep Attention-based Representations for ExplanationSRI International, U. Toronto, UCSD, U. GuelphInterpretable, Scene Graph-based VQA System with Active Attention• Generate “show-and-tell” explanations
with justifications of decisions
accompanied by visualizations of input
data used to generate inferences• Scene and Situation Graphs, inferredfrom images and videos, support richmultimodal data analytics andexplanations• Scene Graphs guide attentionalscanning for interpretable analytics

Approved for public release: distribution unlimited.

27

EQUAS: Explainable QUestion Answering System
Raytheon BBN, Georgia Tech, UT Austin, MIT
Explainable Model

Explanation Interface

Challenge Problem

Deep Learning

Argumentation Theory

Data Analytics• Semantic labelling of
DNN neurons• DNN audit trail
construction• Gradient-weighted
Class Activation
Mapping• Comprehensive
strategy based onargumentation theory• NL generation• DNN visualization• VQA for images andvideo• PI: William Ferguson (Raytheon BBN)
• Antonio Torralba (MIT)• Ray Mooney (UT Austin)• Devi Parikh (Georgia Tech)• Dhruv Batra (Georgia Tech)

Approved for public release: distribution unlimited.

28

EQUAS: Explainable QUestion Answering System
Raytheon BBN, Georgia Tech, UT Austin, MIT
Improve the interpretability of units using a new
conditioning method to retrain the network tointentionally include concept detectors

(2)

(3)

(1)
1) Pick units from standard vocabulary2) Train top part of net
3) Use top to train bottom

New ensemble heat map
explanation

Approved for public release: distribution unlimited.

29

Tractable Probabilistic Logic Models
UT Dallas, UCLA, Texas A&M, Indian Institute of Technology
Explainable Model

Explanation Interface

Challenge Problem

Probabilistic Logic

Decision Diagrams

Data Analytics• Tractable Probabilistic
Logic Models (TPLMs)– an important class of(non-deep learning)interpretable models• Enables users to
explore and correct
the underlying model
as well as add
background knowledge• Infer activities in
multimodal data (video
and text)• Wetlab (biology) andTACoS (cooking)datasets• PI: Vibhav Gogate (UT Dallas)• Adnan Darwiche (UCLA)• Eric Ragan (Texas A&M)
• Guy Van Den Broeck (UCLA) • Parag Singla (IIT-Delhi)• Nicholas Ruozzi (UT Dallas)

Approved for public release: distribution unlimited.

30

Tractable Probabilistic Logic Models
UT Dallas, UCLA, Texas A&M, Indian Institute of Technology
Use interpretable and tractable models based on well-founded principles from logic and probability theory

Tractable
Probabilistic Logic
Model (TPLM)

Find all videos in which a person peels oranges(explanations are captions generated by TPLMs)

Person using his hands topeel oranges. I can see theorange skin

Person using his hands. I
can see the orange skinand skinless orange

Person using his hands to peel oranges. I can see theorange skin on the table and peeled oranges
Approved for public release: distribution unlimited.

31

Transforming Deep Learning to Harness theInterpretability of Shallow Models
Texas A&M, Washington State
Explainable Model

Explanation Interface

Challenge Problem

Mimic Learning

Interactive Visualization

Data Analytics• Mimic learning
framework combines DL
models for prediction
and shallow models for
explanations• Interpretable learningalgorithms extract
knowledge from DNNs
for relevant explanations• Interactive visualization
over multiple views,
using heat maps andtopic modeling clusters toshow predictive features• Multiple tasks using datafrom Twitter, Facebook,
ImageNet, and news
websites• PI: Xia Hu (Texas A&M)
• Shuiwang Ji (Wash. State)• Eric Ragan (Texas A&M)

Approved for public release: distribution unlimited.

32

Transforming Deep Learning to Harness theInterpretability of Shallow Models
Texas A&M, Washington State
Develop an end-to-end interpretable deep learning infrastructure with image and text datasets

Forest Model

Approved for public release: distribution unlimited.

33

Model Explanation by Optimal Selection of Teaching Examples
Rutgers University
Explainable Model

Explanation Interface

Challenge Problem

Model Induction

Bayesian Teaching

Data Analytics• Select the optimaltraining examples to
explain model
decisions based on
Bayesian Teaching• Example-based
explanation of• Full model• User-selected substructure• User submitted
examples• Image processing• Text corpora• VQA• Movie events• PI: Patrick Shafto (Rutgers)• Scott Cheng-Hsin Yang (Rutgers)

Approved for public release: distribution unlimited.

34Model Explanation by Optimal Selection of Examples
Rutgers University
Extend Bayesian teaching to enable automatic explanation by selecting the subset of data that are mostrepresentative of the model’s generative process
Good and bad examples for teaching a category(illustrates model strengths and weaknesses)

Approved for public release: distribution unlimited.

35

XAI Program Schedule
CY2017FY2017APR MAY JUNJUL AUG SEP OCT NOV DEC JANCY2018CY2019FY2018FY2019FEB MAR APR MAY JUNJUL AUG SEP OCT NOV DEC JANFEB MAR APR MAY JUN

Explainable
Learning
Systems

Develop and Demonstrate Explainable
Learning Systems

FEB MAR APR MAY

Analyze ResultsandAccept Toolkits

Eval Refine and Test Explainable Eval Refine and Test Explainable Eval
1
Learning Systems
2
Learning Systems
3

Deliver
SoftwareLibraries andPrep for Eval 2

Summarize Current Psychological Develop Computational Model ofTheories of Explanation
Explanation

Progress ReportJUL AUG SEP OCT NOV DEC JAN

Eval
3

Prep for Eval Analyze
Eval 1
1 Results

Define Evaluation Framework

Kickoff

FEB MAR APR MAY JUN

PHASE 2Government Evaluations

Evaluator

Meetings

CY2021
FY2021

FY2020JUL AUG SEP OCT NOV DEC JAN

PHASE 1
Technology Demonstrations

PsychologicalModels of
Explanation

CY2020Tech Demos

Eval Analyze
2 Results

Prep for Eval 3

Refine and Test
Computational Model of Explanation

Eval 1 Results

Eval 2 Results

Deliver
Computational
Model Software
Final

Phase 1 Evaluations
Approved for public release: distribution unlimited.

36

Evaluation Framework
Explanation Process & Measures

Experimental Conditions
Without Explanation - The explainable
learning system is used to perform a task
without providing an explanation to the user

With Explanation - The explainable learning
system is used to perform a task andgenerates explanations for every
recommendation or decision it makes, andevery action it takes

Partial Explanation - The explainable

learning system is used to perform a task andgenerates only partial or ablated explanations(to assess various explanation features)

Control - A baseline state-of-the-art non-explainable system is used to perform a task

Approved for public release: distribution unlimited.

37

www.darpa.mil

Approved for public release: distribution unlimited.

38