Broad Agency Announcement
Explainable Artificial Intelligence (XAI)DARPA-BAA-16-53
August 10, 2016

Defense Advanced Research Projects Agency
Information Innovation Office
675 North Randolph Street
Arlington, VA 22203-2114

Table of Contents
PART I: OVERVIEW INFORMATION .......................................................4I.

Funding Opportunity Description ....................................................................................................5A.

Introduction .................................................................................................................................5

B.

Program Scope .............................................................................................................................6

C.

Challenge Problems and Evaluation ..........................................................................................10

D.

Technical Areas ..........................................................................................................................14

E.

Schedule/Milestones .................................................................................................................16

F.

Deliverables................................................................................................................................17G.

Government-furnished Property/Equipment/Information......................................................17

H.

Intellectual Property ..................................................................................................................17

I.

References..................................................................................................................................18

II.

Award Information ........................................................................................................................20
A.

III.
A.

Awards.......................................................................................................................................20
Eligibility Information ....................................................................................................................23
Eligible Applicants....................................................................................................................23

B. Procurement Integrity, Standards of Conduct, Ethical Considerations, and Organizational
Conflicts of Interest ............................................................................................................................23
C.
IV.

Cost Sharing/Matching ............................................................................................................24
Application and Submission Information ......................................................................................25

A.

Address to Request Application Package ...............................................................................25

B.

Content and Form of Application Submission .......................................................................25

C.

Submission Dates and Times ...................................................................................................35

D.

Funding Restrictions ................................................................................................................36

E.

Other Submission Requirements.............................................................................................36

V.

Application Review Information ....................................................................................................40
A.

Evaluation Criteria ...................................................................................................................40

B.

Review and Selection Process ..................................................................................................40

VI.

Award Administration Information ...............................................................................................42A.

Selection Notices .......................................................................................................................42

B.

Administrative and National Policy Requirements ...............................................................42C.

Reporting...................................................................................................................................49

VII.

Agency Contacts .........................................................................................................................50

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)2

VIII.

Other Information ......................................................................................................................51

A.

Frequently Asked Questions (FAQs) ......................................................................................51

B.

Proposers Day ...........................................................................................................................51

C.

Submission Checklist................................................................................................................51

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)3

PART I: OVERVIEW INFORMATIONFederal Agency Name: Defense Advanced Research Projects Agency (DARPA),
Information Innovation Office (I2O)



Funding Opportunity Title: Explainable Artificial Intelligence (XAI)

Announcement Type: Initial Announcement

Funding Opportunity Number: DARPA-BAA-16-53



Catalog of Federal Domestic Assistance Numbers (CFDA): 12.910 Research andTechnology Development



Dates
o Posting Date: August 10, 2016
o Proposers Day: August 11, 2016
o Abstract Due Date: September 1, 2016, 12:00 noon (ET)o Proposal Due Date: November 1, 2016, 12:00 noon (ET)o BAA Closing Date: November 1, 2016, 12:00 noon (ET)

Anticipated Individual Awards: Multiple awards are anticipated.

Types of Instruments that May be Awarded: Procurement Contracts, Cooperative
Agreements, or Other Transactions (OTs). No grants will be awarded under thissolicitation.

Agency Contactso Technical POC: David Gunning, Program Manager, DARPA/I2O
o BAA Email: XAI@darpa.milo BAA Mailing Address:
DARPA/I2O
ATTN: DARPA-BAA-16-53
675 North Randolph Street
Arlington, VA 22203-2114
o I2O Solicitation Website: http://www.darpa.mil/work-with-us/opportunities

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

4

PART II: FULL TEXT OF ANNOUNCEMENT
I.

Funding Opportunity Description
DARPA is soliciting innovative research proposals in the areas of machine learning and humancomputer interaction. The goal of Explainable Artificial Intelligence (XAI) is to create a suite of
new or modified machine learning techniques that produce explainable models that, whencombined with effective explanation techniques, enable end users to understand, appropriately
trust, and effectively manage the emerging generation of Artificial Intelligence (AI) systems.
Proposed research should investigate innovative approaches that enable revolutionary advancesin science, or systems. Specifically excluded is research that primarily results in evolutionary
improvements to the existing state of practice.
This Broad Agency Announcement (BAA) is being issued, and any resultant selection will be
made, using procedures under Federal Acquisition Regulation (FAR) 35.016. Any negotiations
and/or awards will use procedures under FAR 15.4 or 32 CFR 22 for grants and cooperative
agreements. Proposals received as a result of this BAA shall be evaluated in accordance withevaluation criteria specified herein through a scientific review process.
DARPA BAAs are posted on the Federal Business Opportunities (FBO) website(https://www.fbo.gov/) and, as applicable, the Grants.gov website (http://www.grants.gov/).
The following information is for those wishing to respond to this BAA.
A. Introduction
Dramatic success in machine learning has led to an explosion of new AI capabilities. Continued
advances promise to produce autonomous systems that perceive, learn, decide, and act on theirown. These systems offer tremendous benefits, but their effectiveness will be limited by themachine’s inability to explain its decisions and actions to human users. This issue is especially
important for the Department of Defense (DoD), which is facing challenges that demand thedevelopment of more intelligent, autonomous, and symbiotic systems. Explainable AI will beessential if users are to understand, appropriately trust, and effectively manage this incoming
generation of artificially intelligent partners.
The problem of explainability is, to some extent, the result of AI’s success. In the early days ofAI, the predominant reasoning methods were logical and symbolic. These early systems
reasoned by performing some form of logical inference on (somewhat) human readable symbols.
Early systems could generate a trace of their inference steps, which then became the basis for
explanation. As a result, there was significant work on how to make these systems explainable(Shortliffe & Buchanan, 1975; Swartout, Paris, & Moore, 1991; Johnson, 1994; Lacave & Díez,
2002; Van Lent, Fisher, & Mancuso, 2004).
Yet these early systems were much less effective; they proved too expensive to build and were
too brittle against the complexities of the real world. Success came as researchers developed
new techniques that employed machine learning to construct models of the world in their own
internal representations. These new techniques include support vector machines, random forests,
probabilistic graphical models, reinforcement learning, and deep learning neural networks.
Although these more opaque models are more effective, they are less explainable.
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)5

DARPA is interested in creating technology to make this new generation of AI systems
explainable. Because the most critical and most opaque components are based on machinelearning, XAI is focusing on the development of explainable machine learning techniques. By
creating new machine learning methods to produce more explainable models and combining
them with explanation techniques, XAI aims to help users understand, appropriately trust, andeffectively manage the emerging generation of AI systems.
B. Program Scope
The target of XAI is an end user who depends on decisions, recommendations, or actions
produced by an AI system, and therefore needs to understand the rationale for the system’s
decisions. For example, an intelligence analyst who receives recommendations from a big data
analytics algorithm needs to understand why the algorithm has recommended certain activity forfurther investigation. Similarly, a test operator of a newly developed autonomous system willneed to understand why the system makes its decisions so that he/she can decide how to use it in
future missions. Figure 1 illustrates the XAI concept—to provide end users with an explanationof individual decisions, enable users to understand the system’s overall strengths andweaknesses, convey an understanding of how the system will behave in the future, and perhaps
how to correct the system’s mistakes.

Figure 1: XAI ConceptThis concept presents three related research and development challenges: (1) how to produce
more explainable models; (2) how to design the explanation interface; and (3) how to understandthe psychological requirements for effective explanations. For the first development challenge,
the XAI program envisions developing a range of new or modified machine learning techniques
to produce more explainable models. For the second development challenge, the program
anticipates integrating state-of-the-art human-computer interaction (HCI) techniques (e.g.,visualization, language understanding, language generation, and dialog management) with new
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

6principles, strategies, and techniques to generate effective explanations. For the third
development challenge, XAI plans to summarize, extend, and apply current psychologicaltheories of explanation to assist both the XAI developers and evaluator. The program will be
structured to ask for multiple development teams (under Technical Area 1) to address the firstand second challenges by developing an explainable learning system that contains both an
explainable model and an explanation interface, and one team (under Technical Area 2) toaddress the third challenge.
B.1. Explainable Models
The XAI program would like to improve explainability while maintaining a high level oflearning performance for a range of machine learning techniques. There is an inherent tension
between machine learning performance (predictive accuracy) and explainability; often thehighest performing methods (e.g., deep learning) are the least explainable, and the mostexplainable (e.g., decision trees) are less accurate. The program plans to fund a variety ofmachine learning techniques to provide future developers with a range of design options
covering the performance versus explainability trade space.
Explainable models might be created by learning to associate explanatory semantic information
with features of the model, by learning simpler models that are easier to explain, by learningricher models that contain more explanatory content, or by inferring approximate models solely
for the purpose of explanation. Recent research offers examples of promising directions, eventhough none of these examples provides a complete solution, nor are these considered the only
possible solutions. Here are three possible strategies to illustrate the range, but not the entire
space, of approaches that may be pursued:


Deep Explanation: Develop modified or hybrid deep learning techniques that learn more
explainable features, explainable representations, or explanation generation facilities.
There are a number of design choices that affect deep learning: the selection of training
data, initial conditions, architectural layers, loss functions, regularization, optimization
techniques, and training sequences. These design choices might be engineered to
produce more explainable representations. Recent work has made several small steps
towards explainability. Researchers have used deconvolutional networks to visualize thelayers of convolutional networks (Zeiler & Fergus, 2014). There are a variety oftechniques for associating semantic concepts with nodes of a deep network, such as thoseused to identify items in a visual scene and recount multimedia events (Yu, Liu, Cheng,
Divakaran, & Sawhney, 2012; Gan, Wang, Yang, Yeung, & Hauptmann, 2015).
Moreover, it might be possible to extend the approaches used to generate image captions(LeCun, Bengio, & Hinton, 2015) to train a second deep network to generate
explanations without explicitly identifying the sematic features of the original network(Hendricks et al., 2016). Because deep learning is so prominent, it is important for the
XAI program to pursue several approaches to make these systems more explainable.

Interpretable Models: Develop alternative machine learning techniques that learn morestructured, interpretable, or causal models. There is a wide variety of techniques that
may be pursued: learning simpler, more compact models, such as Bayesian Rule Lists(Letham, Rudin, McCormick, & Madigan, 2015); learning richer, more conceptual,
generative models, such as Bayesian Program Learning (Lake, Salakhutdinov, &Tenenbaum, 2015); learning models of causal relationships (Maier, Taylor, Oktay, &Jensen, 2010); using stochastic grammars to learn more interpretable structure (Brendel

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)7

& Todorovic, 2011; Park, Nie, & Zhu 2016); or many others. Because these techniques
are inherently more explainable (even if their performance is less than optimal), it isimportant for the XAI program to also pursue promising techniques for learninginterpretable models.


Model Induction: Develop techniques that would experiment with any given machinelearning model—as a black box—to infer an approximate, explainable model. Ribeiro,
Singh, & Guestrin (2016) demonstrate an example of such a model-agnostic explanation
system that infers explanations by observing and analyzing the input-output behavior of
the original black-box model. Moreover, it may be possible to develop richer modelagnostic techniques by using abduction, reasoning, and story generation to “rationalize”
plausible explanations of the system’s reasoning. Because these techniques could apply
to almost any machine learning application, it is important for the XAI program to also
pursue promising techniques for model induction.

B.2. Explanation Interface
DARPA anticipates the design and development of effective interfaces by integrating state-ofthe-art HCI techniques with new strategies for presenting understandable explanations. Kulesza,
Burnett, Wong, & Stumpf (2015) provide an example of the development and evaluation of abasic explanation interface. The work followed a complete development strategy that included
identifying principles of explainability, developing a prototype interface from those principles,and evaluating the effectiveness of the explanations provided by the prototype. It was basic in
the sense that the system explained a very simple machine learning model, a naïve Bayesian textclassifier. The XAI program is seeking developments that go beyond this work by explaining
more complex learning systems, by devising additional explanation strategies, and by applying
more sophisticated HCI techniques where appropriate.
The design of an effective explanation interface should consider a range of HCI and cognitive
science techniques. This would certainly include basic user interface design, but XAI isinterested in user interaction in the broadest sense. The design of the interface should consider
the user’s cognitive tasks and mental model of the system being explained. It should consider
possible principles, such as the ones mentioned in Kulesza, Burnett, Wong, & Stumpf (2015),
and possible strategies, such as explanation by example or analogy. It should consider theappropriate use of visualization and natural language understanding. Furthermore, it should
include an explanation dialog to allow the user to interact with and drill down into the
explanation. XAI is interested in a broad range of HCI and cognitive science techniques that willproduce the most effective explanations.
Moreover, DARPA believes it is important to develop the explanation interface in conjunction
with explainable models as a part of an integrated system. Breakthroughs are more likely tocome from the right combination of machine learning and HCI. The XAI program encourages a
dialog between these two research communities and believes that effective XAI systems will
depend on significant interaction and contribution from both research groups. The program will
be asking for proposals that jointly develop explainable models and explanation interfaces in an
integrated explainable learning system.
XAI emphasizes the development of new/modified machine learning techniques to produce
explainable models. However, it is also possible for a well-designed explanation interface towork with existing machine learning techniques to make those models more explainable.
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)8

Developers are encouraged to design and implement explanation interfaces that cover as wide a
range of machine learning techniques as possible, old and new.
B.3. Psychology of Explanation
In addition to the application of psychology to the design of the explanation interface, theprogram is interested in work focused primarily on psychological theories of explanation. For
example, there is relevant psychological research on the structure and function of explanation(Lombrozo, 2006) and its role in learning and reasoning (Lombrozo, 2012). XAI in interested inhaving this work summarized and consolidated to inform both the development of the
explainable learning systems by the developers and the definition of explanation effectivenessmeasures by the evaluator. XAI is particularly interested in the development of computationalmodels of these theories that could be used to understand, model, and predict explanation
effectiveness.
B.4. Emphasis and Scope of XAI Research
The XAI program is interested in new technology at the intersection of machine learning, HCI,
and the psychology of explanation (Figure 2). XAI is not interested in research unrelated to thespecific problem of explainable AI. For example, designing an effective explanation dialog may
involve aspects of user modeling, personalization, and theory of mind, but XAI is not interested
in research on these topics for the sake of that research alone. Similarly, there are many
interesting research problems related to interactive machine learning and visual analytics, but
XAI is most interested in research directly related to the problem of explaining machine learningmodels to end users.

Figure 2: XAI Emphasis

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)9

C. Challenge Problems and Evaluation
C.1. Overview
The XAI program will focus the development of explainable learning systems on addressing
challenge problems in two areas: (1) machine learning problems to classify events of interest inheterogeneous, multimedia data, and (2) machine learning problems to construct decision
policies for a simulated autonomous system. These two challenge problem areas, illustrated in
Figure 3, were chosen to represent the intersection of two important machine learning
approaches (classification and reinforcement learning) and two important operational problem
areas for DoD (intelligence analysis and autonomous systems).

Figure 3: XAI Challenge Problem Areas

The government will provide an evaluator and is not soliciting proposals for the evaluation role.
The program is structured in two phases: Phase 1, Technology Demonstrations (18 months); andPhase 2, Comparative Evaluations (30 months). During Phase 1, developers will demonstrate
their technology against their own test problems. During Phase 2, developers will test theirtechnology against common problems defined by the government evaluator.
Proposals for TA1 and TA2 should reflect a four-year base program effort and a nominal startdate of May 1, 2017. Each proposal may address either TA1 or TA2, but not both.
Organizations may submit separate proposals to TA1 and TA2 and could possibly receive
awards under both.
Developers should propose their own test problems, within one or both of two general areas
listed above, to work on during Phase 1 of the program. During Phase 2 of the program,developers will be asked to demonstrate their XAI systems against a set of common test
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

10

problems defined by the XAI government evaluator. During Phase 1, the XAI government
evaluator will work with XAI developers to define a set of common test problems, in each ofthese challenge problem areas, to be used in Phase 2. The exact number and nature of the Phase
2 problems will depend on the technical approaches and test problems proposed by the selected
XAI performers.
Proposers should suggest creative and compelling test problems that would be the mostproductive drivers of XAI research and development. The program seeks test problems that are
sufficiently general and compelling to be useful for multiple XAI approaches in order to avoid
having a unique, tailored problem for each research project.
Below, we describe the motivation and desired characteristics of these problems to inform
proposer’s selection of test problems for Phase 1.
C.2. Data Analysis
The choice of a data analytics challenge is motivated by a common problem in intelligence
analysis—analysts are presented with decisions and recommendations from big data analytics
algorithms and must decide which items to pursue with more resources or which items to reportas supporting evidence in their analysis. These algorithms often produce false alarms that theanalyst must prune, and are subject to model drift that the analyst must detect. Furthermore,these algorithms often recommend items that the analyst must spend time assessing to determine
if the evidence supports or contradicts hypotheses they are evaluating. Effective explanations
would greatly help with all of these tasks.
Ideally, test problems in this area will emphasize machine learning to classify items, events, orpatterns of interest in heterogeneous, multimedia data, which would include structured or semistructured data, in addition to images and video. XAI is interested in problems that involve bothvideo as well as other types of data, and that require meaningful explanations that are notobvious in the video alone. Although the main problem would involve supervised classification,
unsupervised techniques might be very useful to organize the data into meaningful categories for
explanation.
A variety of candidate data sets and open test problems exist, including those provided by the
National Institute of Standards and Technology (NIST), the Knowledge Discovery and Data
Mining (KDD) conference, Kaggle, and other published data sets and test problems. Proposers
interested in this problem area should describe their version of the challenge problem to include
the data sets, training data, classification function to be learned, types of explanation to beprovided, and user decisions to be supported.
C.3. Autonomy
The choice of an autonomy challenge is motivated by the question of how to effectively manage
the incoming generation of artificially intelligent partners. For example, in the future DoD is
likely to develop, test, and use a variety of autonomous systems. There will be test pilots oroperators who will conduct operational tests and evaluations to determine how and when to use
these systems in future missions. Operational conditions often vary from the original design
specifications. There will be opportunities to use these systems for unanticipated mission
scenarios. Test operators will need explanations to effectively make those determinations.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)11

Ideally, test problems in this area will emphasize the use of reinforcement learning to learn
sequential decision policies for the autonomous agent. However, because the decision
machinery is likely to include additional planning, decision, or control modules, DARPA
anticipates that these explanations may cover those reasoning modules, as well as decision
policies learned through reinforcement learning. Moreover, the program is interested in
explaining decisions that would be meaningful to the end user (e.g., the test operator).
Reinforcement learning might also be used to learn low-level motor control, as well as higherlevel decisions. XAI is most interested in explanations of higher-level decisions that would be
relevant to the end user and the missions he/she needs to manage.
DARPA is considering simulation environments such as the ArduPilot/Software in the Loop
(SITL) environment, Research Environment for Supervising Control of Heterogeneous
Unmanned Vehicles (RESCHU), and those available from the OpenAI Gym for reinforcement
learning. Proposers interested in this problem area should describe their version of the challenge
problem to include the simulation environment, the types of missions to be covered, the decision
policies and mission tasks to be learned, the type of explanations to be provided, and the user
decisions to be supported.
C.4. Expected Evaluation Sequence
The general evaluation sequence to be administered by the government evaluator for the Phase 2
problems, in both challenge problem areas should be as follows. Proposals for Phase 1 test
problems should use this sequence as a model, but may vary to provide a more effective
development progression.
1. XAI developers are presented with a problem domain that includes training data or asimulated training environment and a description of the tasks that might be included in
the final user test. The specific problems presented should vary to avoid overengineering and to provide a progression of increasing difficulty.
2. XAI developers are asked to apply their machine learning technique to learn an
explainable model to accomplish those tasks. The evaluator may require some light
instrumentation of this process to estimate the person hours and other resources required
to learn the model. The program understands that some human involvement may beneeded during the learning process, but encourages developers to avoid depending on ahuman-intensive knowledge engineering effort.
3. The resulting explainable model is combined with the XAI developer’s explanationinterface to construct an explainable system.
4. The explainable system delivers and explains decisions or actions from the learned model
while the user is performing related domain tasks.
5. The system’s decisions and explanations contribute (positively or negatively) to theuser’s performance of the domain related tasks.
6. The evaluator measures the learning performance (e.g., predictive accuracy) of the modelto accomplish the target tasks and the effectiveness of the explanations in helping the user
understand and manage the AI system.
7. The evaluator also conducts evaluations of existing machine learning techniques, with abasic user interface, to establish baseline measurements for learning performance andexplanation effectiveness against the target challenge problem.
Measuring explanation effectiveness is a research challenge in its own right. XAI developers
should propose their own measurement scheme to use in conjunction with their proposed test
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

12

problems for Phase 1. During that first phase, the XAI government evaluator will work with the
XAI developers to define more detailed test problems and evaluation metrics to be used in thesecond phase of the program. Figure 4 shows the framework and initial ideas for measuring
explanation effectiveness. The list of measures roughly represents increasing difficulty andsophistication. The most basic capability would explain a model’s individual decisions. A moreadvanced capability would explain the strengths and weaknesses of the overall model. The mostsophisticated capability would enable the user to identify and correct mistakes. It may not be
possible to achieve all of these desired capabilities, especially the capability to correct mistakes,
which would be beyond the reach of most current machine learning techniques. This list
represents the ideal capabilities of an explanation system.
In addition, the evaluator will define a range of evaluation techniques from fully automated tohuman-intensive. XAI envisions three levels of evaluation: (1) fully automated techniques thatdevelopers can use immediately and often to evaluate the system’s explanations; (2) Amazon
Mechanical Turk (AMT) techniques that developers can use less frequently (perhaps monthly) toobtain some human evaluation of their system’s explanations; and (3) human-in-the-loop
techniques that developers can use annually for more intensive human evaluations while
performing more complex tasks.

Figure 4: Evaluation Framework for Measuring Explanation Effectiveness

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)13

D. Technical Areas
Figure 5 shows the overall structure of the program. XAI shall be organized into two technicalareas (TAs): Explainable Learners and Psychological Models of Explanation. As mentioned in
Section C.1. Overview, DARPA is not soliciting proposals for the evaluator role. Proposers may
propose to either or both of the technical areas

Figure 5: XAI Program Structure

Technical Area 1 (TA1) (Explainable Learners):
Multiple TA1 teams will develop prototype explainable learning systems. Each prototype should
include both a machine learning technique that produces an explainable model and an
explanation interface that will deliver useful explanations to an end user.
TA1 proposers should include the following topics among those discussed in their proposal.
This list will be evaluated as part of the Overall Science and Technical Merit of TA1 proposals:
Challenge Problem Area: Select one or both of the challenge problems areas, data
analytics or autonomy, to work on in both Phase 1 and Phase 2. Describe theproposed test problem(s) to work on in Phase 1, including the data sets or simulation
environment, training data, functions to be learned, types of explanation to beprovided, and user decisions to be supported.



Explainable Model: Describe the proposed machine learning approach for learning
more explainable models. Describe the scope of the learning problems that would be
handled by this approach. Describe the learning process. What training data isrequired? What human involvement is required? Propose how to measure learning
performance and explainability of the learned models? The proposed machinelearning techniques may follow any of the three strategies described earlier (deepexplanation, interpretable models, or model induction), any combination of the three,or something completely different. It is acceptable to propose a range of machinelearning techniques, especially if this widens the scope of machine learning problems
covered by the proposer’s approach.

Explanation Interface: Describe the approach for building the explanation interface.
What are the initial design ideas for this interface? What principles or design

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

14

strategies will be employed? Describe examples of the explanations the system will
generate. Describe the range of the machine learning models that could be explainedby this approach. It is understood that the explanation interface may be developed
through iterative prototyping with users. Nevertheless, proposers should describe
their initial ideas for designing the first iteration of the explanation interface.


Development Progression: Describe the development sequence proposers intend tofollow. What capabilities would proposers expect to develop during each phaseof the
program? It is understood that this progression is likely to change, but proposers
should describe the capabilities of the final system they hope to produce by the end of
the program and the interim capabilities they expect to develop along the way.



Test and Evaluation Plan: How will proposers test and evaluate their work in the firstphase of the program? How will proposers measure learning performance andexplanation effectiveness? If the test problem requires some government provided
data or infrastructure, specify that in the proposal.

Technical Area 2 (TA2) (Psychological Model of Explanation):
DARPA plans to fund at least one TA2 performer to provide expertise in the psychology of
explanation. This performer will summarize current psychological theories of explanation,
relevant to a XAI system, in an information package to assist the TA1 developers and thegovernment evaluator. This summary may include recommendations on theories the performer
believes are most useful for XAI. The TA2 performer will then develop a computational modelof explanation from those theories that could be used to predict explanation effectiveness of XAI
systems. The TA2 performer would also provide advice and consultation to the TA1 developers
on the requirements for an effective explanation and to the XAI government evaluator regarding
the development of metrics to measure explanation effectiveness.
TA2 proposers should include the following topics among those discussed in their proposal.
This list will be evaluated as part of the Overall Science and Technical Merit of TA2 proposals:


Theories of Explanation: Summarize the current psychological theories of explanationand describe how to further develop and refine this summary. Describe how this work
will inform the development of the TA1 XAI systems. Describe how this work will
inform the definition of the evaluation framework for measuring explanation
effectiveness by the XAI evaluator.

Computational Model: Describe how to develop and implement a computational modelof explanation from the available theories. Describe how to model elements of thetheory. Identify predictions that might be tested with the computational model. Explain
how to test and refine the model.

Model Validation: Describe how to validate the computational model of the theory andhow to test the model against the TA1 evaluation results in Phase 2 of the XAI program.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)15

E. Schedule/Milestones
Figure 6 below shows the XAI program schedule and milestones.
2017
APR MAY JUN

JUL

AUG

SEP

2018
OCT NOV DEC

JAN

FEB

MAR APR MAY JUN

JUL

2019
AUG

SEP

OCT NOV DEC

JAN

FEB

MAR APR MAY JUN

JUL

2020
AUG

PHASE 1: Technology Demonstrations

Evaluator

TA 1

Develop & Demonstrate Explainable Models(against proposed problems)

TA 2

Summarize Current Psychological
Theories of Explanation

OCT NOV DEC

JAN

FEB

MAR APR MAY JUN

JUL

2021
AUG

SEP

OCT NOV DEC

JAN

FEB

MAR APR MAY

PHASE 2: Comparative Evaluations

Prep for
Eval 1

Define Evaluation Framework

SEP

Eval Analyze
1
Results

Eval
1

Prep for Eval 2

Refine & Test Explainable
Learners(against common problems)

Develop Computational Model of
Explanation

Eval Analyze
2
Results

Eval
2

Prep for Eval 3

Refine & Test Explainable
Learners(against common problems)

Refine & Test
Computational Model

Eval
3

Analysze Results&
Accept Toolkits

Eval
3

Deliver Software
Toolkits
Deliver
Computational
Model

Meetings
KickOff

Progress Report

Tech Demos

Eval 1 Results

Eval 2 Results

Final

Figure 6: XAI Schedule

Technical Area 1 Milestones:

Demonstrate the explainable learners against problems proposed by the developers) at thesecond and third program review meetings (month 6 and month 12)
Demonstrate the explainable learners against problems proposed by the developers during
Eval 1 (months 14-16) and demonstrate a minimal interface to evaluator’s infrastructure.
Demonstrate the effectiveness of the explainable learners in a series of evaluations (Eval
2 and 3) against the common problems and metrics defined in collaboration with theevaluation team (every 12 months during Phase 2)
Deliver software libraries and toolkits (at the end of Phase 2)

Technical Area 2 Milestones:






Deliver an interim draft summary of current psychological theories of explanation (after 6
months during Phase 1)
Deliver a final written summary of current psychological theories of explanation (after 12months, during Phase 1)
Deliver a computational model of explanation (after 24 months, during Phase 2)
Demonstrate the ability of the computational model to predict the performance of
explanations generated by the TA1 systems (each year during Phase 2)
Deliver the computational model software (at the end of Phase 2)

Meetings and travel:

There will be six program reviews, three in the first year, and one in each of thefollowing three years. The first of these will be held in or near DARPA in Arlington,
VA. For planning purposes, assume the remaining meetings will alternate between aWest Coast and East Coast location.
All performers should expect to host a site visit from DARPA at least once a year.
All performers should budget for two additional trips to the Washington D.C. area in thelast two years of the program for possible demonstrations and transition discussions.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)16

F. Deliverables
Performers are responsible for providing, at a minimum, the following deliverables, asapplicable:
 Slide Presentations – Annotated slide presentations will be submitted within two weeks
after program kick‐off meeting and after each review.
 XAI Web Page – DARPA intends to have a teaming website for performers, maintainedby the XAI evaluator. Periodic updates for the XAI program web page (maintained by
the XAI evaluator) with links to information describing the proposer’s work on the
program (presentations, papers, video demos, and evaluation results).
 Monthly Coordination Reports – A monthly technical coordination report describing
progress made, resources expended, and any issues requiring the attention of theGovernment team will be provided within 10 days after the end of each month.
 Monthly expenditure reports (for cost reimbursable awards) and uploading of requireddeliverables to the DARPA Technology Financial Information Management System(TFIMS), or its replacement system, are required by all program performers.
 Software – All computer software developed or delivered under the program must be
delivered as source and as object code (as consistent with the Intellectual Property rights
asserted in the proposal). Include the source listings and source code for the target
computer systems. DARPA encourages delivered software under this effort to becompletely maintainable and modifiable with no reliance on any non‐delivered computer
programs or documentation.
 Software Documentation – It should document source code, hardware description
language specifications, system diagrams, part numbers and other data necessary tomaintain and to produce copies of the software (as consistent with the Intellectual
Property rights asserted in the proposal).
 Final Technical Report – Due at project completion, this will concisely summarize theeffort conducted and provide any lessons learned during the development of thetechnology.
Proposals are not limited to these and may contain additional deliverables.
G. Government-furnished Property/Equipment/Information
DARPA and the evaluation team will manage a demonstration/evaluation infrastructure
described in Section C. Challenge Problems and Evaluation. During Phase 2 of the program, theXAI government evaluator will provide test data, problem definitions, evaluation infrastructure(including AMT, other human subjects, and IRB approvals), and data analysis of the evaluation
results. During Phase 1 of the program, the XAI government evaluator may provide assistanceas needed. Proposers should describe any assistance they expect to be provided by the XAI
government evaluator during Phase 1.
H. Intellectual Property
The program will emphasize creating and leveraging open source technology and architecture. A
key goal of the program is to establish an open, standards-based, multi-source, plug-and-playarchitecture that allows for interoperability and integration. This includes the ability to easily
add, remove, substitute, and modify software and hardware components. This will facilitate
rapid innovation by providing a base for future users or developers of program technologies and
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

17

deliverables. Therefore, it is desired that all noncommercial software (including source code),
software documentation, hardware designs and documentation, and technical data generated by
the program be provided as deliverables to the Government, with a minimum of Government
Purpose Rights (GPR), as lesser rights may adversely impact the lifecycle costs of affected items,
components, or processes. Intellectual property rights asserted by proposers are strongly
encouraged to be aligned with open source regimes. See Section VI.B.1 for more details on
intellectual property.
I. References
Brendel, W., & Todorovic, S. (2011, November). Learning spatiotemporal graphs of humanactivities. In International Conference on Computer Vision (pp. 778-785). Barcelona, Spain:
IEEE Press Core, M. G., Lane, H. C., Van Lent, M., Gomboc, D., Solomon, S., & Rosenberg,
M. (2006, July). Building explainable artificial intelligence systems. In Proceedings of the
National Conference on Artificial Intelligence (Vol. 21, No. 2, p. 1766). Menlo Park, CA;
Cambridge, MA; London; AAAI Press; MIT Press; 1999.
Gan, C., Wang, N., Yang, Y., Yeung, D. Y., & Hauptmann, A. G. (2015). Devnet: A deep event
network for multimedia event detection and evidence recounting. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (pp. 2568-2577).
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016).
Generating Visual Explanations. arXiv preprint arXiv:1603.08507.
Johnson, W. L. (1994, October). Agents that Learn to Explain Themselves. In Proceedings of the
12th National Conference on Artificial Intelligence (pp. 1257-1263). Seattle, WA: AAAI
press..
Kulesza, T., Burnett, M., Wong, W. K., & Stumpf, S. (2015, March). Principles of explanatory
debugging to personalize interactive machine learning. In Proceedings of the 20th
International Conference on Intelligent User Interfaces (pp. 126-137). ACM.
Lacave, C., & Díez, F. J. (2002). A review of explanation methods for Bayesian networks. The
Knowledge Engineering Review, 17(02), 107-127.
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015). Human-level concept learning
through probabilistic program induction. Science, 350(6266), 1332-1338.
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.
Letham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015). Interpretable classifiers using
rules and Bayesian analysis: Building a better stroke prediction model. The Annals of Applied
Statistics, 9(3), 1350-1371.
Lombrozo, T. (2006). The structure and function of explanations. Trends in cognitive sciences,10(10), 464-470.
Lombrozo, T. (2012). Explanation and abductive inference. Oxford handbook of thinking andreasoning, 260-276.
Maier, M. E., Taylor, B. J., Oktay, H., & Jensen, D. (2010, July). Learning Causal Models ofRelational Domains. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial
Intelligence. Atlanta, GA: AAAI Press.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

18

Park, S., Nie, B. X., & Zhu, S. C. (2016). Attribute And-Or Grammar for Joint Parsing of Human
Attributes, Part and Pose. arXiv preprint arXiv:1605.02112.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "Why Should I Trust You?" Explaining thePredictions of Any Classifier. arXiv preprint arXiv:1602.04938.
Shortliffe, E. H., & Buchanan, B. G. (1975). A model of inexact reasoning in medicine.
Mathematical Biosciences, 23(3), 351-379.
Swartout, W., Paris, C., & Moore, J. (1991). Explanations in knowledge systems: Design for
explainable expert systems. IEEE Expert, 6(3), 58-64.
Van Lent, M., Fisher, W., & Mancuso, M. (2004, July). An explainable artificial intelligencesystem for small-unit tactical behavior. In Proceedings of the National Conference on
Artificial Intelligence (pp. 900-907). Menlo Park, CA; Cambridge, MA; London; AAAI
Press; MIT Press; 1999.
Yu, Q., Liu, J., Cheng, H., Divakaran, A., & Sawhney, H. (2012, October). Multimedia event
recounting with concept based representation. In Proceedings of the 20th ACM international
conference on Multimedia (pp. 1073-1076). ACM.
Zeiler, M. D., & Fergus, R. (2014, September). Visualizing and understanding convolutional
networks. In European Conference on Computer Vision (pp. 818-833). Springer International
Publishing.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

19

II.

Award Information
A. Awards
Multiple awards are anticipated. The level of funding for individual awards made under thissolicitation has not been predetermined and will depend on the quality of the proposals received
and the availability of funds. Awards will be made to proposers whose proposals are determined
to be the most advantageous to the Government, all factors considered, including the potential
contributions of the proposed work, overall funding strategy, and availability of funding. See
Section V for further information.
The Government reserves the right to:select for negotiation all, some, one, or none of the proposals received in response to thissolicitation;
make awards without discussions with proposers;
conduct discussions with proposers if it is later determined to be necessary;
segregate portions of resulting awards into pre-priced options;
accept proposals in their entirety or to select only portions of proposals for award;fund proposals in increments and/or with options for continued work at the end of one ormore phases;request additional documentation once the award instrument has been determined (e.g.,
representations and certifications); andremove proposers from award consideration should the parties fail to reach agreement onaward terms within a reasonable time or the proposer fails to provide requested
additional information in a timely manner.
Proposals selected for award negotiation may result in a procurement contract, cooperative
agreement, or Other Transaction (OT) depending upon the nature of the work proposed, therequired degree of interaction between parties, and other factors.
In all cases, the Government contracting officer shall have sole discretion to select award
instrument type and to negotiate all instrument terms and conditions with selectees. Proposers
are advised that regardless of the instrument type proposed, DARPA personnel, in consultation
with the Government contracting officer, may select other award instruments, as they deem
appropriate. DARPA will apply publication or other restrictions, as necessary, if it determinesthat the research resulting from the proposed effort will present a high likelihood of disclosing
performance characteristics of military systems or manufacturing technologies that are uniqueand critical to defense. Any award resulting from such a determination will include arequirement for DARPA permission before publishing any information or results on the
program. For more information on publication restrictions, see the section below on
Fundamental Research.
B. Fundamental Research
It is DoD policy that the publication of products of fundamental research will remain unrestricted
to the maximum extent possible. National Security Decision Directive (NSDD) 189 establishedthe national policy for controlling the flow of scientific, technical, and engineering information
produced in federally funded fundamental research at colleges, universities, and laboratories. The
Directive defines fundamental research as follows:
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)20

'Fundamental research' means basic and applied research in science and engineering, theresults of which ordinarily are published and shared broadly within the scientificcommunity, as distinguished from proprietary research and from industrial development,
design, production, and product utilization, the results of which ordinarily are restricted
for proprietary or national security reasons.
As of the date of publication of this BAA, the Government expects that program goals as
described herein may be met by proposers intending to perform fundamental research. The
Government does not anticipate applying publication restrictions of any kind to individualawards for fundamental research that may result from this BAA. Notwithstanding this statementof expectation, the Government is not prohibited from considering and selecting research
proposals that, while perhaps not qualifying as fundamental research under the foregoing
definition, still meet the BAA criteria for submissions. If proposals are selected for award thatoffer other than a fundamental research solution, the Government will either work with theproposer to modify the proposed statement of work to bring the research back into line with
fundamental research or else the proposer will agree to restrictions in order to receive an award.
Proposers should indicate in their proposal whether they believe the scope of the research
included in their proposal is fundamental or not. While proposers should clearly explain theintended results of their research, the Government shall have sole discretion to select award
instrument type and to negotiate all instrument terms and conditions with selectees. Appropriateclauses will be included in resultant awards for non-fundamental research to prescribe
publication requirements and other restrictions, as appropriate.
For certain research projects, it may be possible that although the research being performed by
the prime contractor is restricted research, a subawardee may be conducting fundamental
research. In those cases, it is the prime contractor’s responsibility to explain in its proposal why
its subawardee’s effort is fundamental research.
The following statement or similar provision will be incorporated into any resultant nonfundamental research procurement contract or other transaction:
There shall be no dissemination or publication, except within and between the contractor
and any subawardees, of information developed under this contract or contained in thereports to be furnished pursuant to this contract without prior written approval of
DARPA’s Public Release Center (DARPA/PRC). All technical reports will be given
proper review by appropriate authority to determine which Distribution Statement is to be
applied prior to the initial distribution of these reports by the contractor. With regard to
subawardee proposals for Fundamental Research, papers resulting from unclassified
fundamental research are exempt from prepublication controls and this review
requirement, pursuant to DoD Instruction 5230.27 dated October 6, 1987.
When submitting material for written approval for open publication, thecontractor/awardee must submit a request for public release to the DARPA/PRC andinclude the following information: (1) Document Information: document title, document
author, short plain-language description of technology discussed in the material (approx.
30 words), number of pages (or minutes of video) and document type (e.g., briefing,report, abstract, article, or paper); (2) Event Information: event type (conference,
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

21

principal investigator meeting, article or paper), event date, desired date for DARPA's
approval; (3) DARPA Sponsor: DARPA Program Manager, DARPA office, and contract
number; and (4) Contractor/Awardee's Information: POC name, email and phone. Allow
four weeks for processing; due dates under four weeks require a justification. Unusualelectronic file formats may require additional processing time. Requests may be sent
either via email to public_release_center@darpa.mil or by mail at 675 North Randolph
Street, Arlington VA 22203-2114, telephone (571) 218-4235. Refer to the following forlink for information about DARPA’s public release process: http://www.darpa.mil/workwith-us/contract-management/public-release.”

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

22

III.

Eligibility Information
A. Eligible Applicants
All responsible sources capable of satisfying the Government's needs may submit a proposal thatshall be considered by DARPA. Please note that FFRDC, government entity, and foreign
participation is welcomed, however, some limitations may apply. Please see below.
1. Federally Funded Research and Development Centers (FFRDCs) and Government
Entities
Federally Funded Research and Development Centers (FFRDCs) and Government entities(e.g., Government/National laboratories, military educational institutions, etc.) are subject toapplicable direct competition limitations and cannot propose to this BAA in any capacity
unless they meet the following conditions: (1) FFRDCs must clearly demonstrate that the
proposed work is not otherwise available from the private sector; and (2) FFRDCs must
provide a letter on official letterhead from their sponsoring organization citing the specificauthority establishing their eligibility to propose to Government solicitations and competewith industry, and their compliance with the associated FFRDC sponsor agreement’s terms
and conditions. This information is required for FFRDCs proposing to be prime contractors
or subawardees. Government entities must clearly demonstrate that the work is not otherwise
available from the private sector and provide written documentation citing the specificstatutory authority and contractual authority, if relevant, establishing their ability to propose to
Government solicitations. At the present time, DARPA does not consider 15 U.S.C. § 3710a
to be sufficient legal authority to show eligibility. While 10 U.S.C.§ 2539b may be theappropriate statutory starting point for some entities, specific supporting regulatory guidance,
together with evidence of agency approval, will still be required to fully establish eligibility.
DARPA will consider FFRDC and Government entity eligibility submissions on a case-bycase basis; however, the burden to prove eligibility for all team members rests solely with theproposer.
2. Foreign Participation
Non-U.S. organizations and/or individuals may participate to the extent that such participants
comply with any necessary nondisclosure agreements, security regulations, export control
laws, and other governing statutes applicable under the circumstances.
B. Procurement Integrity, Standards of Conduct, Ethical Considerations, andOrganizational Conflicts of Interest
Current federal employees are prohibited from participating in particular matters involving
conflicting financial, employment, and representational interests (18 U.S.C. §§ 203, 205, and
208). Once the proposals have been received, and prior to the start of proposal evaluations, theGovernment will assess potential conflicts of interest and will promptly notify the proposer if
any appear to exist. The Government assessment does NOT affect, offset, or mitigate theproposer’s responsibility to give full notice and planned mitigation for all potential
organizational conflicts, as discussed below.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)23

Without prior approval or a waiver from the DARPA Director, in accordance with FAR 9.503, a
contractor cannot simultaneously provide scientific, engineering, technical assistance (SETA) or
similar support and also be a technical performer. As part of the proposal submission, allmembers of the proposed team (prime proposers, proposed subawardees, and consultants) must
affirm whether they (their organizations and individual team members) are providing SETA orsimilar support to any DARPA technical office(s) through an active contract or subcontract. Allaffirmations must state which office(s) the proposer, subawardees, consultant, or individual
supports and identify the prime contract number(s). All facts relevant to the existence or
potential existence of organizational conflicts of interest (FAR 9.5) must be disclosed. The
disclosure must include a description of the action the proposer has taken or proposes to take to
avoid, neutralize, or mitigate such conflict. If in the sole opinion of the Government after fullconsideration of the circumstances, a proposal fails to fully disclose potential conflicts of interest
and/or any identified conflict situation cannot be effectively mitigated, the proposal will berejected without technical evaluation and withdrawn from further consideration for award.

If a prospective proposer believes a conflict of interest exists or may exist (whetherorganizational or otherwise) or has questions on what constitutes a conflict of interest, theproposer should send his/her contact information and a summary of the potential conflict via
email to the BAA email address before time and effort are expended in preparing a proposal andmitigation plan.
C. Cost Sharing/Matching
Cost sharing is not required; however, it will be carefully considered where there is an applicablestatutory condition relating to the selected funding instrument (e.g., OTs under the authority of
10 U.S.C. § 2371).

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

24

IV.

Application and Submission Information
A. Address to Request Application Package
This document contains all information required to submit a response to this solicitation. No
additional forms, kits, or other materials are needed except as referenced herein. No request for
proposal (RFP) or additional solicitation regarding this opportunity will be issued, nor isadditional information available except as provided at the Federal Business Opportunities
website (https://www.fbo.gov), the Grants.gov website (http://www.grants.gov/), or referenced
herein.
B. Content and Form of Application Submission
1. Abstracts
Proposers are highly encouraged to submit an abstract in advance of a proposal to minimize
effort and reduce the potential expense of preparing an out of scope proposal. The abstract
provides a synopsis of the proposed project.
DARPA will respond to abstracts with a statement as to whether DARPA is interested in theidea. If DARPA does not recommend the proposer submit a full proposal, DARPA will
provide feedback to the proposer regarding the rationale for this decision. Regardless of
DARPA’s response to an abstract, proposers may submit a full proposal. DARPA will review
all full proposals submitted using the published evaluation criteria and without regard to any
comments resulting from the review of an abstract.
Abstract Format: Abstracts shall not exceed a maximum of 7 pages including the coversheet and all figures, tables, and charts, 5 pages for technical approach, 1 for capabilities andmanagement plan. The page limit does not include a submission letter (optional).
All pages shall be formatted for printing on 8-1/2 by 11 inch paper with 1-inch margins andfont size not smaller than 12 point. Font sizes of 8 or 10 point may be used for figures,tables, and charts. Document files must be in .pdf, .odx, .doc, .docx, .xls, or .xlsx formats.
Submissions must be written in English.
Abstracts must include the following components:
 Cover Sheet (1 page): Provide the administrative and technical points of contact(name, address, phone, email, lead organization). Include the BAA number, title of the
proposed project, primary subcontractors, estimated cost, duration of the project, and thelabel “Abstract.” Technical Approach (5 pages): Outline and address all technical challenges inherent
in the approach and possible solutions for overcoming potential problems. Provide
appropriate specific milestones (quantitative, if possible) at intermediate stages of the
project to demonstrate progress.
 Capabilities/Management Plan (1 page): Provide a brief summary of expertise of theorganization or team, including subcontractors (if any) and key personnel. Identify a
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)25

principal investigator for the project and include a description of the team’s organization
including roles and responsibilities. Describe the organizational experience in this area,
existing intellectual property required to complete the project, and any specialized
facilities to be used as part of the project.
2. Proposals
Proposals consist of Volume 1: Technical and Management Proposal (including mandatory
Appendix A and optional Appendix B) and Volume 2: Cost Proposal.
All pages shall be formatted for printing on 8-1/2 by 11-inch paper with 1-inch margins,
single-line spacing, and a font size not smaller than 12 point. Font sizes of 8 or 10 point
may be used for figures, tables, and charts. Document files must be in .pdf, .odx, .doc,.docx, .xls, or .xlsx formats. Submissions must be written in English.
Proposals not meeting the format prescribed herein may not be reviewed.
a. Volume 1: Technical and Management Proposal
The maximum page count for Volume 1 is 31 pages, including all figures, tables and charts
but not including the cover sheet, table of contents or appendices. A submission letter is
optional and is not included in the page count. Appendix A does not count against the page
limit and is mandatory. Appendix B does not count against the page limit and is optional.
Additional information not explicitly called for here must not be submitted with theproposal, but may be included in the bibliography in Appendix B. Such materials will beconsidered for the reviewers’ convenience only and not evaluated as part of the proposal.
Volume 1 must include the following components:
i. Cover Sheet: Include the following information.













Label: “Proposal: Volume 1”
BAA number (DARPA-BAA-16-53)
Technical Area
Proposal title
Lead organization (prime contractor) name
Type of organization, selected from the following categories: Large Business,
Small Disadvantaged Business, Other Small Business, HBCU, MI, Other
Educational, or Other Nonprofit
Technical point of contact (POC) including name, mailing address, telephone,and email
Administrative POC including name, mailing address, telephone number, andemail addressAward instrument requested: procurement contract (specify type), cooperative
agreement or OT.1
Total amount of the proposed effort.
Place(s) and period(s) of performance
Other team member (subcontractors and consultants) information (for each,

1 Information on award instruments can be found at http://www.darpa.mil/work-with-us/contract-management.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

26







include Technical POC name, organization, type of organization, mailingaddress, telephone number, and email address)
Proposal validity period (minimum 120 days)
Data Universal Numbering System (DUNS) number2
Taxpayer identification number3
Commercial and Government Entity (CAGE) code4
Proposer’s reference number (if any)ii. Table of Contentsiii. Executive Summary: Provide a synopsis of the proposed project, including answers
to the following questions: What is the proposed work attempting to accomplish or do?
 How is it done today, and what are the limitations?
 Who or what will be affected and what will be the impact if the work is
successful?
 How much will it cost, and how long will it take?
The executive summary should include a description of the key technical challenges, aconcise review of the technologies proposed to overcome these challenges and achieve
the project’s goal, and a clear statement of the novelty and uniqueness of the proposed
work.
iv. Innovative Claims and Deliverables: Describe the innovative aspects of the projectin the context of existing capabilities and approaches, clearly delineating the uniquenessand benefits of this project in the context of the state of the art, alternative approaches,and other projects from the past and present. Describe how the proposed project isrevolutionary and how it significantly rises above the current state of the art.
Describe the deliverables associated with the proposed project and any plans tocommercialize the technology, transition it to a customer, or further the work. Discuss
the mitigation of any issues related to sustainment of the technology over its entire
lifecycle, assuming the technology transition plan is successful.
v. Technical Plan: Outline and address technical challenges inherent in the approachand possible solutions for overcoming potential problems. Demonstrate a deep
understanding of the technical challenges and present a credible (even if risky) plan toachieve the project’s goal. Discuss mitigation of technical risk. Provide appropriate
measurable milestones (quantitative if possible) at intermediate stages of the project todemonstrate progress, and a plan for achieving the milestones.
2 The DUNS number is used as the Government's contractor identification code for all procurement-related

activities. Go to http://fedgov.dnb.com/webform/index.jsp to request a DUNS number (may take at least one
business day). See Section VI.B.8 for further information.
3 See http://www.irs.gov/businesses/small/international/article/0,,id=96696,00.html for information on requesting aTIN. Note, requests may take from 1 business day to 1 month depending on the method (online, fax, mail).
4 A CAGE Code identifies companies doing or wishing to do business with the Federal Government. See Section
VI.B.8 for further information.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)27

vi. Management Plan: Provide a summary of expertise of the proposed team, including
any subcontractors/consultants and key personnel who will be executing the work.
Resumes count against the proposal page limit so proposers may wish to include them inAppendix B below. Identify a principal investigator (PI) for the project. Provide a clear
description of the team’s organization including an organization chart that includes, asapplicable, the relationship of team members; unique capabilities of team members; task
responsibilities of team members; teaming strategy among the team members; and key
personnel with the amount of effort to be expended by each person during the project.
Provide a detailed plan for coordination including explicit guidelines for interaction
among collaborators/subcontractors of the proposed project. Include risk management
approaches. Describe any formal teaming agreements that are required to execute thisproject. List Government-furnished materials or data assumed to be available.
vii. Personnel, Qualifications, and Commitments: List key personnel (no more than
one page per person), showing a concise summary of their qualifications, discussion ofprevious accomplishments, and work in this or closely related research areas. Indicate thelevel of effort in terms of hours to be expended by each person during each contract year
and other (current and proposed) major sources of support for them and/or commitmentsof their efforts. DARPA expects all key personnel associated with a proposal to makesubstantial time commitment to the proposed activity and the proposal will be evaluatedaccordingly. It is DARPA’s intention to put key personnel conditions into the awards, so
proposers should not propose personnel that are not anticipated to execute the award.
Include a table of key individual time commitments as follows:

Key
Individual

Project

Status(Current,
Pending,
Proposed)

Hours on Project (by FY)

2017

Individual
Name 1

Individual
Name 2

XAI

Proposed

Project 1

Current

Project 2

Pending

XAI

Proposed

Project 3

Current

2018

2019

2020

2021

viii. Capabilities: Describe organizational experience in relevant subject area(s),existing intellectual property, or specialized facilities. Discuss any work in closely
related research areas and previous accomplishments.
ix. Statement of Work (SOW): The SOW must provide a detailed task breakdown,
citing specific tasks and their connection to the interim milestones and metrics, asapplicable. Each year of the project should be separately defined. The SOW must notinclude proprietary information. For each defined task/subtask, provide:
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)28 A general description of the objective.
 A detailed description of the approach to be taken to accomplish each definedtask/subtask.
 Identification of the primary organization responsible for task execution(prime contractor, subcontractor(s), consultant(s)), by name.
 A measurable milestone, (e.g., a deliverable, demonstration, or otherevent/activity that marks task completion).
 A definition of all deliverables (e.g., data, reports, software) to be provided to
the Government in support of the proposed tasks/subtasks.
 Identify any tasks/subtasks (by the prime or subcontractor) that will beaccomplished at a university.
x. Schedule and Milestones: Provide a detailed schedule showing tasks (task name,duration, work breakdown structure element as applicable, performing organization),milestones, and the interrelationships among tasks. The task structure must be consistent
with that in the SOW. Measurable milestones should be clearly articulated and defined intime relative to the start of the project.
xi. Level of Effort Summary by Task: Provide a one-page table summarizing
estimated level of effort per task (in hours) broken out by senior, mid-level and juniorpersonnel, in the format shown below in Figure 2. Also include dollar-denominatedestimates of travel, materials and equipment. For this table, consider materials to include
the cost of any data sets or software licenses proposed. For convenience, an Excel
template is available for download along with the BAA.
Duration

Intensity

SOW Task(months)(hrs/mo)

Sr

Mid

Jr

Total

SubC

ConsltTotal

1.1.0 <Phase 1 Task 1 name>

7

135

240

680

24

944

-

200

1,144

1.1.1 <Subtask 1.1.1 name>

4

90

80

280

-

360

-

200

560

1.1.2 <Subtask 1.1.2 name>

3

195

160

400

24

584

-

-

584

1.2.0 <Phase 1 Task 2 name>

6

385

108

400

1,800

2,308

1,400

-

3,708

1.2.1 <Subtask 1.2.1 name>

3

656

48

320

1,600

1,968

600

-

2,568

1.2.2 <Subtask 1.2.2 name>

3

113

60

80

200

340

800

-

1,140

:

:

:

:

Phase 1 Total Hours
Phase 1 Costs First column is prime, second isLabor Hours

:

:

:

348

1,080

1,824

:

:

:

:

3,252

1,400

200

4,652

Travel $ 44,000 $ 12,000 $ 2,000

total subcontractor, third is total consultant, fourth is total Materials & Equipment $

8,000

$

-

$

-

$ 58,000$

8,000

2.1.0 <Phase 2 Task 1 name>

8

100

176

560

64

800

100

100

1,000

2.1.1 <Subtask 2.1.1 name>

7

51

96

240

24

360

100

100

560

2.1.2 <Subtask 2.1.2 name>

4

110

80

320

40

440

-

-

440

2.2.0 <Phase 2 Task 2 name>

6

417

180

520

1,800

2,500

1,240

-

3,740

2.2.1 <Subtask 2.2.1 name>

4

435

140

400

1,200

1,740

400

-

2,140

2.2.2 <Subtask 2.2.2 name>

4

190

40

:

:

:

:

Phase 2 Total Hours
Phase 2 Costs First column is prime, second is

120

600

:

:

:

356

1,080

1,864

760

1,600

:

:

:

3,300

1,340

100

4,640

704

2,160

3,688

4,000

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

$

6,552

-

$

2,740

300

$ 61,000
$

4,000
9,292

Travel $ 91,000 $ 24,000 $ 4,000 $ 119,000

total subcontractor, third is total consultant, fourth is total Materials & Equipment $ 12,000

DARPA-BAA-16-53

840

Travel $ 47,000 $ 12,000 $ 2,000

total subcontractor, third is total consultant, fourth is total Materials & Equipment $Project Total Hours
Total Project Costs First column is prime, second is

-

:

$

-

$

-

$ 12,00029

Figure 7: Example level-of-effort summary table. Numbers illustrate roll-ups and subtotals. The SubC
column captures all subcontractor hours and the Conslt column captures all consultant hours.

xii. Appendix A: This section is mandatory and must include all of the following
components. If a particular subsection is not applicable, state “NONE”.
(1). Team Member Identification: Provide a list of all team members including theprime, subcontractor(s), and consultant(s), as applicable. Identify specificallywhether any are a non-US organization or individual, FFRDC and/or Government
entity. Use the following format for this list:

Individual Name

Role
(Prime,Subcontractor or
Consultant)

Non-US?
Organization
Org.

Ind.

FFRDC
or
Govt?

(2). Government or FFRDC Team Member Proof of Eligibility to Propose: If
none of the team member organizations (prime or subcontractor) are a
Government entity or FFRDC, state “NONE”.
If any of the team member organizations are a Government entity or FFRDC,
provide documentation (per Section III.A.1) citing the specific authority that
establishes the applicable team member’s eligibility to propose to Government
solicitations to include: 1) statutory authority; 2) contractual authority; 3)
supporting regulatory guidance; and 4) evidence of agency approval for
applicable team member participation.
(3). Government or FFRDC Team Member Statement of Unique Capability: If
none of the team member organizations (prime or subcontractor) are a
Government entity or FFRDC, state “NONE”.
If any of the team member organizations are a Government entity or FFRDC,
provide a statement (per Section III.A.1) that demonstrates the work to be
performed by the Government entity or FFRDC team member is not otherwise
available from the private sector.
(4). Organizational Conflict of Interest Affirmations and Disclosure: If none ofthe proposed team members is currently providing SETA or similar support as
described in Section III.B, state “NONE”.
If any of the proposed team members (individual or organization) is currently
performing SETA or similar support, furnish the following information:

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)30

Prime Contract
NumberDARPA
Technical Officesupported

A description of the action the proposer has taken
or proposes to take to avoid, neutralize, or mitigate
the conflict(5). Intellectual Property (IP): If no IP restrictions are intended, state “NONE”.
The Government will assume unlimited rights to all IP not explicitly identified ashaving less than unlimited rights in the proposal.
For all technical data or computer software that will be furnished to theGovernment with other than unlimited rights, provide (per Section VI.B.1) a list
describing all proprietary claims to results, prototypes, deliverables or systems
supporting and/or necessary for the use of the research, results, prototypes
and/or deliverables. Provide documentation proving ownership or possession of
appropriate licensing rights to all patented inventions (or inventions for which apatent application has been filed) to be used for the proposed project. Use thefollowing format for these lists:
NONCOMMERCIAL
Technical Data and/or
Summary ofBasis for
Computer Software To Intended Use in Assertion
be Furnished With
the Conduct ofRestrictions
the Research(List)(Narrative)(List)(List)(Narrative)(List)

Asserted
RightsCategory

Name of Person
Asserting Restrictions(List)(List)(List)(List)

Asserted
RightsCategory

Name of Person
Asserting Restrictions(List)(List)(List)(List)COMMERCIAL
Technical Data and/or
Summary ofBasis for
Computer Software To Intended Use in Assertion
be Furnished With
the Conduct ofRestrictions
the Research(List)(Narrative)(List)(List)(Narrative)(List)(6). Human Subjects Research (HSR): If HSR is not a factor in the proposal, state“NONE”.
If the proposed work will involve human subjects, provide evidence of or a plan
for review by an institutional review board (IRB). For further information on thissubject, see Section VI.B.2.
(7). Animal Use: If animal use is not a factor in the proposal, state “NONE”.
If the proposed research will involve animal use, provide a brief description of theplan for Institutional Animal Care and Use Committee (IACUC) review andapproval. For further information on this subject, see Section VI.B.3.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)31

(8). Representations Regarding Unpaid Delinquent Tax Liability or a Felony
Conviction under Any Federal Law: Per Section VI.B.10, complete thefollowing statements.
(1) The proposer is [ ] is not [ ] a corporation that has any unpaid Federal tax
liability that has been assessed, for which all judicial and administrative remedies
have been exhausted or have lapsed, and that is not being paid in a timely manner
pursuant to an agreement with the authority responsible for collecting the tax
liability,(2) The proposer is [ ] is not [ ] a corporation that was convicted of a felonycriminal violation under a Federal law within the preceding 24 months.
(9). Cost Accounting Standards (CAS) Notices and Certification: Per Section
VI.B.11, any proposer who submits a proposal which, if accepted, will result in a
CAS-compliant contract, must include a Disclosure Statement as required by 48
CFR 9903.202. The disclosure forms may be found at
http://www.whitehouse.gov/omb/procurement_casb.
If this section is not applicable, state “NONE”.
xiii. Appendix B: If desired, include a brief bibliography to relevant papers, reports, or
resumes. Do not include technical papers. This section is optional, and the materials will
not be evaluated as part of the proposal review.
b. Volume 2 - Cost Proposal
This volume is mandatory and must include all the listed components. No page limit isspecified for this volume.
The cost proposal should include a working spreadsheet file (.xls or equivalent format) thatprovides formula traceability among all components of the cost proposal. The spreadsheet
file should be included as a separate component of the full proposal package. Costs must be
traceable between the prime and subcontractors/consultants, as well as between the cost
proposal and the SOW.
Pre-award costs will not be reimbursed unless a pre-award cost agreement is negotiatedprior to award.
i. Cover Sheet: Include the same information as the cover sheet for Volume 1, but withthe label “Proposal: Volume 2.”
ii. Cost Summary Tables: Provide a single-page summary table broken down by fiscal
year listing cost totals for labor, materials, other direct charges (ODCs), indirect costs(overhead, fringe, general and administrative (G&A)), and any proposed fee for the
project. Include costs for each task in each fiscal year of the project by prime and major
subcontractors, total cost and proposed cost share, if applicable. Provide a second tablecontaining the same information broken down by project phase.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)32

iii. Cost Details: For each task, provide the following cost details by month. Include
supporting documentation describing the method used to estimate costs. Identify any
cost sharing.
(1) Direct Labor: Provide labor categories, rates and hours. Justify rates by
providing examples of equivalent rates for equivalent talent, past commercial or
Government rates or Defense Contract Audit Agency (DCAA) approved rates.
(2) Indirect Costs: Identify all indirect cost rates (such as fringe benefits, labor
overhead, material overhead, G&A, etc.) and the basis for each.
(3) Materials: Provide an itemized list of all proposed materials, equipment,
and supplies for each year including quantities, unit prices, proposed vendors (ifknown), and the basis of estimate (e.g., quotes, prior purchases, catalog pricelists, etc.). For proposed equipment/information technology (as defined in FAR
2.101) purchases equal to or greater than $50,000, include a letter justifying thepurchase. Include any requests for Government-furnished equipment or
information with cost estimates (if applicable) and delivery dates.
(4) Travel: Provide a breakout of travel costs including the purpose andnumber of trips, origin and destination(s), duration, and travelers per trip.
(5) Subcontractor/Consultant Costs: Provide above info for each proposed
subcontractor/consultant. Subcontractor cost proposals must include
interdivisional work transfer agreements or similar arrangements. If the
proposer has conducted a cost or price analysis to determine reasonableness,
submit a copy of this along with the subcontractor proposal.
The proposer is responsible for the compilation and submission of all
subcontractor/consultant cost proposals. At a minimum, the submitted cost
volume must contain a copy of each subcontractor or consultant non-proprietary
cost proposal (i.e. cost proposals that do not contain proprietary pricing
information such as rates, factors, etc.) Proprietary subcontractor/consultant
cost proposals may be included as part of Volume 2. Proposal submissions will
not be considered complete unless the Government has received all
subcontractor/consultant cost proposals.
If proprietary subcontractor/consultant cost proposals are not included as part ofVolume 2, they may be emailed separately to XAI@darpa.mil. Email messages
must include “Subcontractor Cost Proposal” in the subject line and identify theprincipal investigator, prime proposer organization and proposal title in the
body of the message. Any proprietary subcontractor or consultant proposal
documentation which is not uploaded to BAAT as part of the proposer’s
submission or provided by separate email by the closing timeframe shall bemade immediately available to the Government, upon request, under separate
cover (i.e., mail, electronic/email, etc.), either by the proposer or by thesubcontractor/consultant organization.
Please note that a Rough Order of Magnitude (ROM), or similar budgetary
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)33

estimate, is not considered a fully qualified subcontract cost proposal
submission. Inclusion of a ROM, or similar budgetary estimate, or failure toprovide a subcontract proposal, will result in the full proposal being deemed
non-compliant.
(6) ODCs: Provide an itemized breakout and explanation of all otheranticipated direct costs.
iv. Proposals Requesting a Procurement Contract: Provide the following information
where applicable.
(1) Proposals for $750,000 or more: Provide “certified cost or pricing data”(as defined in FAR 2.101) or a request for exception in accordance with FAR
15.403.
(2) Proposals for $700,000 or more: Pursuant to Section 8(d) of the Small
Business Act (15 U.S.C. § 637(d)), it is Government policy to enable small
business and small disadvantaged business concerns to be considered fairly as
subcontractors to organizations performing work as prime contractors or
subcontractors under Government contracts, and to ensure that prime
contractors and subcontractors carry out this policy. In accordance with FAR
19.702(a)(1) and 19.702(b), prepare a subcontractor plan, if applicable. The
plan format is outlined in FAR 19.704.
(3) Proposers without an adequate cost accounting system: If requesting a
cost-type contract, provide the DCAA Pre-award Accounting System Adequacy
Checklist to facilitate DCAA’s completion of an SF 1408. Proposers without an
accounting system considered adequate for determining accurate costs mustcomplete an SF 1408 if a cost type contract is to be negotiated. To facilitate thisprocess, proposers should complete the SF 1408 found at
http://www.gsa.gov/portal/forms/download/115778 and submit the completed
form with the proposal. To complete the form, check the boxes on the second
page, then provide a narrative explanation of your accounting system tosupplement the checklist on page one.
v. Proposals Requesting an Other Transaction for Prototypes Agreement: Proposers
must indicate whether they qualify as a nontraditional Defense contractor5, have teamedwith a nontraditional Defense contractor, or are providing a one-third cost share for thiseffort. Provide information to support the claims. See 10 U.S.C. § 2371b.
Provide a detailed list of milestones including: milestone description, completion criteria,
due date, and payment/funding schedule (to include, if cost share is proposed, contractor
and Government share amounts). Milestones must relate directly to accomplishment oftechnical metrics as defined in the solicitation and/or the proposal. While agreement type
(fixed price or expenditure based) will be subject to negotiation, the use of fixed pricemilestones with a payment/funding schedule is preferred. Proprietary information must
5 For definitions and information on OT for prototype agreements see http://www.darpa.mil/work-with-us/contract-

management.
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)34

not be included as part of the milestones.
3. Proprietary and Classified Information
DARPA policy is to treat all submissions as source selection information (see FAR 2.101 and3.104) and to disclose the contents only for the purpose of evaluation. Restrictive notices
notwithstanding, during the evaluation process, submissions may be handled by support
contractors for administrative purposes and/or to assist with technical evaluation. All DARPA
support contractors performing this role are expressly prohibited from performing DARPAsponsored technical research and are bound by appropriate nondisclosure agreements.
a. Proprietary Information
Proposers are responsible for clearly identifying proprietary information. Submissions
containing proprietary information must have the cover page and each page containing suchinformation clearly marked.
b. Classified Information
Classified submissions (classified technical proposals or classified appendices to
unclassified proposals) WILL NOT be accepted under this solicitation.
If a determination is made that the award instrument may result in access to classified
information, a DD Form 254, “DoD Contract Security Classification Specification,” will be
issued by DARPA and attached as part of the award. A DD Form 254 will not be provided
to proposers at the time of submission. For reference, the DD Form 254 template isavailable at http://www.dtic.mil/whs/directives/forms/eforms/dd0254.pdf.
C. Submission Dates and Times
Proposers are warned that submission deadlines as outlined herein are strictly enforced. Note:
some proposal requirements may take from 1 business day to 1 month to complete. See theproposal checklist in Section VIII.C for further information.
When utilizing the DARPA BAA Submission Website, as described below in Section IV.E.1
below, a control number will be provided at the conclusion of the submission process. This
control number should be used in all further correspondence regarding your abstract/proposal
submission.
For proposal submissions requesting cooperative agreements, Section IV.E.1.c, you must request
your control number via email at XAI@darpa.mil. Please note that the control number will not
be issued until after the proposal due date and time.
Failure to comply with the submission procedures outlined herein may result in the submission
not being evaluated.
1. Abstracts
Abstracts must be submitted per the instructions outlined herein and received by DARPA nolater than September 1, 2016 at 12:00 noon (ET). Abstracts received after this date and time
may not be reviewed.
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)35

2. Proposals
The proposal package -- full proposal (Volume 1 and 2) and, as applicable, proprietary
subcontractor cost proposals, classified appendices to unclassified proposals -- must be
submitted per the instructions outlined herein and received by DARPA no later thanNovember 1, 2016, at 12:00 noon (ET). Proposal submissions received after this date andtime will not be reviewed.
Proposers are warned that submission deadlines as outlined herein are strictly enforced. DO
NOT WAIT UNTIL THE LAST MINUTE TO FINALIZE AND COMPLETE YOURSUBMISSION.
D. Funding Restrictions
Not applicable.
E. Other Submission Requirements
1. Unclassified Submission Instructions
Proposers must submit all parts of their submission package using the same method;submissions cannot be sent in part by one method and in part by another method nor should
duplicate submissions be sent by multiple methods. Email submissions will not be accepted.
a. Abstracts
DARPA/I2O will employ an electronic upload submission system (https://baa.darpa.mil/)for all UNCLASSIFIED abstract responses under this solicitation. Abstracts should not be
emailed and should not be submitted via Grants.gov. If the submission instructions are notfollowed, your abstract will not be reviewed.
First time users of the DARPA BAA Submission Website must complete a two-stepaccount creation process at https://baa.darpa.mil/. The first step consists of registering for
an Extranet account by going to the above URL and selecting the “Account Request” link.
Upon completion of the online form, proposers will receive two separate emails; one willcontain a user name and the second will provide a temporary password. Once both emails
have been received, proposers must go back to the submission website and log in using thatuser name and password. After accessing the Extranet, proposers must create a user
account for the DARPA BAA Submission Website by selecting the “Register Your
Organization” link at the top of the page. The DARPA BAA Submission Website will
display a list of solicitations open for submissions. Once a proposer’s user account iscreated, they may view instructions on uploading their abstract.
Proposers who already have an account on the DARPA BAA Submission Website may
simply log in at https://baa.darpa.mil/, select this solicitation from the list of open DARPA
solicitations and proceed with their abstract submission. Note: Proposers who have created
a DARPA BAA Submission Website account to submit to another DARPA Technical
Office’s solicitations do not need to create a new account to submit to this solicitation.
All submissions submitted electronically through DARPA's BAA website must be uploaded
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)36

as zip files (.zip or .zipx extension). The final zip file should contain only the files
requested herein and must not exceed 50 MB in size. Only one zip file will be accepted per
submission. Note: Submissions not uploaded as zip files will be rejected by DARPA.
Please note that all submissions MUST be finalized, meaning that no further editing will bepossible, when submitting through the DARPA BAA Submission Website in order for
DARPA to be able to review your submission. If a submission is not finalized, thesubmission will not be deemed acceptable and will not be reviewed.
Website technical support may be reached at Action@darpa.mil and is typically available
during regular business hours (9:00 AM – 5:00 PM ET, Monday-Friday). Questions
regarding submission contents, format, deadlines, etc. should be emailed to
XAI@darpa.mil.
Since abstract submitters may encounter heavy traffic on the web server, they should not
wait until the day abstracts are due to request an account and/or upload the submission.
b. Proposals Requesting a Procurement Contract or Other Transaction
DARPA/I2O will employ an electronic upload submission system (https://baa.darpa.mil/)for UNCLASSIFIED proposals requesting award of a procurement contract or Other
Transaction under this solicitation.
First time users of the DARPA BAA Submission Website must complete a two-stepaccount creation process at https://baa.darpa.mil/. The first step consists of registering for
an Extranet account by going to the above URL and selecting the “Account Request” link.
Upon completion of the online form, proposers will receive two separate emails; one willcontain a user name and the second will provide a temporary password. Once both emails
have been received, proposers must go back to the submission website and log in using thatuser name and password. After accessing the Extranet, proposers must create a user
account for the DARPA BAA Submission Website by selecting the “Register Your
Organization” link at the top of the page. The DARPA BAA Submission Website will
display a list of solicitations open for submissions. Once a proposer’s user account iscreated, they may view instructions on uploading their proposal.
Proposers who already have an account on the DARPA BAA Submission Website may
simply log in at https://baa.darpa.mil/, select this solicitation from the list of open DARPA
solicitations and proceed with their proposal submission. Note: Proposers who havecreated a DARPA BAA Submission Website account to submit to another DARPA
Technical Office’s solicitations do not need to create a new account to submit to thissolicitation.
All submissions submitted electronically through DARPA's BAA website must be uploaded
as zip files (.zip or .zipx extension). The final zip file should contain only the files
requested herein and must not exceed 50 MB in size. Only one zip file will be accepted per
submission. Note: Submissions not uploaded as zip files will be rejected by DARPA.
Please note that all submissions MUST be finalized, meaning that no further editing will bepossible, when submitting through the DARPA BAA Submission Website in order for
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)37

DARPA to be able to review your submission. If a submission is not finalized, thesubmission will not be deemed acceptable and will not be reviewed.
Website technical support may be reached at Action@darpa.mil and is typically available
during regular business hours (9:00 AM – 5:00 PM ET, Monday-Friday). Questions
regarding submission contents, format, deadlines, etc. should be emailed to
XAI@darpa.mil.
Since proposers may encounter heavy traffic on the web server, they should not wait until
the day proposals are due to request an account and/or upload the submission.
c. Proposals Requesting a Cooperative Agreement
Proposers requesting cooperative agreements may submit proposals through one of thefollowing methods: (1) hard copy mailed directly to DARPA; or (2) electronic upload per
the instructions at http://www.grants.gov/applicants/apply-for-grants.html. Cooperative
agreement proposals may not be submitted through any other means. If proposers intend to
use Grants.gov as their means of submission, then they must submit their entire proposal
through Grants.gov; applications cannot be submitted in part to Grants.gov and in part as a
hard-copy. Proposers using the Grants.gov do not submit paper proposals in addition to the
Grants.gov electronic submission.
Proposers choosing to mail hard copy proposals to DARPA must include one paper copy
and one electronic copy (e.g., CD/DVD) of the full proposal package.
Grants.gov requires proposers to complete a one-time registration process before a proposal
can be electronically submitted. If proposers have not previously registered, this process
can take between three business days and four weeks if all steps are not completed in atimely manner. See the Grants.gov user guides and checklists at
http://www.grants.gov/web/grants/applicants.html for further information.
Once Grants.gov has received an uploaded proposal submission, Grants.gov will send two
email messages to notify proposers that: (1) their submission has been received by
Grants.gov; and (2) the submission has been either validated or rejected by the system. It
may take up to two business days to receive these emails. If the proposal is rejected by
Grants.gov, it must be corrected and re-submitted before DARPA can retrieve it (assuming
the solicitation has not expired). If the proposal is validated, then the proposer has
successfully submitted their proposal and Grants.gov will notify DARPA. Once theproposal is retrieved by DARPA, Grants.gov will send a third email to notify the proposer.
The proposer will then receive an email from DARPA acknowledging receipt and providing
a control number. For more information on submitting proposals to Grants.gov, visit the
Grants.gov submissions page at: http://www.grants.gov/web/grants/applicants/apply-forgrants.html.
To avoid missing deadlines, proposers should submit their proposals to Grants.gov inadvance of the proposal due date, with sufficient time to complete the registration andsubmission processes, receive email notifications and correct errors, as applicable.
Technical support for the Grants.gov website may be reached at 1-800-518-4726 andDARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)38

support@grants.gov. Questions regarding submission contents, format, deadlines, etc.
should be emailed to XAI@darpa.mil.
2. Classified Submission Instructions
As a reminder, classified submissions (classified technical proposals or classified
appendices to unclassified proposals) WILL NOT be accepted under this solicitation.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

39

V.

Application Review Information
A. Evaluation Criteria
Proposals will be evaluated using the following criteria listed in descending order of importance:
Overall Scientific and Technical Merit; Potential Contribution and Relevance to the DARPA
Mission; and Cost Realism.
Overall Scientific and Technical Merit:
The proposed technical approach is feasible, achievable, complete and supported by aproposed technical team that has the expertise and experience to accomplish theproposed tasks.
The task descriptions and associated technical elements are complete and in a logicalsequence, with all proposed deliverables clearly defined such that a viable attempt toachieve project goals is likely as a result of award. The proposal identifies major
technical risks and clearly defines feasible mitigation efforts.

Potential Contribution and Relevance to the DARPA Mission:
The potential contributions of the proposed effort are relevant to the national technology
base. Specifically, DARPA’s mission is to maintain the technological superiority of the
U.S. military and prevent technological surprise from harming our national security bysponsoring revolutionary, high-payoff research that bridges the gap between fundamental
discoveries and their application.
This includes considering the extent to which any proposed intellectual property
restrictions will potentially impact the Government’s ability to transition the technology.



Cost Realism:
The proposed costs are realistic for the technical and management approach andaccurately reflect the technical goals and objectives of the solicitation. The proposed
costs are consistent with the proposer's Statement of Work and reflect a sufficientunderstanding of the costs and level of effort needed to successfully accomplish theproposed technical approach. The costs for the prime proposer and proposed subawardees
are substantiated by the details provided in the proposal (e.g., the type and number oflabor hours proposed per task, the types and quantities of materials, equipment andfabrication costs, travel and any other applicable costs).

B. Review and Selection Process
The review process identifies proposals that meet the evaluation criteria described above and are,therefore, selectable for negotiation of awards by the Government. DARPA policy is to ensure
impartial, equitable, comprehensive proposal evaluations and to select proposals that meet
DARPA technical, policy, and programmatic goals. If necessary, panels of experts in theappropriate areas will be convened. As described in Section IV, proposals must be deemed
conforming to the solicitation to receive a full technical review against the evaluation criteria;
proposals deemed non-conforming will be removed from consideration.
DARPA will conduct a scientific/technical review of each conforming proposal. Proposals will
not be evaluated against each other since they are not submitted in accordance with a common
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)40

work statement. DARPA’s intent is to review proposals as soon as possible after they arrive;
however, proposals may be reviewed periodically for administrative reasons.
Selections may be made at any time during the period of solicitation. Pursuant to FAR 35.016,
the primary basis for selecting proposals for award negotiation shall be technical, importance toagency programs, and fund availability. Conforming proposals based on a previously submitted
abstract will be reviewed without regard to feedback resulting from review of that abstract.
Furthermore, a favorable response to an abstract is not a guarantee that a proposal based on the
abstract will ultimately be selected for award negotiation. Proposals that are determined
selectable will not necessarily receive awards.
For evaluation purposes, a proposal is defined to be the document and supporting materials as
described in Section IV.B. Subject to the restrictions set forth in FAR 37.203(d), input ontechnical aspects of the proposals may be solicited by DARPA from non-Government
consultants/experts who are strictly bound by the appropriate non-disclosure requirements. No
submissions, classified or unclassified, will be returned.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

41

VI.

Award Administration Information
A. Selection Notices
After proposal evaluations are complete, proposers will be notified as to whether their proposal
was selected for award negotiation as a result of the review process. Notification will be sent by
email to the technical and administrative POCs identified on the proposal cover sheet. If a
proposal has been selected for award negotiation, the Government will initiate those negotiations
following the notification.
B. Administrative and National Policy Requirements
1. Intellectual Property
Proposers should note that the Government does not own the intellectual property of technical
data/computer software developed under Government contracts; it acquires the right to use thetechnical data/computer software. Regardless of the scope of the Government’s rights,
performers may freely use their same data/software for their own commercial purposes (unless
restricted by U.S. export control laws or security classification). Therefore, technical data andcomputer software developed under this solicitation will remain the property of theperformers, though DARPA desires to have a minimum of Government Purpose Rights (GPR)
to software developed through DARPA sponsorship.
If proposers desire to use proprietary software or technical data or both as the basis of their
proposed approach, in whole or in part, they should: (1) clearly identify such software/data
and its proposed particular use(s); (2) explain how the Government will be able to reach its
program goals (including transition) within the proprietary model offered; and (3) provide
possible nonproprietary alternatives in any area that might present transition difficulties orincreased risk or cost to the Government under the proposed proprietary solution.
Proposers expecting to use, but not to deliver, commercial open source tools or other materials
in implementing their approach may be required to indemnify the Government against legal
liability arising from such use.
All references to "Unlimited Rights" or "Government Purpose Rights" are intended to refer tothe definitions of those terms as set forth in the Defense Federal Acquisition Regulation
Supplement (DFARS) Part 227.
a. Intellectual Property Representations
All proposers must provide a good faith representation of either ownership or possession of
appropriate licensing rights to all other intellectual property to be used for the proposed
project. Proposers must provide a short summary for each item asserted with less thanunlimited rights that describes the nature of the restriction and the intended use of theintellectual property in the conduct of the proposed research.
b. Patents
All proposers must include documentation proving ownership or possession of appropriate
licensing rights to all patented inventions to be used for the proposed project. If a patentapplication has been filed for an invention, but it includes proprietary information and is not
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)42

publicly available, a proposer must provide documentation that includes: the patent
number, inventor name(s), assignee names (if any), filing date, filing date of any related
provisional application, and summary of the patent title, with either: (1) a representation of
invention ownership, or (2) proof of possession of appropriate licensing rights in theinvention (i.e., an agreement from the owner of the patent granting license to the proposer).
c. Procurement Contracts
 Noncommercial Items (Technical Data and Computer Software): Proposers
requesting a procurement contract must list all noncommercial technical data andcomputer software that it plans to generate, develop, and/or deliver, in which theGovernment will acquire less than unlimited rights and to assert specificrestrictions on those deliverables. In the event a proposer does not submit the list,
the Government will assume that it has unlimited rights to all noncommercial
technical data and computer software generated, developed, and/or delivered,unless it is substantiated that development of the noncommercial technical dataand computer software occurred with mixed funding. If mixed funding is
anticipated in the development of noncommercial technical data and computer
software generated, developed, and/or delivered, proposers should identify thedata and software in question as subject to GPR. In accordance with DFARS
252.227-7013, “Rights in Technical Data - Noncommercial Items,” and DFARS
252.227-7014, “Rights in Noncommercial Computer Software andNoncommercial Computer Software Documentation,” the Government willautomatically assume that any such GPR restriction is limited to a period of 5
years, at which time the Government will acquire unlimited rights unless theparties agree otherwise. The Government may use the list during the evaluation
process to evaluate the impact of any identified restrictions and may request
additional information from the proposer, as may be necessary, to evaluate theproposer’s assertions. Failure to provide full information may result in a
determination that the proposal is not compliant with the solicitation. A templatefor complying with this request is provided in Section IV.B.1.a.xii.(5).
 Commercial Items (Technical Data and Computer Software): Proposers
requesting a procurement contract must list all commercial technical data andcommercial computer software that may be included in any noncommercialdeliverables contemplated under the research project, and assert any applicable
restrictions on the Government’s use of such commercial technical data and/or
computer software. In the event a proposer does not submit the list, theGovernment will assume there are no restrictions on the Government’s use ofsuch commercial items. The Government may use the list during the evaluation
process to evaluate the impact of any identified restrictions and may request
additional information from the proposer to evaluate the proposer’s assertions.
Failure to provide full information may result in a determination that the proposal
is not compliant with the solicitation. A template for complying with this request
is provided in Section IV.B.1.a.xii.(5).
d. Other Types of Awards
Proposers responding to this solicitation requesting an award instrument other than a
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

43

procurement contract shall follow the applicable rules and regulations governing thoseaward instruments, but in all cases should appropriately identify any potential restrictions
on the Government’s use of any intellectual property contemplated under those award
instruments in question. This includes both noncommercial items and commercial items.
The Government may use the list as part of the evaluation process to assess the impact of
any identified restrictions, and may request additional information from the proposer, toevaluate the proposer’s assertions. Failure to provide full information may result in a
determination that the proposal is not compliant with the solicitation. A template forcomplying with this request is provided in Section IV.B.1.a.xii.(5).
2. Human Subjects Research
All research selected for funding involving human subjects, to include use of humanbiological specimens and human data, must comply with the federal regulations for humansubjects protection. Further, research involving human subjects that is conducted or
supported by the DoD must comply with 32 CFR 219, Protection of Human Subjects (andDoD Instruction 3216.02, Protection of Human Subjects and Adherence to Ethical Standards
in DoD-Supported Research (http://www.dtic.mil/whs/directives/corres/pdf/321602p.pdf).
Institutions awarded funding for research involving human subjects must provide
documentation of a current Assurance of Compliance with Federal regulations for humansubjects protection, such as a Department of Health and Human Services, Office of Human
Research Protection Federal Wide Assurance (http://www.hhs.gov/ohrp). All institutions
engaged in human subjects research, to include subawardees, must also hold a valid
Assurance. In addition, all personnel involved in human subjects research must provide
documentation of completion of human subjects research training.
For all proposed research that will involve human subjects in the first year or phase of the
project, the institution must provide evidence of or a plan for review by an Institutional
Review Board (IRB) upon final proposal submission to DARPA as part of their proposal,
prior to being selected for funding. The IRB conducting the review must be the IRB
identified on the institution’s Assurance of Compliance with human subjects protection
regulations. The protocol, separate from the proposal, must include a detailed description of
the research plan, study population, risks and benefits of study participation, recruitment andconsent process, data collection, and data analysis. It is recommended that you consult thedesignated IRB for guidance on writing the protocol. The informed consent document mustcomply with federal regulations (32 CFR 219.116). A valid Assurance of Compliance withhuman subjects protection regulations along with evidence of completion of appropriate
human subjects research training by all investigators and personnel involved with humansubjects research should accompany the protocol for review by the IRB.
In addition to a local IRB approval, a headquarters-level human subjects administrative
review and approval is required for all research conducted or supported by the DoD. The
Army, Navy, or Air Force office responsible for managing the award can provide guidance
and information about their component’s headquarters-level review process. Note that
confirmation of a current Assurance of Compliance with human subjects protection
regulations and appropriate human subjects research training is required before headquarterslevel approval can be issued.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)44

The time required to complete the IRB review/approval process varies depending on thecomplexity of the research and the level of risk involved with the study. The IRB approval
process can last between one and three months, followed by a DoD review that could last
between three and six months. Ample time should be allotted to complete the approval
process. DoD/DARPA funding cannot be used towards human subjects research until ALL
approvals are granted.
3. Animal UseAward recipients performing research, experimentation, or testing involving the use ofanimals shall comply with the rules on animal acquisition, transport, care, handling, and use
as outlined in: (i) 9 CFR parts 1-4, Department of Agriculture rules that implement the
Animal Welfare Act of 1966, as amended, (7 U.S.C. § 2131-2159); (ii) National Institutes of
Health Publication No. 86-23, "Guide for the Care and Use of Laboratory Animals" (8thEdition); and (iii) DoD Instruction 3216.01, “Use of Animals in DoD Programs.”
For projects anticipating animal use, proposals should briefly describe plans for Institutional
Animal Care and Use Committee (IACUC) review and approval. Animal studies in the
program will be expected to comply with the Public Health Service (PHS) Policy on Humane
Care and Use of Laboratory Animals, available at http://grants.nih.gov/grants/olaw/olaw.htm.
All award recipients must receive approval by a DoD-certified veterinarian, in addition to an
IACUC approval. No animal studies may be conducted using DoD/DARPA funding until theUnited States Army Medical Research and Materiel Command (USAMRMC) Animal Careand Use Review Office (ACURO) or other appropriate DoD veterinary office(s) grant
approval. As a part of this secondary review process, the award recipient will be required tocomplete and submit an ACURO Animal Use Appendix, which may be found athttps://mrmc-www.army.mil/index.cfm?pageid=Research_Protections.acuro&rn=1.
4. Export Control
Per DFARS 225.7901-4, all procurement contracts, other transactions and other awards, as
deemed appropriate, resultant from this solicitation will include the DFARS Export Control
clause (252.225-7048).
5. Electronic and Information Technology
All electronic and information technology acquired through this solicitation must satisfy theaccessibility requirements of Section 508 of the Rehabilitation Act (29 U.S.C. § 794d) and
FAR 39.2. Each project involving the creation or inclusion of electronic and information
technology must ensure that: (1) Federal employees with disabilities will have access to anduse of information that is comparable to the access and use by Federal employees who are notindividuals with disabilities; and (2) members of the public with disabilities seeking
information or services from DARPA will have access to and use of information and data thatis comparable to the access and use of information and data by members of the public who are
not individuals with disabilities.
6. Employment Eligibility Verification
As per FAR 22.1802, recipients of FAR-based procurement contracts must enroll as federal
contractors in E-verify and use the system to verify employment eligibility of all employees
DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)45

assigned to the award. All resultant contracts from this solicitation will include FAR 52.22254, “Employment Eligibility Verification.” This clause will not be included in grants,
cooperative agreements, or Other Transactions.
7. System for Award Management (SAM) and Universal Identifier Requirements
Unless the proposer is exempt from this requirement, as per FAR 4.1102 or 2 CFR 25.110 asapplicable, all proposers must be registered in the System for Award Management (SAM) andhave a valid Data Universal Numbering System (DUNS) number prior to submitting a
proposal. All proposers must maintain an active registration in SAM with current information
at all times during which they have an active Federal award or proposal under consideration
by DARPA. All proposers must provide the DUNS number in each proposal they submit.
Information on SAM registration is available at www.sam.gov.
Note that new registrations can take an average of 7-10 business days to process in SAM.
SAM registration requires the following information:
 DUNS number
 TIN
 CAGE Code. If a proposer does not already have a CAGE code, one will be assigned

during SAM registration.
 Electronic Funds Transfer information (e.g., proposer’s bank account number, routing
number, and bank phone or fax number).
8. Reporting Executive Compensation and First-Tier Subcontract Awards
FAR clause 52.204-10, “Reporting Executive Compensation and First-Tier Subcontract
Awards,” will be used in all procurement contracts valued at $25,000 or more. A similaraward term will be used in all grants and cooperative agreements.
9.

Updates of Information Regarding Responsibility Matters

Per FAR 9.104-7(c), FAR clause 52.209-9, Updates of Publicly Available Information
Regarding Responsibility Matters, will be included in all contracts valued at $500,000 or more
where the contractor has current active Federal contracts and grants with total value greaterthan $10,000,000.
10. Representations by Corporations Regarding an Unpaid Delinquent Tax Liability or
a Felony Conviction under any Federal Law
The following representation will be included in all awards:(a) In accordance with section 101(a) of the Continuing Appropriations Act, 2016 (Pub. L.
114-53) and any subsequent FY 2016 appropriations act that extends to FY 2016 funds thesame restrictions as are contained in sections 744 and 745 of division E, title VII, of theConsolidated and Further Continuing Appropriations Act, 2015 (Pub. L. 113-235), none of
the funds made available by this or any other Act may be used to enter into a contract withany corporation that —DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

46

(1) Has any unpaid Federal tax liability that has been assessed, for which all judicial
and administrative remedies have been exhausted or have lapsed, and that is not being
paid in a timely manner pursuant to an agreement with the authority responsible forcollecting the tax liability, where the awarding agency is aware of the unpaid tax
liability, unless the agency has considered suspension or debarment of the corporationand made a determination that this further action is not necessary to protect theinterests of the Government; or(2) Was convicted of a felony criminal violation under any Federal law within thepreceding 24 months, where the awarding agency is aware of the conviction, unlessthe agency has considered suspension or debarment of the corporation and made adetermination that this action is not necessary to protect the interests of theGovernment.
(b) The Offeror represents that –(1) It is [ ] is not [ ] a corporation that has any unpaid Federal tax liability that has
been assessed, for which all judicial and administrative remedies have been exhausted
or have lapsed, and that is not being paid in a timely manner pursuant to an agreementwith the authority responsible for collecting the tax liability,(2) It is [ ] is not [ ] a corporation that was convicted of a felony criminal violation
under a Federal law within the preceding 24 months.
Each proposer must complete and return the representations outlined in IV.B.1.a.xii.(8) with
their proposal submission.
11. Cost Accounting Standards (CAS) Notices and Certification
As per FAR 52.230-2, any procurement contract in excess of the referenced thresholdresulting from this solicitation will be subject to the requirements of the Cost AccountingStandards Board (48 CFR 99), except those contracts which are exempt as specified in 48
CFR 9903.201-1. Any proposer submitting a proposal which, if accepted, will result in a CAS
compliant contract, must submit representations and a Disclosure Statement as required by 48
CFR 9903.202 detailed in FAR 52.230-2. The disclosure forms may be found at
http://www.whitehouse.gov/omb/procurement_casb.
12. Controlled Unclassified Information (CUI) on Non-DoD Information Systems
Controlled Unclassified Information (CUI) refers to unclassified information that
does not meet the standards for National Security Classification but is pertinent to
the national interests of the United States or to the important interests of entities
outside the Federal Government and under law or policy requires protection fromunauthorized disclosure, special handling safeguards, or prescribed limits on
exchange or dissemination. All non-DoD entities doing business with DARPA are
expected to adhere to the following procedural safeguards, in addition to any otherrelevant Federal or DoD specific procedures, for submission of any proposals to
DARPA and any potential business with DARPA:

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)47



Do not process DARPA CUI on publicly available computers or post
DARPA CUI to publicly available webpages or websites that have accesslimited only by domain or Internet protocol restriction.

Ensure that all DARPA CUI is protected by a physical or electronic barrier
when not under direct individual control of an authorized user and limit thetransfer of DARPA CUI to subawardees or teaming partners with a need toknow and commitment to this level of protection.

Ensure that DARPA CUI on mobile computing devices is identified andencrypted and all communications on mobile devices or through wirelessconnections are protected and encrypted.

Overwrite media that has been used to process DARPA CUI before external
release or disposal.

13. Safeguarding of Covered Defense Information and Cyber Incident Reporting
Per DFARS 204.7304, DFARS 252.204-7012, “Safeguarding of Covered Defense
Information and Cyber Incident Reporting,” applies to this solicitation and all FAR-based
awards resulting from this solicitation.
14. Prohibition on Contracting with Entities that Require Certain Internal
Confidentiality Agreements(a) In accordance with section 101(a) of the Continuing Appropriations Act, 2016 (Pub. L.
114-53) and any subsequent FY 2016 appropriations act that extends to FY 2016 funds thesame restrictions as are contained in section 743 of division E, title VII, of the Consolidated
and Further Continuing Appropriations Act, 2015 (Pub. L. 113-235), none of the funds
appropriated (or otherwise made available) by this or any other Act may be used for a contract
with an entity that requires employees or subcontractors of such entity seeking to report fraud,
waste, or abuse to sign internal confidentiality agreements or statements prohibiting orotherwise restricting such employees or contactors from lawfully reporting such waste, fraud,or abuse to a designated investigative or law enforcement representative of a Federal
department or agency authorized to receive such information.
(b) The prohibition in paragraph (a) of this provision does not contravene requirements
applicable to Standard Form 312, Form 4414, or any other form issued by a Federal
department or agency governing the nondisclosure of classified information.
(c) Representation. By submission of its offer, the Offeror represents that it does not require
employees or subcontractors of such entity seeking to report fraud, waste, or abuse to sign orcomply with internal confidentiality agreements or statements prohibiting or otherwise
restricting such employees or contactors from lawfully reporting such waste, fraud, or abuse
to a designated investigative or law enforcement representative of a Federal department or
agency authorized to receive such information.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)48

C. Reporting
1. Technical and Financial Reports
The number and types of technical and financial reports required under the contracted
project will be specified in the award document, and will include, as a minimum, monthly
financial status reports and a yearly status summary. A final report that summarizes the
project and tasks will be required at the conclusion of the performance period for the award.
The reports shall be prepared and submitted in accordance with the procedures contained in
the award document.
2. Representations and Certifications
In accordance with FAR 4.1201, prospective proposers shall complete electronic annual
representations and certifications at http://www.sam.gov.
3. Wide Area Work Flow (WAWF)
Unless using another means of invoicing, performers will be required to submit invoices forpayment directly at https://wawf.eb.mil. If applicable, WAWF registration is required prior to
any award under this solicitation.
4. i-Edison
Award documents will contain a requirement for patent reports and notifications to besubmitted electronically through the i-Edison Federal patent reporting system at https://sedison.info.nih.gov/iEdison.

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

49

VII.

Agency Contacts
DARPA will use email for all technical and administrative correspondence regarding thissolicitation.
Technical POC: David Gunning, Program Manager, DARPA/I2O

Email: XAI@darpa.mil



Mailing address:
DARPA/I2O
ATTN: DARPA-BAA-16-53
675 North Randolph Street
Arlington, VA 22203-2114



I2O Solicitation Website: http://www.darpa.mil/work-with-us/opportunities

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)50

VIII.

Other Information
A. Frequently Asked Questions (FAQs)
Administrative, technical, and contractual questions should be sent via email to
XAI@darpa.mil. All questions must be in English and must include the name, email address,
and the telephone number of a point of contact.
DARPA will attempt to answer questions in a timely manner; however, questions submitted
within 7 days of closing may not be answered. If applicable, DARPA will post FAQs to
http://www.darpa.mil/Opportunities/Solicitations/I2O_Solicitations.aspx.
B. Proposers Day
The Proposers Day will be held on August 11, 2016, in Arlington, VA.
For further information regarding the Dispersed Computing Proposers Day, please see
DARPA‐SN‐16-53, which is the Special Notice announcing the event,
https://www.fbo.gov/index?s=opportunity&mode=form&id=de513fd656437e8124430922e66f58
41&tab=core&_cview=1
C. Submission Checklist
The following items apply prior to proposal submission. Note: some items may take up to 1
month to complete.


Item

BAA
SectionApplicability

Abstract

IV.B.1

Optional, but
recommended

Obtain DUNS
number

IV.B.2.a.i

Required of all
proposers

Obtain Taxpayer
Identification
Number (TIN)

IV.B.2.a.i

Required of all
proposers

Register in the
System for AwardManagement (SAM)

VI.B.7

Required of all
proposers

Register in E-Verify

VI.B.6DARPA-BAA-16-53

Required for
proposers
requesting

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)Comment
Conform to stated page limit.
The DUNS Number is the Federal Government's contractor
identification code for all procurement-related activities. See
http://fedgov.dnb.com/webform/index.jsp to request a DUNS
number. Note: requests may take at least one business day.
A TIN is used by the Internal Revenue Service in theadministration of tax laws. See
http://www.irs.gov/businesses/small/international/article/0,,id=96696,00.html for information on requesting a TIN. Note:requests may take from 1 business day to 1 month dependingon the method (online, fax, mail).
The SAM combines Federal procurement systems and theCatalog of Federal Domestic Assistance into one system. See
www.sam.gov for information and registration. Note: newregistrations can take an average of 7-10 business days. SAM
registration requires the following information:-DUNS number-TIN-CAGE Code. A CAGE Code identifies companies doingor wishing to do business with the Federal Government. If
a proposer does not already have a CAGE code, one will beassigned during SAM registration.
-Electronic Funds Transfer information (e.g., proposer’sbank account number, routing number, and bank phone or
fax number).
E-Verify is a web-based system that allows businesses todetermine the eligibility of their employees to work in the
United States. See http://www.uscis.gov/e-verify for51

procurement
contracts
Ensurerepresentations andcertifications are up
to dateEnsure eligibility of
all team members

Register at
Grants.gov

VI.C.2Required of all
proposers

III

Required of all
proposers

IV.E.1.cRequired for
proposers
requesting grants
or cooperative
agreements

information and registration.
Federal provisions require entities to represent/certify to avariety of statements ranging from environmental rules
compliance to entity size representation. See
http://www.sam.gov for information.
Verify eligibility, as applicable, for in accordance withrequirements outlined in Section 3.
Grants.gov requires proposers to complete a one-timeregistration process before a proposal can be electronicallysubmitted. If proposers have not previously registered, thisprocess can take between three business days and four weeks
if all steps are not completed in a timely manner. See theGrants.gov user guides and checklists at
http://www.grants.gov/help/html/help/index.htm#t=Get_Start
ed%2FGet_Started.htm for further information.

The following items apply as part of the submission package:


Item

BAA
SectionApplicability

Comment

Volume 1(Technical andManagement
Proposal)IV.B.2.aRequired of all proposers

Conform to stated page limits and formatting
requirements. Include all requested information.

Appendix A

IV.B.2.a.xii

Required of all proposers

Volume 2(Cost Proposal)

IV.B.2.bRequired of all proposers

DARPA-BAA-16-53

EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)-Team member identification- Government/FFRDC team member proof ofeligibility- Organizational conflict of interest affirmations- Intellectual property assertions- Human subjects research- Animal use- Unpaid delinquent tax liability/felony conviction
representations-CASB disclosure, if applicable- Cover Sheet- Cost summary- Detailed cost information including justifications
for direct labor, indirect costs/rates,
materials/equipment, subcontractors/consultants,
travel, ODCs
- Cost spreadsheet file (.xls or equivalent format)- If applicable, list of milestones for OTs- Subcontractor plan, if applicable
Subcontractor cost proposals- Itemized list of material and equipment items tobe purchased with vendor quotes or engineering
estimates for material and equipment more than
$50,000- Travel purpose, departure/arrival destinations,and sample airfare52