Comments template for Draft SP 800-207 Please respond by November 22, 2019 Submitted by: IDSA 

Date: 11/21/19

Type: E - Editorial, G - General T - Technical 1 of 2

Submit comments by October 15, 2020 to: 

explainable-AI@nist.gov

Comment # Commenter 

organization

Commenter name Paper Line # (if 

applicable)

Paper 

Section (if 

applicable)

Comment (Include rationale for 

comment)

Suggested change

US Army - Picatinny 

Arsenal - SW 

Quality 

Competency Group Adam Hilburn 122

Consider adding example of which 

law/regulation. By stating an 

example, the reader can have a quick 

relatable understanding versus 

having to dig up the referenced 

paper.

This drive is motivated in part by laws and regulations, 

such as x,y,z, which state that decisions...

US Army - Picatinny 

Arsenal - SW 

Quality 

Competency Group Adam Hilburn 233

Clear knowledge limits aid SW testers 

as they develop their test cases which 

factor in boundaries. 

Consider adding a line saying: The knowledge limits 

principle aids SW testers as they define test cases. 

US Army - Picatinny 

Arsenal - SW 

Quality 

Competency Group Adam Hilburn 286

A Software Quality Engineering (SQE) 

role within AI development is to 

ensure there is unambiguous 

explanations of system outputs. 

...explanations must be highly interpretable and 

unambiguous. 

US Army - Picatinny 

Arsenal - SW 

Quality 

Competency Group Adam Hilburn 324-325

Similar to developing Safety critical 

SW, a solid SRS and SDD must be 

developed to understand the inner 

working of the SW. 

Consider adding how traditional aka non-AI SW 

solutions, implement similar explainable principles. 

Comment Template for First Public Draft of Four 

Principles of Explainable Artificial Intelligence (Draft 

NISTIR 8312)

All comments will be made public as-is, with no edits or redactions. Please be careful to not include confidential business or personal information, otherwise sensitive or 

protected information, or any information you do not wish to be posted.
Comments template for Draft SP 800-207 Please respond by November 22, 2019 Submitted by: IDSA 

Date: 11/21/19

Type: E - Editorial, G - General T - Technical 2 of 2

US Army - Picatinny 

Arsenal - SW 

Quality 

Competency Group Adam Hilburn

What would be the ideal SW 

product/deliverable that would have 

all of the explainable models defined? 

SDD, SRS, other?

Consider adding where the explainable models should 

ideally be documented. 

US Army - Picatinny 

Arsenal - SW 

Quality 

Competency Group Adam Hilburn

Having explainable models helps us 

better understand system 

uncertainties which will drive to 

better risk identification. 

Consider touching on the issue with uncertainty 

within AI systems and how explainable models can 

help better characterize the uncertainty...which leads 

to identifying the proper SW testing strategy. 

US Army - Picatinny 

Arsenal - SW 

Quality 

Competency Group Adam Hilburn

Is there work being done to 

decompose the criteria for what 

constitutes a meaningful model? I'm 

assuming the criteria revolves around 

the traditional quality criteria: 

testable, unambiguous, complete, 

etc. n/a

US Army - Picatinny 

Arsenal - SW 

Quality 

Competency Group Adam Hilburn 608-609

Collaboration is needed to find a 

middle ground for what is considered 

meaningful. This is critical for SQE 

because we cannot create a test 

strategy with an uninterpretable 

model.

...what is considered meaningful caries by context and 

across people. Collaboration amongst all 

stakeholders (SW developers, SQE, system eng) is 

required to establish meaningful models. 

US Army - Picatinny 

Arsenal - SW 

Quality 

Competency Group Adam Hilburn 687

Compare this line to the need for 

properly characterizing the context 

of the system. n/a