Are We on the Right Side 

of the Equation?

John M. Butler, PhD

NIST Fellow & Special Assistant to the Director for Forensic Science

National Institute of Standards and Technology

Forensics@NIST 2018 7 November 2018 Gaithersburg, MD

DAY1 KEYNOTE ADDRESS
NIST Disclaimer

Points of view are mine and do not necessarily represent the 

official position or policies of the National Institute of Standards 

and Technology.

Certain commercial entities are identified in order to specify 

experimental procedures as completely as possible. In no case 

does such identification imply a recommendation or 

endorsement by the National Institute of Standards and 

Technology, nor does it imply that any of the entities identified 

are necessarily the best available for the purpose.
NIST Forensic Science Activities

Partner with Community 

to Strengthen Policies 

and Practices

Convene Meetings 

to Examine Issues

2013 - present

National Commission 

on Forensic Science 

(NCFS) with DOJ

2013 - 2017 Extramural Research

Conduct Research 

and Collaborate

Explore Scientific 

Foundations

2017 - present

Initial efforts with DNA 

mixture interpretation 

and bitemark analysis

Human Factors 

Working Groups 

(with NIJ)

2009 - present

Extramural Research

funding a NIST Center of 

Excellence in Forensic 

Science (CSAFE: since 2014)

1920s - present

Why we are here 

for this meeting...

Intramural Research

DNA

Digital

Fingerprints

Firearms

Footmarks

Statistics

Toxins

Trace
What Has Happened 

in the Past Two Years 

since Forensics@NIST 2016?
Detective X Film awarded an Emmy® this year!

For more information on Wilmer Souder, see June 2016 colloquium:

https://www.nist.gov/video/nist-colloquium-series-detective-x-wilmer-souder-and-early-history-forensic-science-national

NIST staff members Leon Gerskovic, Robin Materese and Jose Garcia show 

off their Emmy® Award for "Detective X: (Re) Discovering Wilmer Souder."

Credit: J. Stoughton/NIST

Awarded June 23, 2018 by the National Academy of Television 

Arts & Sciences: National Capital Chesapeake Bay Chapter

https://www.nist.gov/video/detective-x-re-discovering-wilmer-souder

Previewed at Forensics@NIST 2016

Article: https://www.nist.gov/featured-stories/who-was-detective-x
Wilmer Souder Notebooks (Special Collection)

https://nistdigitalarchives.contentdm.oclc.org/digital/collection/p16009coll67/search

Notebook #9 (Sept 1950)

Notebook #2 (Oct 1933)

Notebook #5 (July-Oct 1930)
New NIST Director

Then NIST Director Willie E. May 

speaking at the Forensics@NIST 2016 

meeting. He retired in January 2017 

after 45 years at NIST. https://www.nist.gov/people/walter-g-copan

Walter G. Copan, NIST Director 

since October 5, 2017
NCFS Closed with Charter Expiration

• The Attorney General's National Commission on Forensic Science's (NCFS) 

charter expired on April 23, 2017 

• Completed two 2-year terms involving 13 meetings and approving 43 work 

products (20 recommendations to the Attorney General and 23 views of the 

Commission)

https://www.justice.gov/archives/ncfs

https://www.justice.gov/archives/ncfs/page/file/959356/download

See summary 

document (58 pages) 

describing what was 

accomplished and 

thoughts on what 

needs still exist

NIST maintains video recordings of the NCFS meetings: 

https://www.nist.gov/topics/forensic-science/national-commission-forensic-science
National Commission on Forensic Science (NCFS) 

operated for 13 meetings from February 2014 to April 2017

NIST Director (Patrick Gallagher), 

Deputy Attorney General (James 

Coles), and the President’s Science 

Advisor (John Holdren) speak at the 

first National Commission on Forensic 

Science meeting in February 2014 

Coordinating Government Efforts

Mark Stolorow introduced 

NIST plans for OSAC at 

the first NCFS meeting

Introducing New Efforts

140 presenters spoke to 

the NCFS in its 13 meetings

Judge Harry Edwards, who chaired 

the National Academy of Sciences 

committee that wrote the 2009 NAS 

Report, addressed the NCFS

Learning from Previous Efforts
Mark Stolorow Has Announced Plans to Retire

https://www.nist.gov/blogs/takingmeasure/forensic-science-my-dna

He started in 1968 

as a beat cop in 

Ann Arbor, Michigan

50 years in 

forensic science!

Mark will retire in 

January 2019

as Deputy Director of the 

NIST Special Programs 

Office and Director for 

OSAC Affairs Michigan and Illinois State Police 

Crime Labs, Cellmark, NIST
NIST Has Organized Multiple Meetings to Assist 

the Community since Forensics@NIST 2016

https://www.nist.gov/topics/forensic-science/conferences-and-events

Coming Next Summer...

June 19-20, 2019
Lots of Great NIST Research...

This is why we are here for Forensics@NIST 2018

Who Was Detective X?

February 9, 2017

Feature Story

Rich Press

richard.press@nist.gov

(301) 975-0501

21 articles 

since 2016

Trace Evidence Databases: 

A Force Multiplier for 

Forensic Investigators

November 7, 2016

Trace

Fentanyl Can Sicken 

First Responders. Here’s 

a Possible Solution.

May 9, 2017

Trace

Do You Have What It 

Takes to be a Forensic 

Fingerprint Examiner?

May 18, 2017

Fingerprints

Scientists Lay the Groundwork for 

a Reliable Marijuana Breathalyzer

July 5, 2017

Toxins

Sniffing Like a Dog Can 

Improve Trace Detection 

of Explosives

December 1, 2016

Trace

NIST Research 

Enables Enhanced 

DNA “Fingerprints”

December 15, 2016

DNA

Database of Software 

“Fingerprints” Expands 

to Include Mobile Apps

December 15, 2016

Digital

Standard for Sampling 

Seized Drugs Approved 

for OSAC Registry

April 3, 2017

OSAC Update

Speaking of Error in 

Forensic Science

September 5, 2017

Meeting Summary

Scientists Automate 

Key Step in Forensic 

Fingerprint Analysis

August 14, 2017

Fingerprints

and (Rich) Press Coverage
Great NIST Research...and (Rich) Press Coverage (cont.)

NIST details plans for 

reviewing the 

scientific foundations 

of forensic methods

September 24, 2018

NIST to Assess the Reliability 

of Forensic Methods for 

Analyzing DNA Mixtures

October 3, 2017

How Good a Match is It? 

Putting Statistics into Forensic 

Firearms Identification

February 8, 2018

Database of Software 

“Fingerprints” Expands to 

Include Computer Games

September 10, 2018

Free Software Can Help 

Spot New Forms of Fentanyl 

and Other Illegal Drugs

March 7, 2018

New Protocol for Measuring 

Background Levels of 

Drugs in Crime Labs

September 25, 2018

Drone Forensics Gets a 

Boost With New Data on 

NIST Website

June 5, 2018

Data-Sharing Website 

May Speed Response 

to New Illegal Drugs

July 5, 2018

NIST Builds Statistical Foundation for 

Next-Generation Forensic DNA Profiling

July 23, 2018

NIST Updates Forensic Standard 

Reference Materials

September 19, 2018

Scientific Foundations

Firearms

DNA Digital

Firearms DNA

Trace

Toxins

Digital

Toxins 21 articles 

since 2016

Scientific Foundations
A Brief History of the Forensics@NIST Meetings

• 2010 (December 6-8): limited to NIST and NIJ staff

• Keynotes: Dr. Patrick Gallagher, NIST Director and Dr. John Laub, NIJ Director

• 2012 (November 28-30)

Keynote: Dr. Tjark Tjin-a-Tsoi, Netherlands Forensic Institute CEO, 

• “Trends, Challenges and Strategy in Forensics”

• 2014 (December 3-4)

Keynote: Judge Jed Rakoff, U.S. District Court Justice, 

• “Are Judges Losing Confidence in Forensic Science?”

• 2016 (November 8-9) 

Keynote: Professor Jules Epstein, Professor of Law at Temple University, 

• “Forensic Evidence: Thoughts of an Accidental Tourist”

• 2018 (November 7-8)

Keynotes: Dr. John Butler (Nov. 7) & Dr. Sheila Willis (Nov. 8)
Previous Keynote Speakers

2012 Dr. Tjark Tjin-a-Tsoi 2014 Judge Jed Rakoff 2016 Professor Jules Epstein

MOU signed between NFI and NIST

12 pages of text 60 slides and 45 minute video

34 slides
2012 Dr. Tjark Tjin-a-Tsoi

“Trends, Challenges and Strategy in Forensics”

•The challenges of backlogs can be addressed 

with (1) service level agreements, (2) process 

redesign, and (3) speed focused R&D

• More objective interpretation can be obtained 

with (1) research, (2) defragmentation of forensic 

disciplines, and (3) improved training & education 
2014 Judge Jed Rakoff

“Are Judges Losing Confidence in Forensic Science?”

• “There remains a tendency...to admit forensic evidence 

because it is traditional to do so.”

• “The many highly responsible forensic specialists who have 

devoted years of their time and skill to one or another forensic 

discipline should not be defensive about this growing judicial 

uncertainty, but should instead see it as an invitation and 

opportunity to help their chosen fields become more 

rigorous and reliable.”

• “...give your attention to what can be done to make forensic 

science a more useful tool of justice...”
2016 Professor Jules Epstein

“Forensic Evidence: Thoughts of an Accidental Tourist”

• “Daubert’s ‘evidentiary’ reliability test does not necessarily mean ‘accurate’” and 

“Frye’s ‘general acceptance’ test varies with the community.” 

• “Daubert is applied far more stringently in civil cases than in criminal ones, and even in 

criminal ones, Daubert serves a more effective screening role for filtering out flawed defense 

expert testimony than weak prosecution and police-generated forensic science.”

• Problems from his perspective:

1. “We still permit risky evidence” – bitemark matching and 2014 study of odontologists

2. Ethics – “honestly communicate with all parties” when “permitted by law or agency practice”

3. Communicating Results – example testimony: “[a match is] probable...I’m 100% confident 

with that opinion”

4. “Do we yet know how reliable experts are?” – what data exists? 

5. “When is enough enough?” – when trying to conclude that two items have a common origin 

if their marks are in “sufficient agreement” – what does that mean?
Are We on the 

Right Side of 

the Equation?
Perspective gained from focused thinking about 

scientific foundations this past year: 

Are We Check-List Focused or Outcome Focused?

• Do we understand principles behind how 

things work and why things are done?

• Do we regularly step back and critically 

consider our performance with activities we 

are involved in as researchers or practitioners?

• What data demonstrate what we think we 

know about the performance of a particular 

methodology or interpretation approach?
Are We on the Right Side of the Equation?

Component(s) + Process(es) = Outcome

How?

What?

Systems Thinking is Looking at the Big Picture and How Inputs Impact Outputs...

How well?

So what?

TaskDriven

Left Side

PerformanceBased

Right Side
Accreditation & Audits: 

Are we considering the right side of the equation?

• Austin, Texas PD DNA Laboratory was 

shutdown in May 2016 over concerns 

with DNA testing protocols 

• They passed 17 audits over a 13 year 

time span

• In response to TX FSC concerns raised, 

a representative of the accrediting body 

stated “there is no consensus on what is 

acceptable in the DNA community” and 

“we [ANAB] do not establish the 

scientific foundation, but we assess to 

that. We expect the technical 

community to be establishing what 

scientifically needs to be done.”

Texas Forensic Science Commission Meeting August 18, 2017

https://www.youtube.com/watch?v=-p_30-20kQI

(at 4:03 of 5:45) Slide courtesy of Lynn Garcia (presentation to the Texas Forensic Science Commission, August 18, 2017)

Austin (TX) Police Department (APD) DNA Laboratory

Accreditation Timeline and Discussion March 2004--May 2016 

Two-thirds of the APD 

audits had zero findings
Validation Studies: 

Are we considering the right side of the equation?

A common claim is that a check-list of criteria have been met: “validation of 

the [DNA test kit] was carried out in accordance with guidelines ...issued by the 

Scientific Working Group on DNA Analysis Methods (SWGDAM) and a series of 

tests ...were conducted.” (FSIG 27:67-73)

Task-driven

4.1 Known and nonprobative evidence samples

4.2 Sensitivity and stochastic studies

4.3 Precision and accuracy: repeatability

4.3 Precision and accuracy: reproducibility

4.4 Mixture studies

4.5 Contamination assessment

4.4 Mixed DNA samples that are representative 

of those typically encountered by the testing 

laboratory should be evaluated

SWGDAM Validation Guidelines for DNA Analysis Methods (2016)

Left side of 

the equation

Right side of 

Performance-based the equation

Studies assessing 

sensitivity and 

specificity
A new standard that proposes working 

from the right side of the equation

Foreword: “Following development, it is critical for a 

laboratory to verify that the interpretation protocols work 

as designed.” (i.e., to be performance-based 

rather than task-driven)

“4.3 The data from the validation studies performed by the 

laboratory shall be the basis for the interpretation 

parameters and protocols developed by the laboratory and 

shall provide guidance for the types of mixed DNA profiles 

that will be interpreted by the laboratory.”

“4.4 The laboratory shall verify and document that the 

mixture interpretation protocols developed from the 

validation studies generate reliable and consistent 

interpretations and conclusions for the types of mixed 

DNA samples typically encountered by the laboratory.”

Initially developed by 

OSAC in 2016 (Biological 

Data Interpretation and 

Reporting Subcommittee)

ANSI/ASB approved 

in September 2018
Dr. Wilmer Souder and the National Bureau of 

Standards’ Identification Laboratory (1935)

One of Our Nation’s First Forensic Laboratories Physics PhD, University of 

Chicago (1916) 

Worked more than 800 

forensic cases from 1929 

to 1953

Handwriting, typewriter 

identification, and ballistics 

analysis

Helped set up the FBI 

Laboratory (1932)

Testified as a handwriting 

expert in the Lindbergh baby 

kidnapping case (1935)

https://www.nist.gov/featured-stories/who-was-detective-x
Perspectives from History: 

Are we on the right side of the equation?

In his 1933 talk to the International Association of Chiefs of Police entitled “Beware 

the Amateur Expert”, Wilmer Souder from the National Bureau of Standards states: 

• “Today many workers are operating without the least supervision or concern 

as to what is correct scientific procedure. Too often their enthusiasm outruns 

their training and ability. Some serious complications naturally develop under such 

conditions.” He continues: “Success comes from skill in selecting the proper 

method and following it through in its correct application.” And later: “The safe 

investigator has standards to be observed.” He concludes: “I hope this bold 

admission of our lack of standards in what should be a highly scientific field will not 

discourage you.” 

In 1933 and 1934, Wilmer Souder spoke to the IACP. His remarks were reprinted in a 1977 book entitled “Silent 

Witness: The Emergence of Scientific Criminal Investigations”, which is the third volume of a Police History Series. 
Perspectives from History: 

Are we on the right side of the equation?

L.J. O’Rourke of the U.S. Civil Service Commission spoke to the International 

Association of Chiefs of Police in 1936; his talk was entitled “Scientific Standards 

in Criminal Investigations”:

• “...the use as a basis for evidence of instruments whose validity is not 

known will merely discredit investigation work.”

• He pleads “for greater knowledge of validity of [scientific] methods and the 

development of more valid measures.” 

• O’Rourke proposes setting up “a National Bureau of Standards in Criminology 

to conduct scientifically controlled experiments and to evaluate present practices.” 

He emphasizes: “To make better use of [scientific] methods, law enforcement 

agencies must be certain of their limitations as well as of their merits.” 

His remarks were reprinted in a 1977 book entitled “Silent Witness: The Emergence of Scientific Criminal 

Investigations”, which is the third volume of a Police History Series. 
With What Mindset Do You Approach a Problem?

• Are you using the left side or the right side of 

your brain in problem solving?

• Left-side = linear, verbal, and sequential thinking

• Right-side = spatial perception & creativity

• We need both sides to create the best solutions

• Are we task-driven (left-side of the equation) or 

performance-based (right-side of the equation) 

in our work?

• We need both sides to create the best solutions in 

forensic science 

• This applies to both researchers and practitioners

Published in 1998
Take Time to Go on a NIST Tour While You are Here...

NIST researcher Mike 

Riley describing NIST 

test methods for 

evaluating body armor 

(bulletproof vests) to 

Forensics@NIST 2016 

participants

NIST Ballistics Laboratory
www.nist.gov/forensics

301-975-4049 john.butler@nist.gov

Thank you for your attention!