Annual Update on Activities of the 

FSSB Statistics Task Group

William Guthrie 

June 5, 2019
Background: Statistics Task Group (STG)

• Made up of statisticians embedded in the SACs or Subcommittees, atlarge affiliates, and observers

• Embedded statisticians and at-large affiliates are voting members

• Embedded statisticians cover the FSSB, 4/5 SACs, and 11/25 Subcommittees

• Resource for OSAC statisticians to leverage expertise and foster cross

fertilization of ideas

• Now beginning to operate like a Resource Committee for OSAC Units

with no embedded statistician as well

• (17+6)=23 current members + 3 observers + 2 Kavi Liaisons
Members

• Madeleine Ausdemore, South Dakota State

University

• David Banks, Duke University

• Fred Bieber, Harvard Medical School

• Georgiy Bobashev, RTI International

• Alicia Carriquiry, Iowa State University

• James Curran, University of Auckland

• Simone Gittelson, Univ. of Technology Sydney

• William Guthrie, NIST Statistical Engineering

• David Kaye, Penn State Law

• Mark Lancaster, Northern Kentucky University

• Steven Lund, NIST Statistical Engineering

• Abhyuday Mandal, University of Georgia

• Max Morris, Iowa State University

• Cedric Neumann, South Dakota State Univ.

• Brent Ostrum, Canada Border Services

Agency

• Mark Ruefenacht, NIST Weights & Measures

• Chris Saunders, South Dakota State Univ.

• Michael A. Smith, FBI

• Hal Stern, University of California Irvine

• Bill Thompson, Univ. of California, Irvine

• Haonan Wang, Colorado State University

• Margaret Warner, National Center for

Health Statistics

• Bruce Weir, University of Washington
3

Individual Self-Directed Tasks

• Review and comment on documents as part of SAC and Subcommittees,

prior to sending documents to SDO’s, and during open comment

• Membership in Task Groups drafting or revising documents

• Revision of ASTM E2764 Standard Practice for Uncertainty Assessment in the

Context of Seized-Drug Analysis

• Drafting of Standard Practices for Evaluating Measurement Uncertainty of

Quantitative Measurements in Forensic Toxicology

• Drafting of conclusions language for the comparison of questioned documents

• Short-term statistical consultation to SACs and Subs in support of

document development

• Comparison of methods for dental age estimation
4

Example – Task Group Work
5

Example – Task Group Work
6

Example – Stat Consulting
7

Centrally Organized Tasks

• Training Sessions

• Two plenary training sessions offered at Nov./Dec. 2018 All-Hands Meetings

• Stats 101 for Forensic Science – Interactive Session on Control Charts

• Forensic Statistics and the Probative Value of Evidence

• Membership in Interdisciplinary Task Groups

• Two STG members participating in FSSB Technical Issues Task Group with other

members of Trace Subcommittee, Chemistry SAC, and affiliates

• One goal to address concerns over definitions for differences between trace

evidence items when compared using expert judgement

• Another goal to investigate use of statistical methods for comparison rather than

expert judgement – with a focus on comparison of automotive paint by FTIR
8

Definitions: Significant and Meaningful Differences

• Old: significant difference - a difference between two samples that

indicates that they do not share a common origin.

• Current: meaningful difference - a feature or property of a sample

that does not fall within the variation exhibited by the comparison

sample, considering the limitations of the sample or technique, and

therefore indicates the two samples do not share a common origin.

• Discussion - The use of this term does not imply the formal

application of statistics.
9

Proposed Definition: Meaningful Difference

• meaningful difference - a difference in a feature or property of items compared,

based on expert judgment by a qualified analyst, deemed substantial enough

to conclude that the two items do not share a common origin.

• Note 1: The finding of a meaningful difference must be based on comparison

methods that account for (or consider) all necessary secondary characteristics

(e.g., environmental exposure).

• Note 2: Determination of a meaningful difference must consider and account

for limitations of the technique(s) used and the items being compared.

• Note 3: This term shall be used when the determination of the difference is

based on expert judgment and not on the formal use of statistical methods. As

a result, meaningful difference does not imply statistical significance.
10

Proposed Definition: Statistically Significant Difference

• statistically significant difference - a difference in a feature or property of items

compared, based on a statistical analysis, that quantifies that the difference is

extreme enough to be unlikely to be observed when similar same-source

evidence is analyzed under similar conditions.

• Note 1: The finding of a statistically significant difference must be based on

comparison methods that account for (or consider) all necessary secondary

characteristics (e.g., environmental exposures).

• Note 2: Determination of a statistically significant difference must consider and

account for limitations of the technique(s) used and the items being compared.

• Note 3: The determination of a statistically significant difference requires that all

known or suspected sources of uncertainty are investigated and accounted for.

This includes uncertainty arising from both random and systematic sources of

measurement error that could affect the results.
11

Summary

• STG members have made many different types of statistical contributions

to OSAC work this year

• Many difficult topics that still require substantial statistical input

• Methods for comparing complex responses (e.g. FTIR of paint)

• Optimization of analytical methods (e.g. GC-MS for fire debris analysis)

• Methods for summarizing results and reporting conclusions

• Many others

• Need more embedded statisticians to cover all OSAC units with relevant

standards in development