Overview of 

NIST Statistical Research

in Forensic Science

Will Guthrie

NIST Statistical Engineering Division

will.guthrie@nist.gov
NIST Statistical Research in Forensics –

View from 105 m 

• 15 current research projects

• Staffing level between 3 and 4 FTE/year

• Projects a mix of:

• collaborative, individual 

• foundational, applied

• Key partnerships include:

• NIST staff working in other Forensic Science Focus Areas

• FBI, MSP, NFI, NFEA, IAI, FIU, CSAFE, OSAC
Research Areas

• New Reference Materials for Trace Elements in Glass

• Uncertainty of Drug Mass Measurements

• Optimization of GC/MS for Fire Debris Analysis

• Challenge Problem: Statistical Comparison of Paint Spectra

• Complex DNA Mixture Interpretation

• Characterization of Noise in Next Generation Sequencing Data

• Use of Next Generation Sequencing for DNA Mixture Analysis

• Assessment of Thresholds for CE STR Profiles
Research Areas

• Error Rate Assessment for Firearms ID

• Uncertainty Budget Framework for Automated 

Firearms Examination

• Quantitative Understanding of Uniqueness and 

Reproducibility of Firearm Toolmark Surfaces

• Reference Population Data for Firearm Toolmarks 

• Likelihood Ratios as Weight of Evidence

• Calibration of Likelihood Ratios

• Quantitative Evaluation of Footwear Evidence 
Challenge Problem: 

Statistical Comparison of Paint Spectra

• Working with an OSAC TG and NIST chemists 

to design a challenge problem on comparison 

of paint spectra in forensic settings 

• hit and run car accidents, other crime-related 

scenes with paint traces left behind

• Goals

• Promote community development of new 

statistical or machine learning algorithms

• Comparison of algorithm performance on 

common test data
Challenge Problem: 

Statistical Comparison of Paint Spectra

• Importance

• Current methods focus on visual comparison and 

rely on analyst expertise and judgment

• Statistical or machine learning methods have 

potential to be more consistent across labs 

• Status

• Paint spectra collected from multiple cars

• Assessment of algorithm comparison methods 

currently underway

• Stay tuned for more news!

Statistical Models for Similarity Score 

Comparisons in Firearm Evidence 

Identifications

Nien Fan Zhang

NIST Statistical Engineering Division
Outline

1. Introduction

2. Statistical models for CMC measurements

3. Estimating model parameters

4. A practical example

5. Discussions and conclusions

6. References

9
3-D topographic images of breech face impressions obtained from a pair of cartridge 

cases ejected from the same firearm slide.

10

1. Introduction
To estimate error rates, the key is to find appropriate probability distributions for 

the relative frequency distribution of the observed CMC values.

11

The congruent matching cells (CMC) method is used to compare pairs 

of topography images of breech face impressions and provide a basis 

for estimating error rates. 
12

2.1 Model 1 for CMC measurements

Binomial distribution

• Matches between each of the N1 cell pairs from first image pair, X1j, are the 

outcomes of a sequence of independent Bernoulli trials with: 

PP XX1jj = 1 = pp PP XX1jj = 0 = 1 − pp XX11, ⋯ , XX1NN1

is the number of CMCs for the first image pair.

= �jj=0

NN1

XX1jj ⇒ PP[1] YY = kk =

NN1

kk ppkk(1 − pp)NN1−kk

• To estimate p, we use an independent sequence of M image pairs each with 

potentially different values of Ni but with the same value of p.

Y1

Y1
13

2.2 Model 2: Correlated binomial distribution

• Matches between each of the N1 cell pairs from first image pair, X1j, are 

the outcomes of correlated Bernoulli trials with: 

• To estimate the parameters, we use an independent sequence of image 

pair comparison results with a total of M images . The cell 

numbers for different image pairs can be different.

1 ( 1) PX p j = = 1 ( 0) 1 PX p j = = −

2

(2) 1 1 Cov( , ) i j r XX = σ

[2] 1 [1] 1 (2) 2 1 P Y P Y r gYp ( ) ( ){1 ( , )} = +

1

1 1

0

N

j

j

Y X

=

= ∑ ⇒

1Y Y ,,, M

Ni
14

2.3 Model 3: Beta-binomial distribution

• Matches between each of the N1 cell pairs from first image pair, X1j, 

are the outcomes of independent Bernoulli trials with: 

PP XX1jj = 1 = pp PP XX1jj = 0 = 1 − pp XX11, ⋯ , XX1NN1

• For different image pairs, they are independent and is random with 

a beta probability distribution, i.e., .

• M independent image pairs are used to estimate the parameters. 

p

1 1

1 1

(, ) ( | ,,) (,)

N Bk N k PY k N

k B

α β α β α β

  + −+

= =    

p Beta  (,) α β

1

1 1

0

N

j

j

Y X

=

= ∑
15

2.4 Model 4: Beta-correlated binomial distribution 

• Matches between each of the N1 cell pairs from first image pair, X1j, are 

the outcomes of correlated Bernoulli trials with: 

• For different image pairs, they are independent and is random with a 

beta probability distribution, i.e., 

• M independent image pairs are used to estimate the parameters. 

p

1 1 1

[2]

1 1 (2)

0

( , ) (1 ) ( | ,,, ) (,)

P kpp p P Y k N r dp B

α β

α β α β

− − − = = ∫

1 ( 1) PX p j = = 1 ( 0) 1 PX p j = = −

2

(2) 1 1 Cov( , ) i j r XX = σ

p Beta  (,) α β

1

1 1

0

N

j

j

Y X

=

= ∑
3. Estimating the parameters of the models

• CMC method applied to a set of cartridge cases to have certain known 

matching (KM) image pairs and certain known non-matching (KNM) 

image pairs. 

• Sum of CMC values for M independent image pairs,

• Likelihood function for corr. Binomial,

The ML estimator of and are obtained when log (L) reaches the 

maximum. 

1 1, , ,..., N MNM

Y Y

[2] (2) (2) 2

1 1

( | , ) (1 ) {1 ( , )}. i ii

M M

i y Ny

i i

i i i

N

L P y pr p p r g y p

y

−

= =

 

= = −+     ∏ ∏

p (2) r

16
17

Nonlinear regression can be used to estimate parameter(s).

• Assume all the CMC values for each image pair is binomial distributed, 

By the law of large number, the frequency curve approaches the binomial 

mass function when . 

• For a sample from an independent sequence , call 

for 

a generalized frequency function. Assume there are L distinct N’s. 

1 1, , { ,..., } N MNM

Y Y

,

1 in ( )

i Ni

M

Y k

i

M

number elements sample k f k M M

=

= =

= =

∑1

0,max{ }i k N =

M → ∞

(,) Y Bin N p i  i M =1,...,
18

Assume the CMC values 

Assume for each distinct N, , there are 

indicators with same . 

Under some regular conditions, by

the law of large number, when 

almost surely 

where . Thus, we have a nonlinear regression model based on 

the correlated binomial distribution

M → ∞

W CM l l =

( 1,..., ) l L =

,

1

1

( ) corr. ( , , )

i N j

M

Y k L

i

M ll

l

f k W Bin k Nd p M

=

=

=

= → ⋅ 

∑

∑

1

corr. ( , ) Y Bin N p i i 

[1], (2) 2

1

( ) ( ){1 ( , , )} l

L

M l Nd l

l

f y W P y r g y p Nd ε

=

=⋅ + + ∑ 0,max{ }i y N =

Ndl

Ndl

1

L

l

l

C M

=

∑ =

Cl
4. A practical example

The Weller data set of cartridge cases has 370 known matching (KM) image pairs, 

but not with same N . The 370 columns representing 370 image pairs. There are L = 15 distinct 

N’s. 

Based on the original data, the CMC values and the corresponding counts are given by

The generalized frequency function is obtained by count/370 since there are 370 image pairs. 

Note the sum of counts = 370.

CMC 

(Y)

21 22 23 24 25 26 27 28 29 30 31 32 .... 43 44 45 46 4

7

Count 1 0 1 2 3 0 7 5 2 14 5 15 .... 16 6 2 1 1

CMC 

(Y)

36 27 37 36 36 37 38 37 27 40 37 36 .... 44 41 37 37 4

3

N 38 38 38 38 38 38 38 38 41 41 41 41 .... 46 46 43 43 4

5

( ) Cj

( ) Cj
20

The M =370 image pairs have L = 15 distinct N’s. The generalized 

frequency function will approach a pmf which is a mixture or a 

weighted mean of 15 pmf’s of the underlying distribution when 

. M → ∞.
21

5. Discussions and conclusions

• We discussed four statistical models for the similarity score comparisons in 

firearm evidence identifications based on pass-or-fail tests, specifically the 

CMC method. In this case, binomial model seems a default or generic one. 

• Because the assumption of independence among the cell pair comparisons 

from the CMC method is most likely invalid in practice, the correlated 

binomial model was proposed to relax the assumption.

• Although using the beta-binomial distribution can relax the assumption of 

the same p for all image pair comparisons, it still assumes that within each 

image pair, all cell pair comparisons are independent. Thus, the betacorrelated binomial was proposed.
22

• From the point view of the number of assumptions, the beta-correlated 

binomial, beta-binomial, and correlated binomial models are better than 

the binomial model.

• Using the smallest sum of squares of difference between the generalized 

frequency function and each of the four mixture pmf’s as a performance 

criterion, we conclude that the correlated binomial, beta-binomial, and 

beta-correlated binomial models fit the example KM data much better 

than the binomial model. 
References

1. J. Song, T. V. Vorburger, W. Chu, J. Yen, J. A. Soons, D. B. Ott, and N. 

F. Zhang (2018) Estimating error rates for firearm evidence 

identification in forensic science, Forensic Science International, 

284, 15-32.

2. N. F. Zhang (2019) The use of correlated binomial distribution in 

estimating error rates for firearm evidence identification, Journal 

of Research of the National In statute of Standards and 

Technology, 124, Article No. 124026.

3. A. W. van der Vaart (1998) Asymptotic Statistics, Cambridge 

University Press, Cambridge, UK, 265-266. 

23

NIST Footwear Impression 

Comparison System

Steve Lund

Statistician

Statistical Engineering Division

steve.lund@nist.gov
Acknowledgements

FBI Examiners

Brian McVicker

Mike Gorn

Funding provided by

NIJ Award DJO-NIJ-17-RO-0202

NIST SPO

NIST ITL

Hari 

Iyer

NIST Footwear Research Team:

Martin 

Herman

Vighnesh 

Hegde

Adam 

Pintar Gautham 

Venkatasubramanian Günay

Doğan
Version 0 - Workflow

Manual Annotation

Test Impression

Questioned Impression

Automated Alignment and Comparison

Pattern: 

Design, Size, 

and Wear

RACs

Final

+
Revamp with Casework Focus

Staged Crime Scene Impressions Test Impression 

Lineup
Test Impression Markup

RACs Wear
Region of Interest

Crime Scene Markup

RACs Contact
Crime Scene Markup

How confident do I feel that my markup will mirror a 

test impression from this shoe?

Red: I can hardly 

see anything

Orange: I can 

generally tell 

contact apart from 

noncontact

Yellow: I can even 

see the edges 

clearly

Green: I think I 

could see RACs!

Blue means 

ignore
Alignment

• First conduct a rigid 

alignment (slide and rotate, 

like doing a puzzle)

Rigid Alignment
Alignment

• First conduct a rigid 

alignment (slide and rotate, 

like doing a puzzle)

• Then allow moderate 

stretching to allow for 

distortions between 

impressions

Flexible Alignment
Alignment

• First conduct a rigid 

alignment (slide and rotate, 

like doing a puzzle)

• Then allow moderate 

stretching to allow for 

distortions between 

impressions

• Promising results with 

partial impressions 

representing as little as 20% 

of the outsole surface
• Contracting boundary 

K size > Q size

• Expanding boundary 

K size < Q size

• Stable boundary  depends on 

shoe manufacturing process 

Size Metric
Results from Close Non-Match 

Reference Comparisons • Several Similarity Metrics

• Agreement of contact and noncontact regions and closeness of 

edges (Alignment score)

• ResNet (Version 0)

• Pixel-specific contact probabilities

• Each metric can be placed in 

different contexts...

Design and Wear Metrics

Could Q and K 

have come from 

the same shoe?

Similarity

Results from Mated Reference 

Comparisons

Could Q and K be from 

different shoes with 

the same size, side, 

and design? 

Results from Reference Comparisons 

with Half Size Difference Current Case

Could Q and K be from 

shoes with the same 

side and design but a 

different size? 
RAC Metrics

Any apparent RACs seen in the 

questioned impression are 

marked

After alignment, use overlap %

(a.k.a. Intersection over union) 

with any RAC regions marked in 

test impression

Crime Scene

Test Impression
RAC Metrics

Developing RAC metric that 

uses regions marked in test 

impression to initiate local 

pattern comparison

Crime Scene

Test Impression

Credit: Pixabay
Relevance Metrics

Credit: Pixabay

Crime Scene Clarity

Feature Density in Known 
Relevance Metrics

Credit: Pixabay

Crime Scene Clarity

Feature Density in Known 

Case Comparison
Relevance Metrics

Credit: Pixabay

Crime Scene Clarity

Feature Density in Known 

Case Comparison
Current 

Case

Same Shoe

0

+ 

1

2

-

1

2

+1

-1

+1 1

2

- 1 1

Size difference (Q

2

-K)

Similarity Score

Data Visualization

Credit: Pixabay
Credit: Nick Youngson of NYPhotographic.com

• Develop automated tools to assist with 

markup

• Use images from black box studies to 

evaluate and improve FICS

• Use NIST FICS outputs and black box 

responses to predict distribution of 

conclusions across examiners

A New Statistical Procedure to 

Assess Calibration Accuracy of 

Likelihood Ratio Systems

Hari Iyer

Statistician

Statistical Engineering Division

hari@nist.gov
Joint work with Prof. Jan Hannig

University of North Carolina – Chapel Hill &

Faculty Appointee, SED/ITL/NIST
• I would like to thank Steve Lund and William Guthrie for many useful 

discussions on topics related to performance assessment of LR systems. 

• Points of view are of the presenter and do not necessarily represent the 

official position or policies of the National Institute of Standards and 

Technology.

• Certain commercial entities are identified in order to specify experimental 

procedures as completely as possible. In no case does such identification 

imply a recommendation or endorsement by the National Institute of 

Standards and Technology, nor does it imply that any of the entities 

identified are necessarily the best available for the purpose.

Acknowledgements and Disclaimers
OUTLINE

• Likelihood Ratio (LR) & LR Systems

• Calibration accuracy of LRs

• Use of validation data in judging LR calibration accuracy

• Proposed method for inference regarding LR calibration discrepancy

• Example(s)
LLLL =

PPPP EE HHpp, II

PPPP EE HHdd, II

Hp: DNA from POI is in the crime sample

Hd: DNA from POI is not in the crime sample

I = Background information available prior to examining 

crime sample (known or assumed to be true) 

Likelihood Ratio
LLLL =

PPPP EE HHpp, II

PPPP EE HHdd, II

Hp: Glass fragments recovered from POI’s clothing is from 

the broken glass door at the scene of the burglary

Hd: The recovered fragments are not from the crime scene 

broken glass door

I = Background Information available prior to examining 

sample obtained from the POI’s shirt

Likelihood Ratio
Likelihood Ratio Systems

 Measurement

• Sample processing & Numerical quantification of 

informative features

 Interpretation

• Statistical/Mathematical Modeling, data analysis, expert’s 

judgements
Likelihood Ratio Systems (DNA Evidence)

Sample

Strength of 

Evidence

(LR)

Measurement EPG
Likelihood Ratio – What is it in plain English?

Suppose a forensic expert determines that an LR value of 1000 provides 

the best assessment of the value of the evidence.

When asked to explain what LR = 1000 actually means, they might say 

something like

The evidence is 1000 times more likely if Hp is true 

than if Hd is true (or something to this effect)
Meaning of Calibration Property of LR Systems

Is there empirical data to support the expectation that the system 

used by the expert is accurately assessing the value of evidence?

When considering RELIABILITY of expert witness 

testimony (or report), a key question is:

• What does empirical validation data say about ‘how much more 

likely the evidence is IF Hp is true than IF Hd is true? 

• How does this compare with the expert’s assessment?
Some Available Diagnostic Tools

Turing’s Lemma:

If the LR system is providing proper probabilistic assessments of value of 

evidence, then

the average of LR values obtained from Hd true cases must equal 1. 

(this is a necessary condition but not sufficient)
Calibration discrepancy is assessed as part of an overall prediction accuracy analysis by 

comparing posterior odds (across all possible prior odds) obtained using expert’s LR. 

Grzegorz Zadora, Agnieszka Martyna, 

Daniel Ramos, Colin Aitken

An R-package called comparison can be used to 

apply their method

David Lucy, James Curran, Agnieszka Martyna

Some Available Diagnostic Tools

Empirical Cross-entropy Analysis (ECE Plots)
Our Approach

LR systems that provide accurate probabilistic assessments of 

value of evidence (relative to empirical observations) must 

satisfy the following property:

LR of LR = LR

If gg xx is the probability density function for Hp-true LRs 

and

ff xx is the probability density function for Hd-true LRs

then

gg xx

ff xx

= xx for all xx
LR value of x is x times more likely to occur 

under Hp than under Hd. 

• LR value of 1 is equally likely under Hp as it is under Hd

• LR value of 10 is 10 times more likely to occur under Hp than it is under Hd.

• LR value of 100 is 100 times more likely under Hp than it is under Hd.

• LR value of 0.1 is 10 times more likely under Hd than it is under Hp.

What does LR of LR = LR mean?
Noncontributors LR

Contributors LR

Calibration Accuracy: Empirical Assessment
Calibration Accuracy: Empirical Assessment

Noncontributors LR

Contributors LR
Calibration Accuracy: Empirical Assessment

Noncontributors LR

Contributors LR
A Different LR system

Fails Calibration Check

Calibration Accuracy: Empirical Assessment
Our Approach

Rather than compare frequencies of occurrence of a given LR value in 

Hp-true cases and Hd-true cases we focus on frequencies of LR values 

that fall within any specified interval.

For instance, we can count the number of LR values that fall in the intervals 

[1,100], [100,10000], [10000, 1000000], etc.

Such counts in Hp-true cases and Hd-true cases must exhibit a relationship 

that is dictated by the property “LR of LR = LR”. 

GG bb − GG aa = bb FF bb − aa FF aa − ∫ FF xx dd

aa

bb

GG xx is the cumulative distribution function for Hp-true LRs

FF xx is the cumulative distribution function for Hd-true LRs
Our Approach

For any given interval of interest, say (a, b), we count

(1) the actual counts for Hp-true samples in the interval

(2) expected counts if the LR system makes accurate weight 

of evidence assessments using 

and calculate the ratio of (1) to (2). This is the estimated 

calibration discrepancy.

• If the ratio is less than 1 (log10 of the ratio is negative) then the LR system is 

overstating the evidence in the interval being considered.

• If the ratio is greater than 1 (log10 of the ratio is positive) then the LR system is 

understating the evidence in the interval being considered. 

We can visualize the results in an “Interval specific calibration discrepancy plot” 

or simply “calibration discrepancy plot”.

GG bb − GG aa = bb FF bb − aa FF aa − ∫ FF xx dd

aa

bb
Glass Fragments Example

This example is discussed in the book by Zadora et al. (2013) 

(see Chapter 4, Section 4.4.6). 

• Twelve fragments of glass were obtained from each of 200 glass objects and 

each fragment was subjected to an elemental analysis using scanning electron 

microscopy electron diffraction (SED-EDX). 

• Eight elemental concentrations were measured: 

Na, Mg, Al, Si, K, Ca, Fe, and O. 

• Base 10 logarithms of the ratios of the first 7 elemental concentrations to the 

concentration of O (Oxygen) formed the response variables in the analysis. 

• Zadora et al. (2013) used a graphical model approach and kernel density 

estimation for computing the same-source LRs and different-source LRs 

(consult their book for further details). 

• Here we focus on assessing how well-calibrated this particular LR system is.
Ink Example

Nmated = 200

Nnonmated = 19,900
Ink Example
Ink Example

Zadora, Martyna, 

Ramos, Aitken
Ink Example
Ink Example

Overstatement by a factor of 6

Overstatement by a factor of 125

Overstatement by a factor of 30
Summary

• CALIBRATION property is an important component of LR 

system assessment as it focuses directly on the accuracy of 

value of evidence assessments.

• We outlined a new method for examining potential degree 

of discrepancy between 

(a) evidential value inferred from validation data and

(b) evidential value as assessed by an LR system (or an 

expert).

• We illustrated the ideas using an example.

Backup

SLIDES