The Confidence Interval for the Likelihood Ratio

with Application to Biometrics

Larry Tang

George Mason University & National Institute of Standards and Technology

Joint work with Elham Tabassi (NIST) and Xiaochen Zhu (GMU)

1 / 19
Outline

I Evidence interpretation

I Error rates, receiver operating characteristic curve, and

likelihood ratio

I Confidence interval of likelihood ratio on NIST datasets

I Conclusion and future work

2 / 19
Evidence Interpretation

I The forensic source identification – inferential analysis to

identify the origin of a collection of forensic evidence

I Summarization of the observed evidence relative to the

prosecution and defense propositions

I Forensic scientists: interested in source level propositions,

sometimes activity level propositions

I Court system: offense level propositions concerning the guilt

or innocence of the defendant

3 / 19
Sources

Three subsets of objects:

es: Set of objects associated with a specified source

(person, window, ...)

eu: A set of trace objects from an unknown source

ea: Collection of sets of objects from alternative sources

4 / 19
Propositions and Sources

Hp: Same source

Specific source: 

es (ns traces)

Alternative 

sources: 

ea (na sources)

Hd: Different 

sources

M1: Sample from 

one specific 

source

M2: Sample from 

na +1 sources

Unknown source: 

eu (nu traces)

5 / 19
Approaches for Summarizing the Evidence

I Bayesian methods (Lindley, 1977) – trace evidence

I Two-stage approach (Parker, 1966) – trace evidence

I Score-based methods (Royall,1997) – pattern and trace

evidence

6 / 19
Two-stage Approach

Stage One:

Exclusion Stage

Exclusion Different sources

T(eu,es)<!

T(eu,es)≥!

Stage Two:

Atypicality Stage

Non-exclusion

7 / 19
Likelihood Ratio

I Error rates

I Exclusion: random non-match probability (RNMP), or chance

of incorrect non-match

I Non-exclusion (or inclusion): random match probability

(RMP), or chance of incorrect match

I Likelihood ratio (LR) based on binary decision of T(es

, eu) >

a threshold, τ:

LR =

Pr(T(es, eu) > τ|Hp)

Pr(T(es, eu) > τ|Hd )

=

1 − RNMP(τ)

RMP(τ)

I Similar to diagnostic LR: accuracy of a diagnostic test which

has positive and negative results

I Well-studied positive LR in diagnostic medicine:

sensitivity/(1-specificity)

8 / 19
Forensic Error Rates

I Similarity scores for the i

th within-source comparison:

Ts,i, i = 1, . . . , m follows Fθs

I Similarity scores for the j

th between-source comparison:

Td,j, j = 1, . . . , n, follows Fθd

I Random non-match probability:

RNMP(τ) = P(Ts,i ≤ τ) = Fθs(τ)

I Random match probability:

RMP(τ) = P(Ta,j > τ) = 1 − Fθd(τ)

9 / 19
ROC Curves for Forensic Error Rates

I ROC curve plots (1-RNMP) versus the RMP as the threshold

point τ for determining a “match” varies from −∞ to +∞.

I Let t be RMP(τ), and R(t) is 1 − RNMP(1 − t)

I ROC curve R(t):

R(t) = 1 − Fθs(F

−1

θd

(1 − t))

I The derivative of the ROC curve closely related to likelihood

ratio: the instantaneous change in the 1-RNMP in a unit

change of RMP

10 / 19
Relationship between ROC and LR

An illustration of the relationship between ROC and LR:

0.00

0.05

0.10

0 10 20 30 40

Score

Probability density

0.00

0.25

0.50

0.75

1.00

0.00 0.25 0.50 0.75 1.00

Random Match Probability

1−RNMP

Figure: Left panel: dash curve – different-source scores, solid curve –

same-source scores; right panel: solid black curve – ROC curve

11 / 19
Smooth ROC curve – Parametric Method

I Assume after Box-Cox power transformation, Fθs ∼ N(μs

, σ2

s

)

and Fθd ∼ N(μd , σ2

d

)

I RNMP and RMP:

RNMP(τ) = Φ(μs − τ

σs

), 1 − RMP(τ) = Φ(μd − τ

σd

)

I The resulting binormal ROC curve :

R(t) = Φ(μs − μd

σs

+

σs

σp

Φ

−1

(t))

I Explicit expression for LR estimate and its confidence interval

12 / 19
NIST SD4 Data

I NIST Special Database 4 (SD4)

I SD4 database contains 512-by-512-pixels gray scale fingerprint

images

I Two representations for each finger – rolled impressions of the

finger

I Bozorth matcher was run on all pairs of fingerprints from SD4

database

13 / 19
Histograms of NIST SD4 Data

0

1000

2000

3000

0 25 50 75 100

Log Likelihood Ratio

count

Histogram of LogLR for Genuine Matches

0

10000

20000

0 25 50 75 100

Log Likelihood Ratio

count

Histogram of LogLR for Imposter Matches

14 / 19
Confidence Intervals of Log(LR) for SD4

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

−5

−4

−3

−2

−1

−2 −1 0 1 2

Log Sample Size Ratio

Log Likelihood Ratio

Method

● PE

KDE

LRE

Log Likelihood Ratio and Confidence Interval 

when Sample Size Ratio Varies

Figure: PE – parametric, KDE – kernel density estimation; LRE – logistic

regression estimation (Zhu, Tang, Tabassi, 2017 IJCB)

15 / 19
Likelihood Ratios for Facial Recognition

I The Good, the Bad, and the Ugly Face Challenge Problem

(Phillips, et al, 2012)

I Frontal face images taken with a digital single-lense reflex

camera

I The data set has three categories, which are “good”, “bad”,

and “ugly”, based on the quality of the images

I The comparison scores measures characteristic difference

16 / 19
Confidence Intervals of Log(LR) for Facial Recognition

● ●

● ● ● ● ● ● ● ● ● ●

● ●

● ● ●

●

● ●

●

●

●

● ●

● ●

● ● ●

●

● ● ●

●

●

●

●

●

●

●

−1

0

1

2

3

−2 −1 0 1 2

Log Sample Size Ratio

Log Likelihood Ratio

Method

● PE

LRE

KDE

17 / 19
Conclusion and Future Work

I Sampling variability of likelihood ratio for fingerprint and

facial recognition data

I Paradigm for the reasoning about the source of traces based

on error rates

I Characterize the uncertainty about estimated forensic error

rates

18 / 19
Acknowledgement

I NIST forensic program

I National Institute of Justice (Collaborators: Danica Ommen,

Chris Saunders, Elham Tabassi, and Don Gantz)

19 / 19
Acknowledgement

I NIST forensic program

I National Institute of Justice (Collaborators: Danica Ommen,

Chris Saunders, Elham Tabassi, and Don Gantz)

Thank you!

19 / 19