Distribution-Free Statis

Methods for Biometric Perf

Evaluation 

Michael E. Schuckers 

St. Lawrence University 

Center for Identifcation Technology Research (CIT

schuckers@stlawu.edu 

aThis work is supported by NSF grants CNS-0325640 and CNS-0520990 which is 

National Science Foundation and the United States Department of Homeland Securi

Distribution-Free Statistica
Overview 

Goal: Develop inferential statistical methods FMR, FNMR without worrying about sampling those statistics. Note: We can use a central limit theorem for but that is another talk (SPIE 2010). 

Distribution-Free Statistica
Failure to Enrol (FT

Notation 

( 1 if individual 

i is unable to 

(1) 

Ei = 0 otherwise.

Failure to enroll (FTE) rate is 

P

[

i∈E

Ei (2) FTE = . NE

where 

NE is the total number of individuals w

to enrol in the system and 

E is the collection 

that tried to enrol. 

Distribution-Free Statistica
Failure to Acquire (FT

Notation 

 1 if the 

jth acquisition atte

(3) 

Aij = individual 

i is not acquir

 0 otherwise.

where 

a

i is the total number of attempts for the 

i

A represents all individuals who attempt 

biometric image collected. 

Failure to acquire (FTA) rate is 

P P

ai

[

i∈A

j=1

Aij (4) FTA = 

P

i∈A

a

i

Distribution-Free Statistica
False Non-Match Rate (F

Let 

Yiw,kw′ be the match score for comparing 

presentation from the 

ith individual to the 

w′th 

from the 

kth individual. 

Notation 

( 1 if 

i = k,

Y ′ iw,i′w > τ (5) 

Dikl = 0 if 

i = k,

Y ′ iw,i′w

≤

τ

False non-match rate (FNMR) is then 

P P

\ i l

Diil (6) FNMR = 

P , i

nii

where 

nii is the number of decisions on indiv

We assume some ordering of decisions 

l = 1

Distribution-Free Statistica
False Match Rate (FM

Let 

Yiw,kw′ be the match score for comparing 

presentation from the 

ith individual to the 

w′th 

from the 

kth individual. 

Notation 

(7) 

(

Dikl

=

0 1 

if if 

i

6= k,

Yiw,kw′

i

6= k,

Yiw,kw′

>

τ

≤

τ

False match rate (FMR) is then 

P P

P

\ i k

6=i l

Dikl

(8) FMR = P Pi k

6=

i

nik

where 

nik is the number of decisions on the o

individuals 

i and 

k. 

We assume some ordering of decisions 

l = 1

Distribution-Free Statistica
IID Bootstrap Revie

Suppose we have 

n decisions 

X

1,X

2,... ,Xn

Calculate statistic, say 

ˆ = ( ˆ

X

1,X

2,... ,

Sample 

n values from 

{

1, 2,... ,n

} with re

call those values 

b

1,b

2,... ,b

n. 

 Calculate and store ˆ 

∗ = ( ˆ

X

b

1 ,X

b

2 ,... ,X

Repeat the previous two steps some largtimes Use distribution of the 

ˆ

∗ to approximate 

distribution of 

ˆ. 

Only good if data are uncorrelated (FTE) but and improvements exist.... as we shall see. 

Distribution-Free Statistica
Correlation Structure: 

Uncorrelated Between Individuals Structure 

( 1 if 

i = 

i

′

(9) Corr

(

Ei,Ei′) = 0 otherwise

Then estimated variance is 

FTE

[

(1 

− FTE

[

) (10) 

V [ FTE

[

] = 

NE

where 

NE is the total number of enrolment at

Distribution-Free Statistica
Correlation Structure: 

Intra-individual Correlation 

 1 if 

i = 

i

′,j = 

 (11) Corr

(

Aij,A

i′j′) = 

ψ if 

i = 

i

′,j

6= 

 0 otherwis

Then estimated variance is 

"

[

Xn FTA(1 

− FTA

[

) (12) 

V [ FTA

[

] = 

NA + ψ a

N

2

A

i=1

P where 

NA = 

a

i and 

a

i is the number of dec

ith individual. 

Distribution-Free Statistica
Correlation Structure: F

Intra-individual Correlation 

 1 if 

i = 

i

′

, l 

 (13) Corr

(

Diil, D

i′i′l′) = 

ρ if 

i = 

i

′

, l 

 0 otherwi

"

\ \

X

FNMR(1 

− FNMR

)

V [ FNMR \ ] = 

NG + 

ρ

N

2

G

i

(14) 

P where 

NG = 

nii and 

n

i

i is the number of d

the 

ith individual. 

Distribution-Free Statistical 
Subsets Bootstrap: FNMR 

P P

ai i∈Aj=1

A

Calculate statistic, say FTA

[

= 

P 

ai i∈A

n

=| A | is the number of individuals and 

number of decisions on 

ith individual. 

Sample 

n values from 

{

1, 2,... ,n

} with re

call those values 

b

1,b

2,... ,b

n. 

∗ P P

a

bi i∈A

j=1

A

bij Calculate and store FTA

[

= 

P

i∈A

a

bi

Repeat the previous two steps some largtimes 

∗

Use distribution of the FTA

[

to approxim

sampling distribution of FTA

[

. 

Distribution-Free Statistical 
Correlation Structure: 

Two-instance Correlation OR 

ALL HELL BREAKS LOOSE 

 1 if 

i

=

i

′

, k

=

k′

, l

=

l′

 η if 

i

=

i

′ =

l′ , k

=

k′

 , l

6  ω

1 if 

i

=

i

′ =

k′ 

 , k

6 ,i

6= k,

i

 ω

2 if 

=

i

′ 

i

6 , k

=

k′

,i

6= k,

i

 (15) Corr(Dikl, Di′k′l′ ) =

ω

3 if 

i

=

k′

,i

′ =

i

′ 6= k,

i

6 ,i

 ω

3 if 

i

′ =

k′ = k,

i

6 ,i

′ 

 6= i,

k

 ξ

1 if 

i

=

k′ =

i

′ 

, k

=

i

′

,i

6 , k

 ξ

2 if 

i

=

k′ =

i

′ 

, k

=

i

′

,i

6 , k



0 otherwise 

Distribution-Free Statistical 
Correlation Structure: FMR 

 FMR

\

(1

− FMR

\)  X X

V [\

N

2 FMR] =



NI

+ η nik(nik

− 

I i=1

k=1

k

6=i





X X X   X X 

+

ω

1

nik



nik′

+

ω

2

i=1

k=1

k′=1 i=1

k=1 

k

6

k′6=i,k′6

k

6 

=i =

k

=i 



 X X X  X 

+

ω

3

nik



ni′i +

nkk′

i=1

k=1 i′=1

k′=1

k

6 i′6=i,i=i ′=6

k

k′=6 6 i,k′=

k

 X X X Xn

 (16) 

+

ξ

1

nki

+

ξ

2

nki(nki

− 1)



i=1

k=1 i=1

k=1

k

6=i k

=6 i

P P where 

NI

=

nik. i k

=6 i

Distribution-Free Statistical 
Two-instance Bootstr

Resample 

nP individuals from the list of al

in the probe and call those individuals 

b

1,b

Distribution-Free Statistical 
Two-instance Bootstr

Resample 

nP individuals from the list of al

in the probe and call those individuals 

b

1,b

For the 

bth

i individual selected in the previo

then resample from the 

nG individuals in th

call those individuals 

h

bi,k’s for 

k = 1,... ,n

Distribution-Free Statistical 
Two-instance Bootstr

Resample 

nP individuals from the list of al

in the probe and call those individuals 

b

1,b

For the 

bth

i individual selected in the previo

then resample from the 

nG individuals in th

call those individuals 

h

bi,k’s for 

k = 1,... ,n

Take all of the decisions 

D

bih

bi,k

1,... ,D

bih

b

the resampled pairs of individuals, (

b

i,h

bi,k

two previous steps. 

Distribution-Free Statistical 
Two-instance Bootstr

Resample 

nP individuals from the list of al

in the probe and call those individuals 

b

1,b

For the 

bth

i individual selected in the previo

then resample from the 

nG individuals in th

call those individuals 

h

bi,k’s for 

k = 1,... ,n

Take all of the decisions 

D

bih

bi,k

1,... ,D

bih

b

the resampled pairs of individuals, (

b

i,h

bi,k

two previous steps. The replicated value for FMR

\

is then give

P P n

P

n

G

P

m

bih

bi

\

,k

 

i=1

k=1

l=1

D

(20) FMR = P P n 

P

n

G

i=1

k=1

m

bih

bi,k

Distribution-Free Statistical 
Structure and Metho

Metric Distribution model Distribution-FTE Binomial IID FTA Beta-binomial Subset BootFNMR Beta-binomial Subset BootFMR ?? Two-instanc

Subset Bootstrap due to Bolle et al (2003) FMR model generalization of implicit structurTwo-instance Bootstrap Schuckers(2010) Central Limit Theorems apply to all cases. 

Distribution-Free Statistical 
Matcher Threshold νˆ sνˆ 

(FH,MLP) 0.0 0.0038 0.0007 

(DCTs,GMM) 0.0 0.0582 0.0067 

(DCTb,GMM) 0.2 0.0041 0.0007 

(DCTs,MLP) -0.8 0.1057 0.0086 

(DCTb,MLP) -0.5 0.0580 0.0072 

(LFCC,GMM) 3.0 0.0142 0.0039 

(PAC,GMM) 2.0 0.0570 0.0090 

(SSC,GMM) 1.0 0.0692 0.0105 

Variance estimation: F

XM2VTS Database 

sνˆ is the standard error from Schuckers (2009) and sν,boot ˆ is the boo

error. 

Distribution-Free Statistical 
Great! So What? 

Distribution-Free Statistical 
Great! So What? 

There’s that signifcance thing. 

Distribution-Free Statistical 
Great! So What? 

There’s that signifcance thing. And that sampling variability thing. 

Distribution-Free Statistical 
Great! So What? 

There’s that signifcance thing. And that sampling variability thing. Just cause one rate/curve is ‘better’ than doesn’t make it signifcantly better. 

Distribution-Free Statistical 
Great! So What? 

There’s that signifcance thing. And that sampling variability thing. Ju

st cause one rate/curve is ‘better’ than 

doesn’t make it signifcantly better. 

Ask about signifcance, confdence interv

Distribution-Free Statistical 
Great! So What? 

There’s that signifcance thing. And that sampling variability thing. Ju

st cause one rate/curve is ‘better’ than 

doesn’t make it signifcantly better. 

Ask about signifcance, confdence intervTo

ols now in place. 

Distribution-Free Statistical 
Beyond a Single Metr

Move to methods for: 

Comparing two metrics 

Distribution-Free Statistical 
Distribution-Free Statistical 

Beyond a Single Metr

Move to methods for: 

Comparing two metrics Comparing three or more metrics 
Beyond a Single Metr

Distribution-Free Statistical 

Move to methods for: 

Comparing two metrics Comparing three or more metrics Considering paired data collections 
Beyond a Single Metr

Distribution-Free Statistical 

Move to methods for: 

Comparing two metrics Comparing three or more metrics Considering paired data collections Considering independent data collection
Example:FNMR 

We are considering difference between two Fsame matcher but on different groups. Each decisions from 26 individuals. 

FNMR \

1 = 

0.0086 and 

FNMR \

2 = 

0.0214. 

Make 95% confdence interval for the differen

FNMR’s 

Distribution-Free Statistical 
Example:FNMR 

Bootstrap each FNMR and take difference at each boot

Frequency 

0 200 400 600 800 1000 1200 

−0.06 −0.04 −0.02 0.00 0.02 0.04 

diff 

The 95% CI for F NMR

1

− F NMR

2 is 

(

−

0.0384, 0.0

from 10000 bootstrap replicates. 

Distribution-Free Statistical 
Example:FMR 

Comparing equality of FMR for 5 matchers from BANCA upon 312 decisions. 

F

\M

R’s are 

0.0769, 0.0449, 0.0513, 0.0737, 0.1699

Are they signifcantly different? Formally, 

H0 : 

1

=

. . . 

−



5, H

1 : at least one different. 

Distribution-Free Statistical 
Example:FMR 

Calculate F test statistic (red line) for raw data. Bootstra

true and recalculate F 

∗. 

Bootstrapped F 

Frequency 

0 100 200 300 400 500 600 

0.000 0.005 0.010 0.015 0.020 

F 

P-value for this test is 

0.016. 

Distribution-Free Statistical 
−3 −1 1 3 

Theoretical Quantiles 

−3 −1 1 3 

Theoretical Quantiles 

T

Distribution-Free Statistical 

Sample Quantiles 

0.00 0.04 0.08 0.12

Sample Quantiles 

0.02 0.06 0.10 

Sample Quantiles

0.03 0.05 0.07 

Distribution of sampling distributions convergfor large sample sizes Example: Facial FMR (Data from Ross and J

† † 

n = 5 n = 10 

CLT 
Discussion 

All correlations are threshold dependent not 

Distribution-Free Statistical 
Discussion 

All correlations are threshold dependent not Basis for more complicated correlation st

Distribution-Free Statistical 
Discussion 

Distribution-Free Statistical 

All correlations are threshold dependent not Basis for more complicated correlation stParametric approach used for sample sizSchuckers (2009,2010) 
Discussion 

Distribution-Free Statistical 

All correlations are threshold dependent not Basis for more complicated correlation stParametric approach used for sample sizSchuckers (2009,2010) Needed for GLM’s, covariate approaches et al. for FNMR) 
Discussion 

Distribution-Free Statistical 

All correlations are threshold dependent not Basis for more complicated correlation stParametric approach used for sample sizSchuckers (2009,2010) Needed for GLM’s, covariate approaches et al. for FNMR) Computational Methods in Biometrics (Ju

Springer 
Discussion 

Distribution-Free Statistical 

All correlations are threshold dependent not Basis for more complicated correlation stParametric approach used for sample sizSchuckers (2009,2010) Needed for GLM’s, covariate approaches et al. for FNMR) Computational Methods in Biometrics (JuSpringer CITeR funded software PRESS v2.0 
Discussion 

Distribution-Free Statistical 

All correlations are threshold dependent not Basis for more complicated correlation stParametric approach used for sample sizSchuckers (2009,2010) Needed for GLM’s, covariate approaches et al. for FNMR) Computational Methods in Biometrics (JuSpringer CITeR funded software PRESS v2.0 
Schuckers (2010) 

Distribution-Free Statistical 

Computation Methods in Biometric AuthenticStatistical Methods for Performance EvaluatioText by Springer to appear Summer 2010 

Methods 

for FTE, FTA, FNMR, FMR, RO

Methods 

for one sample, two sample, thr

samples Distribution-free 

a

nd Large sample metho

Independent 

a

nd Paired data collections 
Thank You 

Questions? 

schuckers@stlawu.edu 

Distribution-Free Statistical