© L-1 Identity SolutionsTM

Biometric Identification: Metrics & Models

Brian Martin 

March 2010
Overview

• Back of envelope approach to biometric ID systems – and in 

particular the Search Engine Backend:

– What are (some of) the relevant metrics?

– Can we use what we measure to model performance? How? 

• If interested in a more precise examination of the topic (free dl):

– “National Biometric Test Center Collected Works 1997-2000”

Dated, but still many relevant points for 1:N search systems

 Wayman, 2000

 http://www.engr.sjsu.edu/biometrics/publications.html

– “Matching Performance for the US-VISIT IDENT System Using Flat 

Fingerprints (NIST IR-7110)”

One of few published results on large scale testing

 Wilson, Garris, & Watson, 2004

 ftp://sequoyah.nist.gov/pub/nist_internal_reports/ir_7110.pdf

2

© L-1 Identity SolutionsTM
3

© L-1 Identity SolutionsTM

Metrics
Biometric Search Engine Metrics

• There is an analogy between biometric ID search engine tradeoffs and the project management triangle: 

“Good, Fast, Cheap... Pick any two”

• There are arguably several other dimensions... these seem to 

capture the general concepts of most

• All axes are interrelated... 

4

© L-1 Identity SolutionsTM
Fixed cost and accuracy

-Hardware constrained 

solution such as access 

control. 

-Speed varies with users, 

database size, etc... 

Fixed cost and speed

-Accuracy is determined 

by investment cost and 

time

- Perhaps MBGC is an 

example. Cost and 

speed are determined by 

participant.

Fixed accuracy and speed

- RFP sets requirements for 

speed and accuracy

- Cost is the dependent 

variable 

Biometric Search Engine Metrics

5

© L-1 Identity SolutionsTM
Biometric Search Engine Tests

• There are many types of biometric system performance 

tests. Some of the most common:

1. Document and explore current state of the art

 Only test accuracy (“MBGC”, PFT, etc...)

 Sometimes accuracy and speed (IREX, ELFT, FVC)

 Helps answer, “Theoretically, can biometrics provide a solution to a 

particular problem?”

2. Validate existing system performance

 Accuracy, speed and cost are considered

3. Collect data that suggests future system performance 

(procurements)

 Will a system meet requirements at a smaller scale?

 How much will the final, larger, system cost? 

 Accuracy, speed, cost, and a model for scaling

6

© L-1 Identity SolutionsTM
7

© L-1 Identity SolutionsTM

Metrics: Time
Metric: Time

• Processing time (enroll and match) depends on

– Data quality characteristics 

– Imposter / Genuine (strength of match)

– Database size

– HW dependencies

 CPU bound

 Memory bound

 Instruction set support (SSE, NUMA, etc...)

 Scaling approach (multi-core, system architecture, etc...)

8

© L-1 Identity SolutionsTM
Metric: Time

• Two basic flavors of the time metric:

– Latency – how long 

 Real time metric

– Throughput – how much (average rate)

 Batch processing

 Easier to optimize than latency

9

© L-1 Identity SolutionsTM

Template

Generate

Feature

Detection

Search

Prep

Search Report

Δt

Δt = ΔT/N

Initialize

Overhead

Template

Generate

Feature

Detection

Search

Prep

Search Report

Template

Generate

Feature

Detection

Search

Prep

Search Report

Template

Generate

Feature

Detection

Search

Prep

Search Report

parallelism
Metric: Time

• Measure

– Latency and throughput

• Should document 

– Database size

– Data quality aspects

– Imposter / Genuine distribution 

– Hardware description

– One-time overhead, measures at different gallery sizes 

– Architecture (multi-core, multi-server) overhead

10

© L-1 Identity SolutionsTM
11

© L-1 Identity SolutionsTM

Metrics: Accuracy
Metric: Accuracy

• Accuracy depends on

– Algorithmic sophistication 

 Feature detection

 Feature matching 

– Biometric sample quality

• Most independent tests do an excellent job measuring 

accuracy on specific database samples

– Most tests become dated quickly

12

© L-1 Identity SolutionsTM
Metric: Accuracy

• Measure for 1:1 

– Tradeoff between FMR and FNMR (aka FAR and FRR) 

 ROC or DET curve 

– FTE 

– Examples: IREX, MINEX, PFT, FVC

• Measure for 1:N 

– Tradeoff between FPIR and FNIR (aka FAR and FRR, FMR and FNMR, 

Selectivity, „Alarm‟ rates)

 Use open set

 Measures depends on result list size. Here we assume list size of 1.

 Names of the metrics seem to drift from document to document – why?

 FNIR~FNMR1:1

 FPIR~FMR1:1 x NDB

– FTE

– CMC (hit rate) useful when every search is reviewed by human (latent)

– Examples: (FpVTE, FRVT, ICE), ELFT

• Also should document the FTA rate if possible

13

© L-1 Identity SolutionsTM
14

© L-1 Identity SolutionsTM

Metrics: Cost
Metric: Cost

• The cost metric is a direct reflection of

– Hardware required for the solution = system 

footprint

 Depends on computational efficiency 

(speed, size) 

 Engineering sophistication of matcher

 System architecture 

– Maintainability (Power, Cooling, Support, 

etc...)

– Human review workload (accuracy 

dependent)

• Cost usually not an independent variable 

in testing. 

• Cost can be reasonably estimated if the 

system architecture and other metrics are 

understood 

– Modeling cost can be non-linear with project 

scale (both ways!)

15

© L-1 Identity SolutionsTM
16

© L-1 Identity SolutionsTM

ID System Modeling
ID System Models

• Given test results, how do we use measurements?

– Model accuracy, speed, and cost for some other system

• Can we keep things simple? 

17

© L-1 Identity SolutionsTM
ID System Models – one problem

• Real world „black-box‟

– Meets the requirements

– Larger databases

– Minimize cost = efficient

18

© L-1 Identity SolutionsTM

• Lab test „black-box‟

– Configured for best accuracy

– Smaller databases

– Maximize cost/time to limits
ID System Models

• Lab testing can be useful, but the difference between 

the black-box in the lab and the black-box AFIS search 

engine must be acknowledged 

– Measuring more than accuracy helps bring this to light

• Now, lets look at a couple models for biometric search 

engines and examine how we can use test results

19

© L-1 Identity SolutionsTM
Model: 1-to-N = N x (1-to-1)

• The most common assumption for biometric identification 

systems is that 1:N = N x (1:1)

– Rarely the case in practice

– Easy to model and can provides good „back of envelope‟ 

estimations for simple (small) ID systems

• Latency ≈ N x 1:1 latency

• Throughput ≈ 1/Latency

• System size estimated by:

– Number of CPU cores directly calculated from throughput or 

latency

– Amount of RAM required calculated from template size

• Accuracy modeling previously presented (BSYM06)

20

© L-1 Identity SolutionsTM
Model: 1-to-N = N x (1-to-1)

• The 1:1 ROC curve can be used to estimate the CMC and alarm curves

21

© L-1 Identity SolutionsTM

TMR

FMR 0.0001 0.01 1.0

Select FMR = 1/NDB
0.000001

(at least don‟t abuse it) 

Model: 1-to-N = N x (1-to-1)

22

© L-1 Identity SolutionsTM

• Avoid this very tempting technique... !
Model: 1-to-N = N x (1-to-1) FUSION

• The ROC can also be used to model multi-modal fusion

– Needed: 2 ROC curves of uncorrelated biometrics and a calculator

• This result of OR rule fusion is easy to derive

– At each operating point, there are 2 scores S1 and S2

– Chance the person is rejected is the probability of both scores being 

below the threshold (sum the probabilities of independent events)

 FNMR1|2=(FNMR1)x(FNMR2)

– The person is falsely matched when S1 > Tor S2 >T or when S1 and S2 are 

BOTH > T. This is 1 minus probability that the person is correctly rejected. 

This happens when BOTH S1 and S2 are correctly rejected. Again this is 

the sum of probabilities of independent events.

 FMR1|2 = 1 – (TNMR1)x(TNMR2) = 1 – (1 – FMR1)x(1 – FMR2) 

 At low FMRs, FMR1|2 ≈ FMR1+ FMR2

23

© L-1 Identity SolutionsTM
Model: 1-to-N = N x (1-to-1) FUSION

• Example from NIST IR-7346 fusion study.

• Face and Finger fusion compared to score based result:

24

© L-1 Identity SolutionsTM 0.0001

0.001

0.01

0.1

1

0.0001 0.001 0.01 0.1 1 FRR

FAR

Finger

Face

O R-NIST IR7346

O R-Model

FMR

FNMR
Model: 1-to-N = N x (1-to-1)

• Where does this model based on 1-to-1 matching 

breakdown?

– Timing

 Scaling behavior not clear 

• One-time latencies

• Threading efficiency

• Scoring overhead

– Accuracy

 When gallery normalization is used

 When multiple matchers are used selectively

• The simple model doesn‟t handle advanced matching 

approaches which better scale to large DB sizes. 

• The next model shows why things aren‟t so simple.

25

© L-1 Identity SolutionsTM
Model: 1-to-N

• One main issue with the previous model is the assumption 

that the a single 1:1 match event is repeated for 1:N search. 

This is typically not true for large scale, or high throughput 

systems.

• Most modern biometric identification systems employ a 

multi-stage matching approach for improving speed – this 

breaks our simple model 

26

© L-1 Identity SolutionsTM

Error Rate

Cost Time

Error Rate

Cost Time

Error Rate

Cost Time
Model: 1-to-N

• Multi-stage matching breaks the Nx1-to-1 model

• Two examples of multi-stage matching

1. Incremental

 On each match attempt, effort depends on

• Sample quality

• Preliminary evaluation of the likelihood of match

2. Multi-pass

 Rank or filter all matches, apply additional matching effort 

on most promising candidates 

• Good for combining very different matching approaches 

including filtering and multi-modal

• A cascade of the previous Nx(1:1) model

27

© L-1 Identity SolutionsTM
Multi-Pass Matching

• Two pass face identification system

28

© L-1 Identity SolutionsTM

CMC (N=100,000)

Pass 1

Pass 2

Better CMC

Speed : modeled if pass % is known 

and latency between passes

Accuracy : non-trivial interaction between passes

due to correlations

- some hits lost in early pass

- some hits gained in later pass

Identification Rate
face

Incremental Matching

• Increase intensity of matching as needed 

– Increment amount of data examined (minutiae, 

iris code bits, Eigen face coefficients, etc...)

– Incremental algorithmic intensity

29

© L-1 Identity SolutionsTM

Speed : depends on data

Accuracy : some loss, mostly in 

non-relevant operating region,

not trivial to model

iris finger

Face image from www.cs.princeton.edu/~cdecoro/eigenfaces
Multi-Stage Matching

• Unfortunately there is no good generalized „black-box‟ 

or „gray box‟ model

– Matching speed for large systems not easily predictable

– Accuracy on different data not easily predictable

• Therefore, one needs some understanding of how the 

system works for relatively accurate modeling of larger 

systems

– Also requires empirical measurements for several parts of the 

system separately from the whole system

30

© L-1 Identity SolutionsTM
Other Considerations to Keep in Mind

• Search engine architecture matters

– The models presented here do not take into account the 

workload distribution over several match servers

 Synchronizing, collation of results, etc..

 How does architecture scale matchers?

• Divide and conquer – per thread, per machine?

• Parallel search – how many at once? Efficiency?

– System overhead can overwhelm individual matcher timing 

31

© L-1 Identity SolutionsTM
Take Home Messages

• Performance testing of a biometric search engine is 

multi-faceted, not just accuracy

• We can use simple 1:1 measurements to get very rough 

estimates for small biometric ID systems

– In general, everything is much more complicated though

• Use caution when trying to extending lab testing results 

to other (larger) system requirements

– Please don‟t assume anything – talk to vendors about how 

their system scales if needed

32

© L-1 Identity SolutionsTM
33

© L-1 Identity SolutionsTM

Thank You

bmartin@L1ID.com