Biometric Testing 

And 

Performance Extrapolation 

NIST IBPC - Gaithersburg, March 4, 2010 

Jean-Christophe.Fondeur@sagem.com 
Biometric Testing & Performance Extrapolatio

NIST IBPC - Gaithersburg, March 4, 2010 

Introduction Biometric testing 

• Objectives• Test p rotocole• Test database• Test criteria

Performance extrapolation 

• Accuracy• Sizing

Conclusion 
Testing Large Scale Biometric Systems 

Observation: Test results 

• Limited • Potentially biased 

Focus on 

Operational system • Large scale • Complex 

We need: •

To characterize h te i the target s

• A reliable observation 

=> Test protocole, database, 

• A way to extrapolate from 

• Accuracy & Sizing 

• A risk analysis approach 

• No testing is perfect ... 

• large scale back end systems (capture, human factors not in the scope) • Biometric aspects (architecture, security not in the scope) 

NIST IBPC - Gaithersburg, March 4, 2010 
Testing overview 

Interpret 

results Define test Prepare test Run test 

• objectives objectives • Tuning • Tuning 

• Blind test 

• Protocole • Protocole 

• Database • Criteria 

• Qualitative 

analyze 

• Extrapolate 

results 

NIST IBPC - Gaithersburg, March 4, 2010 
Biometric Testing: Objectives 

Testing can have many objectives

• Check feasibility • Estimate system performance, cost, risk 

• Accuracy, HW sizing, operator workload, ... 

• Select best technology provider p • Make some key system design choices 

• Mono/multi modal, enrolment workflow, 

Test shall be explicitely designed for the des

• Impact on test protocole, type and size of database, ... • Specific sub test may be required to reach some of th

NIST IBPC - Gaithersburg, March 4, 2010 
Test Protocole 

Test Protocoles are defined from 

• Objectives of the test 

• « Why are we doing this test » 

• Operational requirement 

• « What the system should do » 

• Risk assessment assessmen t

• « What are the main risks ? » 

Several subtest scenarii are often necessary 

• Accuracy, throughput, resistance to fraud & erors, ... 

No test will ever be perfect, biases must be identified and 

NIST IBPC - Gaithersburg, March 4, 2010 
Test preparation: Database 

Test database must be 

• Representative of system scenario 

• Acquisition conditions and workflow, Population characteris

• Unbiased 

• Sequestered (blind test), not correlated with automated sys• Mix of database shall be avoided (Better to conduct the test 

• Synthetic 

data are lik ley to i t dnrouce biases andshall be avo

• Large enough 

• To capture diversity of situations • To enable extrapolation: database should be at least ~ 1/10

size 

• ... and reasonably known and characterized (Ground truth, qu

No database is perfect 

• Analysis shall be performed to assess risks and biases 

NIST IBPC - Gaithersburg, March 4, 2010 
Test preparation: Criteria 

Test criteria must be directly linked to syste

• Classical biometric indicators are 

• Rejection rate (FTE/FTA) • Accuracy (FMR, FNMR) • Throughput (Speed) Test 

Data 

• Other indicators can be 

• Robustness to biometric errors & fraud • Interoperability, .... 

• Internal parameter may be measured to help modeling system beha

• Filtering rate, number of correct minutiae, .... • They shall not be directly used as system performance criteria 

• Specify the information that will be needed to calculate and interpre

NIST IBPC - Gaithersburg, March 4, 2010 
•

••

Test preparation: Criteria 

• Those Indicators are linked 

• FMR • FNMR: « Decision policy » 

• Speed • FNMR: « Tuning policy » • FTE/FTA • FNMR: « Rejection policy » 

⇒Those policies are business policies, not technology policies 

• Weight of those criteria must be known before testing to ena

⇒Tuning to system requirement is necessary 

• Tuning to business policies and -to some extent- to system da

⇒Those indicators must be measured simultaneously 

NIST IBPC - Gaithersburg, March 4, 2010 
Run test ... 

Set up 

• Install, tuning, smoke test 

Blind test 

• no modification to s

Documentation, logging and monitoring are critical during t• Biometric results • Timing information • Errors and anomalies • System behavior (CPU, I/O, ...) 

NIST IBPC - Gaithersburg, March 4, 2010 
Interprete results 

• First thing to do is to check and validate the resu

• Calculate performance indicators • Is blind test consistent with the tuning tests ? 

• Accuracy, speed 

• Analyze potential errors (corrupted files, problems in the da• Validate ground truth 

• Are there unexpected hits ? • Are there any surprising results ? 

• Then you can do more interpretation 

• Detailled analysis 

• Correlation with data characteristics 

• Extrapolation 

NIST IBPC - Gaithersburg, March 4, 2010 
Accuracy Extrapolation 

• Use statistical methods (parametric and non parametric): 

• See “Biometrics system: Technology, Design and performance evalua

Springer, 2005 by Wayman, Jain, Maltoni, Maio, p 263-287 

• Use empirical methods 

• Plot TAR vs Database size 

• Raises fundamental sta

• Independence of 

• In practice, Simplistic e

• Provide acceptab

by less than 100) 

• Permits to proje

qualitatively) 

• ROC curves: FNMR (FMR, 10xDBSize) ~ FNMR (FMR/10, DBSize) 

NIST IBPC - Gaithersburg, March 4, 2010 
Sizing Extrapolation 

• Extrapolating sizing from test measurements is very co

• Biometric factors 

• Algorithm speed (can be estimated in testing providing test D

• Non biometric factors 

• Architecture considerations, hardware limitation, ... 

• However simple « rule of three » on matching time can 

• It is certainly not sufficient to prove scalability • It is often sufficient to prove non-scalability 

• Throughput = 100,000 requests/day • Database size =100,000,000 people • Measured matching speed = 1,000 record / sec / server 

=> Requires over 115,000 servers ... 

NIST IBPC - Gaithersburg, March 4, 2010 
Conclusion 

• Testing is application dependent 

• Need to define objectives, protocole, database and criteria 

• Main biometric criteria (FTE/FMR/FNMR/Speed) are li

• Trade off must be decided by business, not by technology • Some tuning (to policies and to data) is necessary 

• There are biases in every test and extrapolation 

• They need to be minimized • They need to be known and taken into account when draw

• Simplisitic extrapolation techniques are useful (even if 

Thanks! 

NIST IBPC - Gaithersburg, March 4, 2010