23 S ept emb er 2009 

E valuating the P erformanc

Biometric Algorithms 

Craig Watson – NIST 

Biometric Consortium 2009 
Outline 

• Why NIST?• Impact and Issues with Evaluations• NIST Evaluation Process• Datasets• NIST Evaluations• Future Plans
Why NIST Performs Biometric Eva

• Mandates: 

• P t i t A t (PL 107 Patriot Act (PL 107-56) • Enhanced Border Security and VISA Entry Ref

107-173) 

• Homeland Security Presidential Directive 12 (• 10-Print Transition: mandated by Homeland Se

Council Deputies Committee 

• Develop testing methods for evaluating biom

performance 

• Ad bi t i t hi t h l i Advance biometric matching technologies 

• Other Government Agencies 
Impact of Evaluations 

• Current measure of biometric technology Current measure of biometric technology pe

on operational data. 

• Algorithm developers can evaluate performa

operational datasets. 

• Advance biometric matching technologies. • Improve implementation’s adherence to stan

protocols 
Issues with Performin

g Evalua

• Baseline algorithms (bozorth) give ability to 

evaluation methods 

• Developing API to meet evaluation needs • Dataset sample size • Dataset preparation and security • Repeatable results • Validation/Securit

y of submitted SDK 

• Interpretation of results 
Evaluation Process 

Develop API/Prepare Dataset 

Application Results 

Software Error Check 

Test Driver 

Report 

Validation 

Evaluate Evaluate 
Evaluation Process 

• API 

• Inputs/Outputs to interface with SDK 

• Consider “gaming” issues • Use existing standards when possible • Length of time to run evaluation 

• Dataset Preparation 

• Consolidations Consolidations • Sample selection – random, sample size, 

data 

• 

“True 

” imposter data imposter data 

• Data format • Length of time to run evaluation 
Evaluation Process 

• Software Validation 

• Repeatable – why not? 

• Resolve issues of non compliance • Estimate length of time to run evaluation 

• 

Evaluation 

• Managing across multiple systems – enro

matching stages 

• Unexpected exits Unexpected exits • System maintenance shutdowns • Length of time to run evaluation 

• Confidence in results 

• What to report • Statistical si

gnificance 

• Error check “outliers” 
Public Datasets 

Algorithm Development & SDK Validat

biometrics.nist.gov/databases 

• Fingerprint 

• NIST Special Databases – 4, 9, 10, 

29, 30 

• Face 

• NIST Special Database 18 • ColorFERET 

• Iris 

• ICE dataset 
Sequestered Datasets 

Used for Evaluations 

• Fingerprint Face and Iris Fingerprint , Face, and Iris 

• Operational Datasets from sponso

agency 

• Large number of subjects (100,000 

6,000,000) 

• Multiple Instances • Meta-data for results analysis 
Biometrics Evaluations at NI

biometrics.nist.gov/evaluations 

Face Recognition 

Vendor Technology 

(frvt2006@nist.gov) 

Biometric Usability 

Slap Fingerprint 

Segmentation 

(slapseg@nist.gov) 

Ii E 

Face Recognition 

Vendor Technology 

(frvt@nist.gov) 

Iris Challenge 

(ice@nist.gov) 

002

Minutiae Exchange 

(minex@nist.gov) 

004

006

008

Iris Ex

(irex@n

20

20

20

003

005

007

2

FingerPrint Vendor 

Technology (fpvte@nist.gov) 

Proprietary Fingerprint 

20

20

20

Proprietary Fingerprint 

Templates (pft@nist.gov) 

Evaluation 

(ice@nist.gov) Multiple Biometric 

Grand Challenge 

(mbgc@nist.gov) 

Ongoing MINEX 

(minex@nist.gov) 

Face R R ecogniti on

Grand Challenge 

(frgc@nist.gov) 

Iris Challenge 

Latent Fingerprint 

Technology (ELFT) 

(l t t@ i t ) (l

atent@

nist.gov) 

MINEX II 

(Match on Card, 

minexII@nist.gov) 
Large Scale Evaluations 

• SAN Storage for large datasets (240T

SAN Storage for large datasets (240T

• High end computing 

• 32 blades (4x4) with 192GB memo• 24 blades (2x4) with 48GB memory • 80 blades 

(2x2

) with 16GB memor

y 

• 136 blades -> 336 CPU -> 1024 cor
Evaluation of Latent Fingerp

Technologies Technologies

-Extended Featur

Extended Featur

(ELFT-EFS) 

• Gallery feature space 8-12GB • Current: 80 blades (2x2) 4GB memory

local storage workspace 

• Upgraded: 16GB memory to allow sea

backgrounds 10K subjects with 10 roor slaps each. 

• 300GB local storage workspace • OS upgrades to 64-bit Linux and wind
Large Scale Evaluations 

• Use larger sample sizes in existing 1 Use larger sample sizes in existing 1 -to -1 eva

(PFT, MINEX, ...) 

• Large scale biometric identification evaluati

Large scale biometric identification evaluatimany) 

• Government needs for evaluations on large d• Testing methods and APIs for large-scale eva
Thank You 

Craig Watson 

craig.watson@nist.gov 

biometrics.nist.gov/evaluati