The Need for Large-Scale 

Biometric Testing 

Patrick Grother 

Information Access Division 

NIST 
The purposes of biometric test

» Scientific discovery 

» Basic research, sensors, algorithms, 

» Research and development 

» Is the method better than before? » Internal commercial, or gov-industry 

cooperation 

» Capability testing 

» Is the technology viable? 

• Benchmarking • Core capability 

» Establishing criteria 

» Testing to know what’s possible 

» Could requirements be met » Is the technology viable? 

» Comparative testing 

» Which technology? » Which implementation? 

» Conformance 

» To requirements » To standards 

» Interoperability testin

» Can we upgrade, repl

keeping our data? 

» Jurisdictional interop

» Certification 

» Can we leverage othe

» Regression 

» Has the update helpe
In

Science 

Technology Development 

Product Development 

Conformance to Requirements 

STANDARD 

NIST 

NIST 

NIST 

Certification 

Operational 

teroperability 

Pure Research Time Mature Market 

Procurement, Comparative 

Acceptance Testing 

STANDARD 

STANDARD 

STANDARD 

NIST 

Stages of Testing 
Why test at all? 

» It’s about money 

» Biometric errors cause additional cost 

• Failure to enroll → additional time, procedures, modal

processes, time 

• In 1:N false non-match → benefits fraud, immigration f

• In 1:N false match → identity resolution processes 

» Can the application application requirements requirements be met? be met? » Can the technical requirements be met? » Risk mitigation 

• Characterizing performance allows procedures to mitig

a finding that FTE > 0.03 prompts environmental redesregulate humidity, or ambient light. 

» Conformance, Interoperability 

• Entire system needs to be upgraded / junked if it doesn

interoperate with others 
Trading Time for Accuracy 

An induswide tradspace. 

MATCH RATE LESS ACCURATE TAK

MO

FALSE NON-M 

TIME TO PREPARE LOCATE IRIS + MAKE STD. RECORD 

ST ACCURATE TAK

 

(milliseconds, mean kinds 3 and 7) 
Large scale testing, eh? What’s la

» Statistical definition of large 

» Depends on purpose 

• e.g. show, with 99% confidence, that a match-on-card comparison algorit

10-4 

• e.g. existence proof that a gummy finger can be enrolled in a biometric sy• e.g. given a choice of instructional modes for use of a biometric sensor, d

live test, which is best 

• e.g. testing whether a e-Passport conformance testing suite correctly reje

records or JPEG 2000 streams 

» T h l d d Technology dependent 

• e.g. comparing AFIS systems vs. comparing 1:1 ePassport gates 

» Practically... 

» Systematic effects are larger than random effects and these can b

efficiently 

• Irises with radius greater than 150 pixels always fail to enroll 

» Cost constraints limit population size, test duration 

• Corners are cut (e.g. full-cross comparison of N samples) 

» Test crews get tired 
FNMR(B) 0.0 -0.02 FNMR(A) – F 

– 

Test Size :: Vendor A vs. B 

-0.01 FMR(A) FMR(B) 0.01 
IBPC 2010 

» International Biometric Performance Conference 

» Novel test methods, metrics » Specification, requirements, certification » Accuracy, security, operational » Emphasis on how systems are tested, vs. latest results 

» Co -chairs 

» Patrick Grother, Elham Tabassi, NIST » Christoph Busch, Fraunhofer » Tony Mansfield, NPL 

» Logistics 

» March 2-4, 2010 » Gaithersburg, Maryland » Call for papers: http://biometrics.nist.gov/ibpc2010 
In 119 pages IREX covers 

» http://iris.nist.gov/irex » Compare algorithmic accuracy 

» ROCs » Fixed threshold – effect on FMR 

and FNMR 

» Speed-accuracy trade-space 

» False Match Rate Calibration 

» How to set the threshold 

» Effect of dataset 

» On FNMR, on FMR 

» Algorithm interoperability 

» Enroll on A – Identify on B 

» Segmentation performance 

» Image quality assessments 

» Biometric zoo 

» Compare lossy comp

algs 

» JPEG vs. JPEG 2000 

» Limits of lossless com» Bounds on iris size 

» How closely to crop » Comparison Comparison of spec

spec

formats 

» Masked vs. Polar » Which is fit for pur

» Effect of pupil dilatio

» And change in dilat

» Effect of eyelid occlu

» Effect of iris-pupil 

displacement 
Thanks 

p atrick.g rothe @ rnist.g ov

301 975 4157 

Minutia interoperability: http://fingerprint.nist.gov/minex 

Iris interoperability: http://iris.nist.gov/irex 

Biometric Performance Conference: http://biometrics.nist.gov/ibp