International Biometrics Performance Conference 

March 3, 2010 
Outline of Presenta/on 

• Scope 

•Background 

• QPL Process 

•Contrasting Evaluation Reporting Methods 

• Lessons Learned 
Narrowing the Scope 

Scope is Biometric Product Performance testing using Scenario evaluation 

techniques focused on Access Control applications using 1:1 Verification matching 

Biometric Product 

Performance

Scenario 

Access Control 

1:1 Verification 
Background 

Specific information relates to US DHS TSA QPL 

• Mandated in 2004  (US Public Law 108‐458) 

• Requirements and Process published ‐ 2005 

• Initial QPL – 2007 

• Addition to QPL – 2009 

• Process in place, on‐going 

• Specific to airport PACS 

DHS – Department of Homeland Security  QPL – Qualified Products List 

TSA – Transportation Security  PACS – Physical Access Control System 

Administration  
Biometric por/on of PACS 
QPL Process – Event Sequence 

•Airports demand for biometric product for 

PACS 

•Manufacturers submit to third‐party 

testing (fee based) 

• Suitable testing organization delivers 

performance report 

•TSA approves product          QPL 

•Airports select from QPL for purchase  
Key Test Characteris/cs 

• Test crew size >250 (enroll), >200 (verify) 

• Plan for “drop‐out” 

• 4 visits: Enroll, 3 Revisits (~2 weeks apart) 

• Enrollment includes training and initial 

verification 

• 5 genuine & 5 imposter transactions/revisit 

• Minimum 3000 transaction (each type) 

• Imposters are randomized from other crew 

members 
Performance Metrics and 

Requirements Defini/on 

Pass/Fail testing must establish metrics, requirements and decision criteria 

• False Accept Rate       < 1% 

• False Reject Rate  < 1% 

• Transaction Time  < 6 seconds 

• Failure to Enroll Rate  < 3% 

NOTES: 

• Matching errors based on up to 3‐attempt transactions 

• Confidence bounds used to evaluate Pass/Fail 

• Requirements set with industry, airport and NIST participation 












Use of Confidence Bounds 

F 

R 

R 

FAR 
Scenario Tes/ng Approaches 

• Conformance to Performance Specification

Testing

• E.g. TSA QPL  

• Report only “Pass or Fail” (as one operating point) 

• Graded Performance Testing

• ISO Standard 19795‐5 or INCITS 409.5 

• Matching grades range 0 (worst) to 6 

• Pure Performance Testing

• E.g. NPL or IBG (CBP series) 

• DET or ROC (typically) 
NVLAP Biometrics “Scopes” 

NVLAP Handbook 150-25, Figure A.3. Scopes of accreditation 
Graded Performance 

From ISO FCD 

19795‐5 

• Shows 3‐FAR 

level range (0.1 – 

1.0%) 

•Shows grades 

assigned at 6 

FRR thresholds 

          <1% = grade 4 
Pure Performance Repor/ng 

From IBG CBT 6 public report 
 

 

 

Contrast in Results Repor/ng  

Pure Performance – maximum information, more difficult to interpret 

Graded Performance – typical operating range, shows performance levels 

Pass/Fail – simplest, no indication of preference  Graded 

Target FAR 0.1% 0.3% 1.0%

FRR Grade – BLUE 4 5 5 

FRR Grade – RED 4 4 4 

FRR Grade - GREEN 3 3 3 

Pass/Fail 

FRR Decision – BLUE     Pass

FRR Decision – RED       Pass*

FRR Decision – GREEN Fail 

* (depends on confidence level)

Pure 
Lessons Learned 

• Enrollment with multi‐instance modalities 

• more complicated, needs written policy 

• Identify need for re‐enrollment  

• remove “avoidable goats” 

• Criticality of threshold setting  

• responsibility – defined by process, enforced by lab, 

depends on supplier 
Lessons Learned ‐ 2 

• Pass/Fail testing does NOT require access to 

matching scores (COTS configuration) 

• Who controls Final Report dissemination to 

qualifying organization (lab or supplier?) 

• Cost to participate is a deterrent to suppliers 
Thanks for your aUen/on 

Contact information: 

Rick Lazarick, CBP 

CSC – Chief Scientist, Identity Labs 

rlazarick@csc.com 

609‐883‐6767