Using Challenge Problems 

To Accelerate Biometric 

Technology 

Dr. P. Jonathon Phillips 

National Institute of Standards and 

Technology 
Technology Progress 

 

 

 

 

 

1996 

2000 

2002 

2006 

2010 

1993 

2005 

•

•

•

•

17 Years

8 Evaluations

3 Biometrics

150,000+ Facial and Iris

Images 2009 

2003 

• 5 Challenge Problems

(Technology Development)
 

 

Improved FR Performance 

Single Still 

Controlled 

Different Days 

Face Recognition Error Rate 
Advancing Technology 

and Methodology 

Evaluations Challenge 

Problems 

Human 

Vs 

Computer 

Data 

Collection 
 

 

Methodological Progress 

2010 

1993 

1994: First Modern Technology Evaluation 

2001: 

Re-sampling 

Techniques 

2004: Human 

& Computer 

Performance 2007: 

Iris Meta-analysis 

1996 

2003 

2000 

2002 

2006 

2009 

2008: 

Covariate 

Focus on 

Quality 

2002: First 

Evaluation 

with 100k+ 

samples 

2004: 

Statistical 

2009: Modeling 

Biometric 

Completeness 
 

 

 

 

 

 

 

Challenge Problems 

• What are challenge problems? 

– A series of experiments designed to advance a 

technology’s state-of-the-art 

• Experiments designed 

• Experiments and test data distributed to researchers 

• Researchers complete experiments and submit results 

• Scores are consolidated and reported 

– Introduction of new technology 
 

 

 

 

 

 

What Is A Challenge Problem? 

• Challenge Problem 

– Open book 

• Components—made available to participants 

– Data sets 

– Experiments 

– Ground truth 

– Baseline algorithm 

• Similarity Matrices Submitted 

– Generated by participants 

– Scored by NIST 

• NOT an independent Evaluation 

– NO sequestered data 
 

Ideal Challenge Problem 

Time / Effort / $ 

Progress / Performance 

Today 

Challenge Problem Difficulty 

Time 
 

Challenge Problem Sin: Too Easy 

Time / Effort / $ 

Progress / Performance 

Today 

Difficulty 

Time 
 

Challenge Problem Sin: 

Too Little Time 

Time / Effort / $ 

Progress / Performance 

Today 

Difficulty 

Time 
Evidence of Progress through FRGC 

Carnegie Mellon Innovation 

82.4% @ 0.1 % FAR 

(Latest Performance) 
 

Building a Challenge 

• Goals—Simple and grandiose 

• Setting goals—Cheat 

• Complete infrastructure for challenge 

problems 

• Open to all 
Expanding Technology 
 

 

 

 

What Does Expanding Technology 

Do For You? 

• Development of new or improved technology 

• Focus research on challenge problem 

• Large community working on problem 

• Solutions from novel approaches 
Portal Recognition Cont. 

 

 

120 pixels 2k 

across iris 

2k 

Near Infrared (NIR) Video Sequence 

High Definition (HD) Video Sequence 
Example of Expanding Technology: 

Recognition from Unconstrained Video 

• Still versus Video

• Video versus Video

Still 

Walking Conversation Uncontrolled Face Face Video 
Expanding Science 
 

 

 

 

 

 

What Does Expanding Science 

Do For You? 

• Increases fundamental knowledge of 

biometric modalities. 

• Human and computer performance 

• Covariate analysis 

• Analysis of results on large data set 

• Underlying properties of a biometric 

•••
 

 

 

19 

Example of Expanding Science: 

Iris Biometric Stability 

“Empirical Evidence for Iris Match Score Degradation with Time Lapse in ICE 2006” S. 

Baker, P. J. Flynn, K. W. Bowyer, and Dr. P. Jonathon Phillips, NISTIR 7630, 2009 
 

 

20 

Motivation 

• Iris biometrics assumption: The iris is 

stable throughout one’s life. Is this claim 

accurate? 
 

 

 

 

Introduction 

• 23 subjects 

– 46 irises 

– Collected 2004 through 2008. 

• Three iris recognition algorithms 

– IrisBee baseline algorithm 

– ICE 2006 Algorithm B 

– VeriEye 

21 
 

 

 

 

 

 

 

Experiment 

• Computed match (genuine) scores 

– for images acquired less than 120 days apart (shorttime-lapse). 

– for images acquired more than 1200 days apart (longtime-lapse). 

• Compared short-time-lapse and long-time-lapse 

– Mean of match score distribution 

– Median of match score distribution 

22 
 

 

 

 

 

 

Results 

• IrisBee Algorithm 

– 43 of 46 irises showed degradation, p-value = 2.311 × 10−10 

– mean match score 

• ICE 2006 Cambridge Cam-2 Algorithm 

– 38 of 46 irises showed degradation, p-value = 9.2477 × 10−6 

• VeriEye Algorithm 

– 40 of 46 irises showed degradation, p-value = 3.103 × 10−7 

IrisBee ICE 2006 Cam-2 23 
 

 

 

Iris Stability 

• First study 

– One sensor 

– Limited subjects 

• Recommend further studies 

• Template aging as observed in other 

biometrics 

• Multi-lab criteria 
 

 

 

Conclusions 

• Biometric technology has experienced 

significant progress over the last 15 years. 

• Challenge Problems are Key for advancing 

the ‘State of the Art’. 

• Science is Key to advancing Technology. 
Questions? 
Example of Expanding Technology: 

Portal Recognition 

High 

Definition 

(HD) Video 

Camera 

Near Infrared 

(NIR) Video 

Cameras