Challenge Problems 

Dr. P. Jonathon Phillips 

NIST 
Problem Definition 

• Frontal Faces • One Face Image per Person One Face Image per Person 

Problem 1: Controlled Studio Environment Problem 2: Studio vs. Ambient Lighting 

or 
Technology Progress 

1996 

2000 

• 17 Years

8E l i

2010 

• 8 Evaluations

• 5 Challenge Problems

(Technology p ) Development)• 3 Biometrics• 150,000+ Facial and Iris

MBE

2010

2009 Images

2006 

1993 
From FERET to MBE 2010 

0.2 

0.026 

0.003 

0.00 

1993 1997 2002 2006 2010 

FERET FERET FRVT FRVT MBE 

(Partially Automatic) (Fully Automatic) (Fully Automatic) (Fully Automatic) (Fully Automatic) 

Year of Evaluation 

1 00 

0.79 0.80 

1.00 

1 

0.54

0.60 0.40 

F0. 20RR at FAR = 0.00
Challenge Problems 

• What are challenge problems? 

– A series of experiments designed to advan

technology’s state-of-the-art 

• Experiments designed • Experiments and test data distributed to resear• Researchers complete experiments and submit • 

Scores are consolidate

d an

d reporte

d 

– Introduction of new technology – Not an independent evaluation 

• NO sequestered data 
What Is An Independent Evaluatio

• Independent assessment of performanc

third 

py art

y

• Algorithms submitted to tester • Algorithms are tested on sequestered d• Results scored by tester • Provides snapshot of current technolog

of-the-art 
Advancing Technology and Methodolo

E al ationE al ation

Data Challen

g

e

E

val

uation

g ProblemsProblems Data CollectionCollection 

Human 

Vs StatisticalStatistical AnalysisAnalysis Experimental Experimental Design Compute

Compute

AnalysisAnalysis
1996 

2000 

2010 

2002 

2003 

2006 

2010 

2010: 

1 Million + face 

MBE

2010

2004: 

2006 

2008: 

1 Million + face 

image evaluation 

2008: 

Covariate Analysis 

of Quality 

Methodological Progress 

1994: First Modern TechnologyEvaluation

2007: 

Iris Meta-analysis 

20

Re-sa

Tech

2004: 

& Co

Perfo

1993 
Ideal Challenge Problem 

Today 

Time 

C

h

allegne Proble

m Difficulty

Time / Effort / $ 

e rmanc / Perfo ogress Pr
C

CCCC C

AA

Challenge Problems and Technology Develo

Example: Advanced Correlation Filter MExample: Advanced Correlation Filter M

January 2005 

MACE 

Training - One Filter/Person 

CorrelationCorrelation 

Filter 

Recognition – Compare all Filters 

CorrelationCorrelation 

Filter 

CorrelationCorrelation 

Filter 

Augus

Kernel Corr

Offline Tr

General R

* Xie, Sa
Evolution of Improvement 

es t ion Ra terificatVe 

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 

0 

Exp 4 
Building a Challenge 

• Goals—Simple and grandiose • Setting goals—Cheat • Complete infrastructure for challenge p• Open to all 
Conclusion on Basic Design 

• Challenge problems 

– 

A ver

y powerful tool 

– NOT the only tool 

• A well designed challenge problem 

– Advances technology – Good Investment 

– 10+ year life span 

• Poorly designed challenge problem 

– Waste resources Waste resources – 10+ year life span 
Summary 

• Frontal Face Recognition • Three orders of magnitude improvement • Challenge Problems & Evaluations key comp• Development of new techniques • Interdisciplinary Field • Significant amount of research left 

– Face recognition form video Face recognition form video – Low quality still images – Non-frontal stills recognition