Using Challenge Prob

advance the Technolo

Science of Biome

Dr. P. Jonathon Phillip

National Institute of Standards an
August 1994

FERET Large Gallery Results

Performance for top 50 matche
Sixteen Years Later

MBE 2010 Still Face Track

Performance for top 200 matches 

200 (Nominally) 

False Negative Identification Rate

0

1.0
Orders of Magnitude IncreKnowledge

• Number of people in the field• Sophisticated test protocols• Regular Challenge Problems• Regular Evaluations• 200,000+ Biometric samples availabl• Conferences, Workshops, Journals• Statistical methods• Standards• Human performance comparisons an• National (US) and International partic
Predicting Performance

Target Set

Query Set

New 

New Query Set
Levels of Predictions

• General Assessment
Levels of Predictions

• General Assessment• Measuring Improvement
Levels of Predictions

• General Assessment• Measuring Improvement• Ranking of Algorithms

– Relative performance– Ranking stable across data sets– Limited success
Levels of Predictions

• General Assessment• Measuring Improvement• Ranking of Algorithms

– Relative performance– Ranking stable across data sets– Limited success

• Predict Performance
Levels of Predictions

• General Assessment• Measuring Improvement• Ranking of Algorithms

– Relative performance– Ranking stable across data sets– Limited success

• Predict Performance

Now that’s a Challen
Formulated a

Challenge Pro

Twenty Quest
Examples of Questions

• Distribution of

– Gender– Race– Age

• Distribution of quality measures

– Focus– Illumination– Iris area– Number of minutia
A Challenge Problem in thTwenty Questions

• Set up:• Evaluator

– Algorithm A– Set of sequestered images

• Divided into target and query

– ROC; FRR and FAR at a fixed threshold

• Participant

– Algorithm A
A Challenge Problem in thTwenty Questions

•

k – Questions:

• Participant

– Asks 

k questions 

• Evaluator

– Provides answers
A Challenge Problem in thTwenty Questions

• End:• Participant

– Submits estimated ROC; FRR and FAR a

• Evaluator

– Measure accuracy of estimated ROC; FR
Types of Questions I

• Questions allowed I:g( )

R

• Answer provided:

– Empirical Cumulative Distribution Function (ECDF) 

F

g(x)
Types of Questions II

• Questions allowed II:g( )

R

• Answer provided:

– Joint ECDF between target and query

F

g(x

T, x

Q

)

Target Set

Query Set
Types of Questions III

• Questions allowed III:g( )

R

h( )

R

• Answer provided:

– Joint ECDF between h and 

g

Fg,h(x,y)
Example from FRVT 2006Uncontrolled vs. Uncontro

Overall Performa

VR = 0.80

@ FAR = 0.001

Target

9803 Images

Q

u

e

r

y

9803 Images
What Lurks Within

Target

9803 Images

Q

u

e

r

y

9803 Images

Good

1083

1083

Ugly

1083

1083
Face Pairs

Good Face Pairs Challeng
Good, Bad, Ugly Performanc

?
Good, Bad, Ugly Performanc

Performance ranges over an order 
Challenge

• What k – questions would ch

performance that ranges oveof magnitude (VR 0.98 to 0.
Biometric-completeness

• NP-completeness

– Formal definition

• AI-completeness

– Informal definition

• Biometric-completeness

– Informal definitionA problem is biometric-complete if 

problem is “equivalent” to solvinbiometric recognition problem
Approximate solutions

• Push too hard on “prediction” and 

yourself right back around facing complexity of the biometric identiproblem.

• Approximate solutions count, and 

worth pursuing, just as it is still wcontinue work on AI-complete or problems.
Conclusion

• Significant progress in principles • Predicting performance is not so• Outlined a challenge problem for 

– Based on twenty questions
Thank You

Questions
Types of Questions IV
Types of Questions IV

• ?

?

• ?

?

• ?

?
“You can’t always get wh

want. But if you try someYou just might find you gyou need!” - Roll
An Introduction to Biometric-c

The Equivalence of Matching 

P. Jonathon Phillips

National Institute of Standards and Tec

J. Ross Beveridge

Colorado State University
“The Good, the Bad & the Ug

Still Face Challenge• Encourage development of face 

algorithms that work on “hard” to face pairs.
Equivalence of Matching and 

Formal Model

Q ( ) =

A ( 

Smiling Neutral

Implies

Neutral Neutral

Q ( ) =Implies A 

“You can’t always get what you wa

try sometime ... You just might find 

you need!”

1
FRVT 2002 Fine Print