Face Recognition 

Grand Challenge

Dr. P. Jonathon Phillip

NIST

March 2004
Face Recognition Grand Ch

Systematically pursue two 

(2D and 3D) to reduce t

rate in face recognition 

order of magnitude
Verification - HCInt

False accept rate

Verification rate

0.0001 0.0010 0.0100 0.1

0.00.10.20.30.40.50.60.70.80.9

IdentiCognEyemC−VIImagiViisagVisionDream 

82

90

96

79

87

82

90

0

20406080

100

0.1 1 

False accept rate (%)

Verification rate (%) 

1.0
Select Point to Measure

• Verification rate at :

– False accept rate = 0.1%

• Current: 

– 20% error rate (80% verificat

• Goal: 

– 2% error rate (98% verificatio
Measuring Accuracy w/Error 

Rate of 2%

• Non-match scores: 

– Sufficient

• Match scores: 

– Need to design collection for suff

• Allows for error ellipses

• Minimal demographic analysis

1,000 match scores = ~ 20 err

10,000 match scores = ~ 200 er

50,000 match scores = ~ 1,000 er

50,000 match scores = ~ 1,000 er
Data Collection

Fall Semester

(Gallery)

15 Weeks

Spring 

(Pr

15 

200 

People

All match scores ~ 50,0
Modes Examined

Multiple Stills 3D S

vie

Outdoor/

Uncontrolled Single Still
Measure Progress on:

• Indoor cooperative face rec• Outdoor cooperative face re• Comparison of still & 3D fac

recognition

• Effect of multiple images• Effect of High Dynamic Ran

cameras on outdoor face re

• Comparison between human 

machine performance
Programmatic

• Series of Challenge Problem

– Facilitate development– Systematically measure progre

• FR Challenge Grand Challeng

Evaluation

– Independent measure
Experiment Design

FRVT 2002 versus Grand CFRVT 2002• Flat design• Process all data• Blind data• Black box• Single mode• Fixed design• Central analysis

Grand Challe

• Combinato• Process su• Transpare• Transpare• Multiple m• Adaptive d• Distribute

analysis
Grand Challenge Schedule

Date Activity

Aug-Sep '05 Face Recognition Grand Challenge 

Sep ’04 Release challenge problem v2.0 

Aug-Sep ’04 Baseline performance determinat

April ’04 Release challenge problem v1.0 April ’04 First challenge problem workshop 

Explain challenge problem in d
Challenge Problem Infrastruct

• Based on HumanID gait ch

problem

– Design set of experiments– Baseline algorithms– Infrastructure for running e

• Documented progress

– Forced researchers to concen

problem
BEE Architecture

BEE-core

Algorithm/

System

Analysis tools

ExperimentalData sets•Training•Galleries•Probe sets

Results
BEE: Biometric

Experimentation Enviro

• Flexible Framework for Biom

Algorithms

– Plug & play algorithms– Mix & match modular compone

• Provides Universal XML-Bas

Interfaces

• Facilitates Biometric Evaluat• Uses a Non-Proprietary Ope

Design 
Experiments

• Evaluate single system using 

multiple datasets

• Evaluate multiple systems with 

uniform data

• Complete audit trail• Audit trails are portable and 

can be used as inputs to subsequent runs

• Similarity matrices offer a 

portable representation of raw results

• Rerun trial using intermediate 

results

• View graphs/reports on 

experiments performance

• Supports independent advanced 

post analysis

BEE 
Reproducibility via XML

Datasets

(inputs)

Similarity Matrix

(raw results)

<Signature>

<Subject id=“hgfR00001” /><Image file=“img.jpg” /><Recording id=“hgfR00001”/>

</Signature>

<Audit>

<Target file = “DataSet1.xml” /><Query file=“DataSet2.xml” /><BioBox module=“hFace 1.2”/>

</Audit>
Cross Institution

Experimentation

Institution 1 

BEE 

Algorithm A 

Raw Results 

Data Sets 

Experiment 

Definition

Institution 2 

BEE 

Algorithm BRaw Results

Data Sets 

Experiment

Definition

Challenge Problem Designer 

Experiment

Definition

Challenge Problem Designer 

Experiment

Analysis

• Ex

alginscomde

• Tr

res

• Un

res

• Cro

ana

...
Cross Institutional

Experiment Replication

• I

rr

• T

ed

• U

o

A/B

Experiment 

Definition

Institution 1 Institution 2 

BEE Data Sets 

Experiment

Definition

Algorithm 

Raw Results

Analysis 

Audit 

Trail

Data Sets BEE 

Algorithm 

Raw Results 

Trail

Audit 

Analysis 
Grand Challenge Evaluation 

• Jonathon Phillips—NIST 

– Director Face Recognition Grand Challenge

• Notre Dame (Prof. Kevin Bowyer and Prof. Patr

– Data collections

– Baseline algorithms

• SAIC (Dr. Todd Scruggs)

– Design and implement BEE– Maintain hBase

• Mitre (Joe Marques)

– Analysis– Assist with Grand Challenge

• University of Texas at Dallas (Prof. Alice O’To

– Human performance
Conclusion

• Face Recognition Grand Challeng

– Order of magnitude increase in perf– Systematically investigate still and 3– Formulate series of challenge proble– Final Grand Challenge evaluation

• Biometric Experimentation Envir

(BEE) 

– Infrastructure for Grand Challenge– Uniform structure for challenge pro