Face & Ocular ChallengeS

(FOCS)

Dr. P. Jonathon Phillips

NIST
Collaborators

– Ross Beveridge

– Soma Biswas

– David Bolme

– Kevin Bowyer

– Rama Chellappa

– Bruce Draper

– Patrick Flynn

– Geof Givens

– Patrick Grother

– Fang Jiang

– Yooyoung Lee

– Yui Man Lui

– Alice O’Toole

– George Quinn

– Vishal Patel

– Todd Scruggs
MBE 2010 Still Face
Problem Definition

• Frontal Faces

• One Face Image per Person

Problem 1: Controlled Studio Environment

Problem 2: Studio vs. Ambient Lighting

ορ
Technology Progress

1996

2003

2000

2002

2006

2010

1993

2005

 17 Years

• 8 Evaluations

• 5 Challenge Problems 

(Technology Development)

• 3 Biometrics

• 150,000+ Face and Iris 

Images 2009

MBE

2010
• Evaluation period: Jan – May 2010

– Measure progress since FRVT 2006

– Leverage massive operational data corpora.

– To evaluate face recognition technologies in a proper one-tomany identification mode.

• Multiple Biometric Evaluation 2010: Still Face Report, 

P. Grother, G. Quinn, and P. J. Phillips, NISTIR 7709, 

2010, http://face.nist.gov

6

Goals of MBE 2010 Still Face Track
• One Face Image per Person

Problem 1: Controlled Illumination vs. Controlled 

Illumination

From FERET to MBE 2010
From FERET to MBE 2010

FERET 1993

(Partially Automatic)

Data Sets

FERET 1996

(Fully Automatic)

FRVT 2002

(Fully Automatic)

FRVT 2006

(Fully Automatic)

MBE 2010

(Fully Automatic)

FRR at FAR = 0.001

Year of Evaluation
• All probes in gallery

• Score: Rank 1 Identification

• Selectivity: Number of average matches returned

9

Closed-set Identification

Large Gallery (1.6 million)

Who is this person?
Closed-set Identification

SDK

Gallery of 

1.6 million 

Photo file 

images
False Reject Rate at False Alarm Rate = 0.001

SDK
Main Results

• Improvements in 1-1 verification

– Three order improvement since 1993

– FRR = 0.003 at FAR of 1 in 1,000

• Closed Set Identification

– Gallery of 1.6 million faces

– Rank 1 ID = .93

• Is face recognition solved?

– Not for unconstrained environments
Face & Ocular Challenges (FOCS)

• Video

• The Good, the Bad, & the Ugly

• NIR Ocular

• Performance Prediction
Video

(MBGC ver2)
Walking vs. Activity

Activity vs. Activity

Walking

976 sequences

Activity

784 sequences

Activity

784 sequences

Activity

784 sequences
Human Performance on Video

Recognizing people from dynamic and static faces and bodies: Dissecting identity with a

fusion approach ,” P. J. Phillips, A. J. O'Toole, S. Weimer, D. Roark, J. Ayadd, R. Barwick, J. Dunlop, 

Vision Research, in press, 2010.
• Human subject raters respond...

– 1. sure they are the same person

– 2. think they are the same person

– 3. not sure

– 4. think they are not the same person

– 5. sure they are not the same person

Video: Walking vs. Walking
Video: Human & Machine Performance
Video: Walking vs. Conversation

• Human subject raters respond...

– 1. sure they are the same person

– 2. think they are the same person

– 3. not sure

– 4. think they are not the same person

– 5. sure they are not the same person
Video: Conversation vs. Conversation

• Human subject raters respond...

– 1. sure they are the same person

– 2. think they are the same person

– 3. not sure

– 4. think they are not the same person

– 5. sure they are not the same person
Video: There is Head Room

Human Performance Machine Performance

Walking–Walking

Conversation-Walking

Conversation–Conversation
Gait Experiments

gait video

conversation video

ονλψ

body only face

CG

CC

GG

Static Face 
Next Directions

• In hard cases (poor viewing conditions), humans 

take advantage of video, face, & video

• Evidence: algorithms do NOT take advantage of 

video, face, & body/gait

• Learn from the human visual system.

• Incorporate into algorithm design.
The Good, the Bad, and the Ugly

Still Face Challenge
Goal of GBU

• Encourage development of “hard” still frontal 

face recognition algorithms

• Improvement not at expense of “non-hard” 

images

• Three performance levels

– Good

– Bad

– Ugly

• Discover the “phantom” covariates to which 

humans appear immune.
Experiment Specifics

• Nikon D70-6 Mpixels

• Uncontrolled images

– Indoors

– Outdoors

• 9,307 pool of images

• 437 qualified subjects

• Images in MBGC

• Images included in FRVT 2006

• Select by FRVT 2006 algorithms
Experiment Specifics

• Same number of images per subject

– Each Sig Set

– Each Partition

• Variation in performance on image attributes

Data Set Target Size Query Size

The Good 1085 1085

The Bad 1085 1085

The Ugly 1085 1085
Face Pairs

Good Challenging Very Challenging
Face Pairs

Good Challenging Very Challenging
Face Pairs

Good Challenging Very Challenging
Good, Bad, Ugly Performance
GBU Fusion ROC
Big “Four” Problems in Face Recognition

A – Aging (time lapse)

P – Pose

I – Illumination

E - Expression 
“Four” Big Problems in Face Recognition

A – Aging Collected same academic yr

P – Pose All frontal images

I – Illumination

E - Expression 
Lighting & Expression

Same lighting, Same expression

Same lighting, Different expression Different lighting, Different expression

Different lighting, Same expression
Lighting & Expression

Same lighting, Same expression Different lighting, Different expression

Verification rate @ FAR = 0.1%
Lighting & Expression

Same lighting, Same expression Different lighting, Different expression

Verification rate @ FAR = 0.1%
Lighting & Expression

Verification rate @ FAR = 0.1%
What is the quality of these images? 
Hard to Match

Easy to Match

Hard and Easy to Match
Quality comes in Pairs

Hard to Match

Hard to Match
Quality comes in Pairs

Easy to Match Easy to Match
Quality comes in Pairs

Hard to Match

Easy to Match Easy to Match

Hard to Match
• Human subject raters respond...

– 1. sure they are the same person

– 2. think they are the same person

– 3. not sure

– 4. think they are not the same person

– 5. sure they are not the same person

Human Performance

Procedure
GBU Human Performance
CSU/NIST GBU Baseline Algorithm

Local Region PCA Algorithm

•13 Local Features +Whole Face

•Self Quotient - Lighting Removal

•PCA based whitening 

•250 basis vectors per feature.

•3500 total basis vectors.

•Fisher Criterion Weighting

•All features combined

•Similarity based upon Correlation

Local Regions

Self Quotient Preprocessing
Performance on GBU

Sample match from Good Data

Sample match from Challenging 

Data

Sample match from Very 

Challenging Data
From Face to Ocular GBU Baseline 

Algorithm

Whole Face 14 Local Regions Left Ocular 3 Local Regions
MPF Ocular `94, `98
Performance on GBU

Sample match from Good Data

Sample match from Challenging 

Data

Sample match from Very 

Challenging Data
GBU Performance

• Three partitions

– Same subjects

– Differences are image covariates

– More than expression and lighting direction

• Human Performance

– Bad & Ugly partitions statistically not different

• Machine Performance

– Bad & Ugly partitions different

• Humans “Blind” to Algorithm Differences

• Human Performance as Benchmark
Ocular
Ocular Data

• Individual frames of IOM data can be used 

increase or decrease the level of difficulty in 

ocular recognition.

• Measure improvement over iris only recognition 

algorithm.
54

Ocular data from IOM

Near Infrared (NIR) Video Sequence
55

Ocular data difficulty
56

Recognition performed across all 

combinations
Predicting Performance
Predicting Performance

Target Set

Query Set

New Target Set

New Query Set
59

Levels of Predictions

• General Assessment

• Measuring Improvement

• Ranking of Algorithms

– Relative performance

– Ranking stable across data sets

– Limited success

• Predict Performance
60

Considerations • Modeling – Demographics – Acquisition conditions – Queries to be processed • Deep questions – Ability to generalize? – Specific to algorithm? – Specific to task? • Links – Quality – Failure/error analysis – Biometric-completeness
Conclusions

• Challenges in Unconstrained Face Recognition

– Video: MBGC Video Challenge

– Still: Good, Bad, & Ugly

• Ocular

– Visible: GBU

– NIR: At a distance sequences

• Performance Prediction
Questions?