Face Recognition Vendor Te

Face Recognition Vendor Te

Experiment 4 Covariate S

Experiment 4 Covariate S

Dr. J. Ross Beveridge Dr. J. Ross Beveridge

Dr. GeofH. Givens H. Givens

Dr. Bruce Draper Dr. Bruce Draper

Mr. Yui Man Lui

Colorado State University Colorado State University

Dr. P. Jonathon Phillips Jonathon Phillips

National Institute of Standards and Te

National Institute of Standards and Te

The work was funded in part by the Technical Support Working Group (TSWG) 
Motivation Motivation

Factors that influence face recog

Factors that influence face recog
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Motivation Motivation -Attributes of People Attributes of Peopl

What makes recognition harder/ea

What makes recognition harder/e

Young ...
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Motivation Motivation -Attributes of People Attributes of Peopl

Gender? Gender?

Age
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Motivation Motivation -Attributes of People Attributes of Peopl

Race?

Age Gender
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Motivation Motivation -Smile?

Expression? Expression?

Age Gender Race
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Motivation Motivation -Environment Environment

Control: Mugshot vs. posed indoo

Control: Mugshot vs. posed indoo

Age Gender Race Express
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Motivation Motivation -Glasses Glasses

Age Gender Race Express

Glasses in uncontrolled imagery. Glasses in uncontrolled imagery.
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Motivation Motivation -Recap

Age Gender

Race Expression

Uncontrolled Glasses

Gender

Race Expression

Uncontrolled Glasses
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

But Wait, There But Wait, There’s More, Quality s More, Quality

 You cannot do much about You cannot do much about

 Gender, Age, Race, Gender, Age, Race, ...

 Some control over Some control over

 Setting, Glasses, Expression, Setting, Glasses, Expression, ...

 What about measurable image propert

What about measurable image propert

 Resolution, Focus, Resolution, Focus, ...

ISO SC 37 "Biometrics” - Factors Affecting Face Imag

ACQUISITION PROCESS AND CAPTURE DEVIC

...2. physical properties (e.g. resolution and con

ISO SC 37 "Biometrics 37 "Biometrics”-Factors Affecting Face Imag

Factors Affecting Face Imag

ACQUISITION PROCESS AND CAPTURE DEVICE 

ACQUISITION PROCESS AND CAPTURE DEVIC

...

2. physical properties (e.g. resolution and cont

2. physical properties (e.g. resolution and cont
Covariate Analysis Covariate Analysis

4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

For Analysis We Need For Analysis We Need ...

 Lots of Performance Data Lots of Performance Data 

 FRVT 2006 FRVT 2006

 Methodology Methodology

 Generalized Linear Mixed Effect Model Generalized Linear Mixed Effect Model

 Specific Problem Specific Problem

 Uncontrolled frontal still against mugsh

Uncontrolled frontal still against mugsh
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Introduction Introduction -More on FRVT More on FRVT

22 participants, 22 participants, 22 participants, 10 

10 

10 

China Denmark Denmark Germany Germany Israel J

Romania Romania Spain South 

Korea

United 

Kingdom

US

10 US, 7 foreign, 5 with offices in and ou

10 US, 7 foreign, 5 with offices in and ou

Executing Agency
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Introduction Introduction -Progress Progress

2006 - Falsely turn away 1/100 people,

when only admitting 1/1000 imposters.

2006 -Falsely turn away 1/100 people, Falsely turn away 1/100 people,

when only admitting 1/1000 imposters. when only admitting 1/1000 imposters.
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Our Focus Our Focus -Uncontrolled Stills Uncontrolled Stills
FRVT 2002 Performance

(Controlled vs Controlled

4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Uncontrolled to Controlled Still Uncontrolled to Controlled Still

2006 2006- Falsely turn away 10/100 to 40/1

-

when only admitting 1/1000 impo

Falsely turn away 10/100 to 40/1

Falsely turn away 10/100 to 40/1

when only admitting 1/1000 impo

when only admitting 1/1000 impo
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

FRVT Covariate Analysis FRVT Covariate Analysis

 Algorithm Algorithm - score fusion of 3 top p

score fusion of 3 top p

 Imagery Imagery - Uncontrolled match to C

Uncontrolled match to C

 Subset of FRVT 2006 Experiment 

Subset of FRVT 2006 Experiment 

 345 subjects and 345 subjects and 110,514 110,514 match s

match s
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Performance Variable Performance Variable

 Verification Outcome: Success / F

Verification Outcome: Success / F

Verified FAR 

Yes 1/100 

Verified FAR No 1/1,000 1/1,000 Verified FAR 

Yes 1/10,000 1/10,000 

* 

Levels 1/100 1

Levels 1/100 1

FAR i

FAR i

Distributed 

Distributed 

There are 110,514 pairs like this! There are 110,514 pairs like this! There are 110,514 pairs like this! 
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Face Region In Focus Measure Face Region In Focus Measure

FRIFM: Sum of FRIFM: Sum of Sobeledge magnit

edge magnit

ellipse bounding the face. ellipse bounding the face. 

“Active Computer Vision by Cooperative Focus and Stereo” 
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Face Region In Focus Measure Face Region In Focus Measure

Low FRIFM examples High FRIF
Fitting a Statistical Model Fitting a Statistical Model

4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Generalized Lin

Mixed Mode

Generalized Lin

Generalized Lin

Mixed Mode

Mixed Mode

......

Verification 

Outcomes 

Verification 

Outcomes

Age Age Gender Gender

Focus Focus Glasses Glasses Race Race 

Resolut

Resolut

FAR FAR
Using the Statistical Model Using the Statistical Model

4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Fitted Generalized Linear Fitted Generalized Linear 

Mixed Model 

Fitted Generalized Linear 

Mixed Model Mixed Model

Age Age Gender Gender

Focus Focus Glasses Glasses Race Race Hea

He

Resolution Resolution

FAR FAR

P(success | covariates)
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

 Let A and B be 2 covariates that might 

be 2 covariates that might 

algorithm performance. For example, A

algorithm performance. For example, A

(categorical) and B=Query (categorical) and B=Query-Eye-Distanc

Distanc

 Let a index levels of A. Let a index levels of A.

 Let j index the FAR setting, Let j index the FAR setting, αj Ypabj is 

 1 if Person 1 if Person p is verified correctly, 

is verified correctly, 

 Ypabj depends on: depends on:

 person p, covariates , covariates A and B, an false alarm rate false alarm rate αj.

Analysis is: Mixed Effects Logistic R

with Repeated Measures 

Generalized Linear Mixed Mode

Generalized Linear Mixed Mod
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

GLMM Model Continued GLMM Model Continued ...
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

The outcomes, i. e. verification successuncorrelated when testing different peocorrelated when testing the same persodifferent configurations. 

The outcomes, i. e. verification success

The outcomes, i. e. verification success

uncorrelated when testing different peo

uncorrelated when testing different peo

correlated when testing the same perso

correlated when testing the same perso

different configurations. different configurations.

This means: This means:The Mixed in Generalized Linear The Mixed in Generalized Linear Mixe

Subject Variation Subject Variation
Vendor Test Covariate Analysis Fi

Vendor Test Covariate Analysis Fi

From the highly expected From the highly expected 

...

... 

... to the unexpected. to the unexpected.
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Finding 1: False Accept Rate Finding 1: False Accept Rate

Solid = Indoors

Dashed = Outdoors
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Finding 2: Gender Finding 2: Gender

Solid = Indoors

Dashed = Outdoors
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Finding 3: Race Finding 3: Race

Solid = Indoors

Dashed = Outdoors
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Finding 4: Glasses Finding 4: Glasses

Solid = 

Dashed = 
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Finding 5: Finding 5:

Small Medium 
Small Medium 

Size of query imag

(distance between e

4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Finding 5: Finding 5:
Query environment

4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Finding 5: Finding 5:

Small Medium 
B

4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Finding 5: Finding 5:Small Medium 
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Finding 5: Finding 5:

Small Medium 
Low FRIFM good; even for 

4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Finding 5: Finding 5:Small Medium 
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

FRIFM Conclusion FRIFM Conclusion

 Large performance variation. Large performance variation.

 Indoors [>0.95, ~.0.70] Indoors [>0.95, ~.0.70]  Outdoors [~0.90, ~0.10]. Outdoors [~0.90, ~0.10]. 

 Interaction between covariates Interaction between covariates

 Environments (indoors, outdoors) Environments (indoors, outdoors Query image size Query image size Target and query FRIFM Target and query FRIFM

 Low FRIFM good Low FRIFM good

 Effect if control for only one imag

Effect if control for only one imag

 Outdoors: query size very importa

Outdoors: query size very importa
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

FRIFM Conclusion FRIFM Conclusion

 According to this analysis According to this analysis Out of focus is higher quality Out of focus is higher quality Remember, edge density surroga

Remember, edge density surroga

 Is this really quality, Is this really quality, ... Or other environmental factors, Or other environmental factors, ... Or algorithm aberration? Or algorithm aberration?
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

GLMM and Quality Standards GLMM and Quality Standards

Factors Affecting Face Image Quality 

Character 

RICHNESS OF IDENTIFYING CHARACTERISTIC Š BIOLOGICAL CHARACTERS 

Behavior 

SPOOFING 

Imagin

ACQUISITION PROCAPTURE DEVICE PROPERTIES 

FACE 

1. anatomical characteristic (e.g. head dimensions, eye position) 2. injuries and scars 3. ethnic group 4. impairment 5. Heavy facial wears, such as thick or 

dark glasses 

1. closed eyes 

2. (exaggerated) expression 3. hair across the eye 4. head pose 5. makeup 6. subject posing (frontal / nonfrontal to camera) 

1. image enhancemreduction process 2. physical propertieresolution and contra3. optical distortions 4. static properties obackground (e.g. wa5. camera character

• sensor resolutio

6. scene characteris

• geometric disto

From Covariates to Quality Measure

From Covariates to Quality Measure

From Covariates to Quality Measure
4/21/2008 FRVT 2006 Experiment 4 Covariate Analysis 

Conclusion Conclusion

 Quality is NOT in the eyes of the 

Quality is NOT in the eyes of the 

 It is in the performance numbers It is in the performance numbers Model quantifies performance cha

Model quantifies performance cha

 Turn the knob. Turn the knob.  Read off the change in performan

Read off the change in performan

 Interaction between covariates. Interaction between covariates.

 Tells us where to put our efforts Tells us where to put our efforts

 Indoors it is FRIFM. Indoors it is FRIFM. Outdoors it is Query Image Size. Outdoors it is Query Image Size.

 These models are used in other f

These models are used in other f

 e.g., Biomedical. e.g., Biomedical.

 Studies of Biometrics should use 

Studies of Biometrics should use 
Thank You Thank You