Covariates & Quality Measures Covariates & Quality Measures 

Bruce A. Draper, J. Ross Beveridge, 

Yui-Man Lui, David Bolme, Geoff Givens 

Colorado State University 

P Jonathan Phillips P. Jonathan Phillips 

National Institute of Standards and Technology

Outline 

 Covariate Analysis (a quick review) 

 Methodology: GLMMs 

 Some typical results 

 Covariate Covariate Meta -analysis 

 Are the conventional wisdoms true? 

 Where is more work needed? 

 Quality measures 

 Properties 

 Illumination 

 Focus 

 This talk covers results from 3 p pa ers: 

 FRVT 2006: Quo Vidas Face Quality, To appear in Image and Vision 

Computing. 

 A Meta-analysis of Face Recognition Covariates, IEEE International 

Conference on Biometric Theory, Applications and Systems (BTAS), 2009 

 Quantifying How Lighting and Focus Affect Face Recognition Performance, 

submitted to IEEE Conference on Computer Vision and Pattern Recognition 

(CVPR) 

1
Age Gender

Setting (controlled/uncontrolled) Setting (controlled/uncontrolled) Expression

Resolution Resolution 

Expression 

Focus 

Setting Setting (controlled/uncontrolled) (controlled/uncontrolled) 

CovariatesCovariates-ExamplesExamples 

Covariate Analysis Covariate Analysis 

• Since 2003 Since 2003 

• Six papers Six papers 

• T wo j lourna j l

• Four conference Four conference 

• One workshop 

BEST PAPER 

• Three data Three data-sets/challe sets/challenge problems/evaluations nge problems/evaluations

• Ongoing collaboration betw Ongoing collaboration between CSU & NIST 

• FERET 

• FRGC 

• FRVT 2006 FRVT 2006

AWARD 

 Methodology Methodology 

 Generalized Linear Mixed Generalized Linear Mixed Effect Model Effect Model 

2
Solid= Indoors 

Dashed = Outdoors 

Covariate Effect Covariate Effect - Glasses

5 

Solid= Indoors 

Dashed = Outdoors 

Covariate Effect Covariate Effect - Race

6 

3
                   

                       

   

                           

   

Meta-analysis

 The quantitative synthesis or analysis of

results from multiple experiments or studies 

 Examples 

 Education – Bilingual Education (BRJ, 1997) 

 Medicine – Coronary Heart Disease (BMJ, 2000) 

 Face Recognition, Philips & Newton (AFGR, 

2002) 

 Concluded that the majority of FR research papers 

were working working on “easy ” problems problems and that testing of testing of 

novel algorithms should be accompanied by a control 

algorithm. 

 Iris Recognition, Newton & Phillips (BTAS, 2007) 

 Concluded the results from ITIRT, Iris ‘06, and ICE 

2006 are comparable. 

Meta-Analysis 

Methodology:

Step 1: Assemble over 100 candidate papers from 1993 through 2008

Step 2: 1st Filter Paper must relate a factor X to a measured change

in recognition performance.

Step 3: 2nd Filter We must be able to map the effect to a coarse but

common quantitative scale.

4
     

             

Covariates 

More on these 

Later ! Later...! 

Testing Conventional 

Wisdom 

Motivation:

Make sense of ...

"A Meta-Analysis of Face Recognition Covariates," Y. Man Lui, D. Bolme, B. A. Draper, J. 

R. Beveridge, G. Givens, P. J. Phillips, In Proceedings, Third IEEE International 

Conference on Biometrics: Theory, Applications, and Systems, 2009. 

Same year is as good as same week

5
               

Who is easier to recognize? Who is easier to recognize? 

Young ...

Conventional Conventional wisdom: wisdom: older subjects are easier to older subjects are easier to 

recognize than young ones recognize than young ones 

Analysis of Age as a 

Covariate 

Here is a glimpse of detail presented in Meta‐Analysis ...

... and this is more detail then appropriate here.

6
             

Summary of Findings 

AGE:

Older people are easier to recognize.

(9 Studies)

Elapsed Time 

The older the image, the poorer the match

target 

14 days 140 days 238 days 

Conventional wisdom: smaller Conventional wisdom: smaller time 

delays (query to target) are easier delays (query to target) are easier 

7
Summary of Findings 

AGE:

Older people are easier to recognize.

(9 Studies)

TIME BETWEEN IMAGES:

Recognition degrades

with time between images.

Months and years matter (8

Studies)

Gender 

Conventional Conventional wisdom: wisdom: men are easier to recognize men are easier to recognize 

than women than women 

8
Summary of Findings 

AGE:

Older people are easier to recognize.

(9 Studies)

TIME BETWEEN IMAGES:

Recognition degrades with

time between images. Months and years

matter. matter. (8 Studies) Studies)

GENDER:

Murky outcome, modest

and depends upon study,

algorithm, setting, etc.  (8

Studies)

Resolution 

Conventional wisdom: low resolution imagery Conventional wisdom: low resolution imagery is 

sufficient for icient for face recognition recognition 

9
GENDER:

Murky outcome, modest and

depends upon study, algorithm,

setting, etc.  (8 Studies)

RESOLUTION:

Older algorithms don’t

care. Newer algorithms

like more pixels. (10

Stud ) ies

Summary of Findings

AGE:

Older people are easier to recognize.

(9 Studies)

TIME BETWEEN IMAGES:

Recognition degrades with

time between images. Months and years

matter. matter.  (8 Studies)

Expression 

Neutral to Neutral Smiling to Smiling 

Smiling to Neutral Neutral to Smiling

Conventional wisdom: Conventional wisdom: 

always match always match neutral expressions neutral expressions 

10
       

Summary of Findings 

AGE:

Older people are easier to recognize.

(9 Studies)

TIME BETWEEN IMAGES:

Recognition degrades with

time between images. Months and years

matter. matter.  (8 Studies) Studies)

GENDER:

Murky outcome, modest and

depends upon study, algorithm,

setting, etc.  (8 Studies)

RESOLUTION:

Older algorithms don’t care. Newer

algorithms like more pixels. (10

Studies)

EXPRESSION:

        

   

Race 

Conventional Conventional wisdom: wisdom: Caucasians Caucasians are more dif are more difficult 

to recognize than East to recognize than East Asians 

11

Same expression better,

Otherwise smile/neutral

same. (4 Studies)
           

   

     

           

       

       

  

         

         

               

               

         

         

Meta-analysis of Race 

All of these studies confound 

Race with samp g lin effects

All systems trained on 

majority-Caucasian data sets 

All systems tested on 

majority majority-caucasian caucasian data sets

Summary of Findings 

AGE:

Older people are easier to recognize.

(9 Studies)

TIME BETWEEN IMAGES:

Recognition degrades with

time between images. Months and years

matter. matter.  (8 Studies)

GENDER:

Murky outcome, modest and

depends upon study, algorithm,

setting, etc. (8 Studies)

EXPRESSION:

RESOLUTION:

Older algorithms don’t care. Newer

algorithms like more pixels. (10

Studies)

RACE:

Same expression better,

Otherwise smile/neutral same. (4

Studies)

East Asians easier, BUT, this

may be because fewer East

Asians in data sets. (6

Studies)

12

(fewer possible East Asian confusions) 

Therefore, no conclusion is supported 
What is image quality? 

Depends on the 

query/target pair 

Smiling 

Size 

Focus 

13
Quality Measures 

 Quality measures should be: 

 Statistically predictive of success 

 Directly Directly computable computable from an image pair 

 Explainable 

 Operationally Controllable 

Quality Measures

Edge Density 

14
 

Edge Density Effects 

Algorithms and Location Matter 

FRVT 2006: Quo Vidas Face Quality” J. Ross Beveridge, Geof H. Givens, P. Jonathon Phillips, 

Bruce A. Draper, David S. Bolme, Yui Man Lui. Image and Vision Computing, Under Review. 

Edge Density: Why? 

 Why is edge density predictive of recognition 

performance across algorithms? 

 Why is lower edge density density better than high? 

 One post-hoc explanation: edge density as a 

focus measure 

 Implies that out-of-focus is good 

 Another post -hoc explanation: edge density as

an indirect lighting measure 

 Focus & illumination as possible quality 

measures 

15
Post-hoc Quality 

 In FRVT 2006, the 

best predictor of

recognition rate is 

date of acquisition. 

 Date of acquisition 

corresponds to

setting. 

Date of Acquisition 



 Image location (camera set up once per day) 

 Backgrounds Backgrounds 

 Illumination 

 Approximate time of day (short sessions) 

 Relates to lighting in outdoor settings 

 Possibly focus 

 Some sessions in better focus than others? 

 A good quality measure should be as good as 

date of acquisition on FRVT 2006, but 

generalizable across data sets 

Why? Date of acquisition subsumes: 

16
Illumination 

 Extensively studied 

 PIE (shown) & Yale B 

 Question: have modern algorithms “solved” lighting? 

0.067728 

Frontal 

-0.237399 

Right 

-0.23879 

Left 

where and 

Illumination Models 

Y.M Lui, JR Beveridge, and M. Kirby, Canonical Stiefel Quotient and its Application to Generic Face 

Recognition in Illumination Spaces, BTAS, 2009. 

T. Sim and T. Kanade, Combining Models and Exemplars for Face Recognition: An Illumination 

Example, CVPR Workshop, 2001. 

17
Lighting Direction as 

Quality Measure 

 Directly estimated 

from query image 

 Target images

had frontal 

illumination 

 Highly predictive 

of success 

 Explains most of

the information in 

setting (date) 

SEMC Focus Measure

 We developed a “true” focus measure 

s Out of Focu In Focus

18
Focus ≠ Edge Density 

 Focus less predictive than as edge density 

 Focus is subsumed by lighting 

Recent Conclusions 

 Lighting direction is an important quality 

measure 

 I li li h Implies lighti ng h as not b een “ sol ved” 

 Lighting direction explains previous edge 

density result 

 Edge density loses significance when lighting 

direction feature is added 

 Focus not significant in FRVT 2006. 

19
Summary 

 Some Covariates Matter 

 Age, Time Delay are important 

 Gender less Gender lessso 

 Resolution depends on algorithm 

 Race, Expression : more work needed 

 Quality Measures 

 It th li hti t id Its the lighting, stupid 

 Focus is insignificant in FRVT 

20