March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Human versus Machine Perfo

Alice J. O’Toole

The University of Texas at D
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Acknowledgemen

• In collaboration with

– P. J. Phillips - NIST– Fang Jiang - Univ. of Texas at Dal– Nils Pénard - UTD– Janet Ayyad - UTD– Hervé Abdi - UTD

• Work supported by TSWG
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Overview

• rationale• Face Recognition Grand Challen• human-machine comparison
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Problem

• Are face recognition algorithms re

security applications?

– enormous improvements over last d– accuracy of algorithms tested intens

• How accurate do they have to be t

– meet or exceed human performance 
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Why?

• humans are the competition!

– human-machine comparisons virtually n

• putting algorithms in the field

– security improved or put at greater 
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

How accurate are algor
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

U.S. Government-sponsored Co

• standardize comparisons

– test multiple algorithms– identical, LARGE sets of face ima– Face Recognition Grand Challenge 

• (2004-ongoing) 
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Present work

• purpose 

– extend standardization of FRGC to 

humans and algorithms on a challerecognition task

– matching face identity across chan

illumination (FRGC Exp. 4)
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Why Illumination Chan

• recognized to be difficult for:

– humans (e.g., Braje et al., 2000; Troje & Bülthoff, 19– algorithms (e.g., Phillips et al. 2005; Gross et al. 20
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Most Challenging FRGC Exp

• controlled illumination experim

– match images with controlled illum– 20 participating algorithms – median performance of

• .91 verification rate• .001 false acceptance rate 
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

• uncontrolled illumination (Exp. 4

– match images with controlled and 

illumination 

– 7 participating algorithms– median performance

• .42 verification rate• .001 false acceptance rate 
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

FRGC Uncontrolled Illumina

• Match identity in target and prob

– target - controlled illumination– probes - uncontrolled illumination 
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Specifics

• similarity matrix

– target (n = 8014)– probe (n =16028)– s(i,j) = similarity between the ith an

• 128,041,040 similarity scores• 407,352 of same people• remainder of different people
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Results

• ROC

– verification rate– false acceptance rate
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Algorithm Performan

0

0.20.40.60.8

1

0 0.2 0.4 0.6 0.8 

false acceptance
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Comparing Humans and Alg

• problem

– 128 million face pairs?

• solution

– sample face pairs

• most difficult• easiest
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Sampling

• homogeneous 

– caucasian males/females 20-30 yrs – comparisons made on identity not

• age, race, sex

• caution on the FRGC results
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Easy and Difficul

• PCA Baseline Algorithm

– scaled and aligned images (SAIC)– available and widely used since th– but not state-of-the-art 
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Match Pairs

• “easy” match pairs

– 2 “similar” images of same person 

• similarity scores > 2 sd above mean similari

• “difficult” match pairs

– 2 “dissimilar” images of same person 

• similarity scores < 2 sd below mean similari
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

No-Match Pairs

• “easy” no-match pairs

– 2 “dissimilar” images of different people

• similarity scores < 2 sd below mean similar

pairs

• “difficult” no-match pairs

– 2 “similar” images of different person

• similarity scores < 2 sd above mean similari

pairs
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

• Experiment 1 

– unlimited exposure time– male face pairs

• Experiment 2 

– 2 sec. exposure time– male and female face pairs 

• Experiment 3 

– 500 msecs. exposure time– male and female face pairs 
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Methods

• Stimuli

– 240 pairs of faces 

• 120 male pairs

– 60 easy– 60 difficult

• 120 female pairs

– 60 easy– 60 difficult
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

• Subjects

– 91 volunteers from UTD

• Expt. 1 

– n = 22 (12 males; 10 females)

• Expt. 2

– n = 49 (24 males; 25 females)

• Expt. 3

– n = 20 (10 males; 10 females)
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Procedure

• 2 faces appear side by side• Human subject raters respond...

– 1. sure they are the same person– 2. think they are the same person– 3. not sure– 4. think they are not the same pers– 5. sure they are not the same perso
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Results

• PCA predicts difficulty (d’ analy

– Experiment 1 

• F(1,20) = 19.78, p < .002

– Experiment 2 

• F(1,48) = 96.53, p < .0001

– Experiment 3 

• F(1,18) = 62.65, p < .0001
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Experiment 2 

Human Face Matching 

0

0.20.40.60.8

1

0 0.2 0.4 0.6 0.8 

false accept rate

difficult paireasy face p
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Experiment compari

• Humans no more accurate with u

time than with 2 secs. presentatio

– F(1,176)= 2.01, ns.

• Human accuracy declined with expos

500 msecs.

– F(1,176)= 26.97, p < .0001
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Stability of human perfo

• supports use of these data for ben

comparisons with machines
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Human-Machine Comp

• Seven state-of-the-art algorithms

– 4 from industry– 3 from academic institutions

• Comparisons

– 120 difficult face pairs– 120 easy face pairs 
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Results
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Identity Matching for Difficult Face Pairs

0

0.20.40.60.8

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 

False Accept Rate

NJITCMUViisageHuman AlgorithAlgorithAlgorithAlgorithChance 
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Results Summary

Difficult Face Pairs

• 3 algorithms surpass humans

– NJIT (Liu, IEEE: PAMI, in press)

– CMU (Xie et al., 2005)– Viisage (Husken et al., 2005)

• 4 less accurate than humans
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Identity Matching for Easy Face Pairs

0

0.20.40.60.8

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 

False Accept Rate

CMUNJITAlgorithm Algorithm ViisageAlgorithm Algorithm Human PeChance P
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Results Summary

Easy Face Pairs

• 6 algorithms surpass humans!• 7th less accurate than humans at hi

acceptance rates
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Human Attention

• Did attention waver during expe

– no correlation between accuracy a

• verification (r = .07, ns)• false acceptance rate (r = -.04, ns.)
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Are human skills overr

• “familiar” versus “unfamiliar” • unfamiliar matching

– correct task for comparing “huma

machine security systems

• evidence that human expertise fo

may be limited to recognizing “ffaces (Hancock et al., 2001; O’Toole et al., 2003
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Familiarization

• Can we improve human perform• Experiment 4

• select face pairs that generated errors • familiarize subjects with people in pa

– 5 exposures to one face in pair

• n = 77 subjects• results

– improvement, but not significant (F(1,7
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

0

0.10.20.30.40.50.60.70.80.9

1

0 0.2 0.4 0.6 0.8 

p(false alarm)

familiarizedunfamilariz

Human Performance with Familiarization
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Conclusions

• Algorithms compete favorably with h

difficult task of matching faces acrosillumination 

– some algorithms are better than humans 

face pairs

– nearly all are better than humans on “ea
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Implications

• Algorithms may improve securit

situations 

– even if they perform poorly in abso
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

Implications

• We accept on “face” value the ne

any algorithm that we put in the important security application 

• Tools available for testing huma

– We rarely do!?
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG

What next?

• Why?

– Analysis of the variability of algor– Which face pairs separate algorith

• Hybrid strengths & weaknesses 
March 22, 2006 Evaluating Algorithms with Human 

Benchmarks - supported by TSWG