ICE Mining: 

Quality and Demographic Inve

ICE 2006 Performance R

P. Jonathon Phillips

NIST

Patrick J. Flynn

University of of Notre Dame

(NIST Associate, 2007-2008
Outline

• Motivation• Quality measure correlation• Quality measure effects on perfo

– Nested quality intervals– Disjoint quality quartiles

• Performance variations by demo• Conclusions and comments• Note: a report on this work is in p
Motivation• Iris image acquisition typically ex

controlled environment

– Cooperative subject (minimize iris o– Active lighting– Active focusing– Standoff manipulation

• Strong texture contrast & focus y

“good quality”

– Strong texture filter responses– Reliable phase estimates
Iris Quality in the Literature• Common biometric sample q

concepts

– Fidelity vs. application-specific crite– Methodology for quality based perfo

(Grother and Tabassi, PAMI 2007)

– Subject and sensor effects on quali

• Iris-specific aspects

– Focus (spectral content)– Occlusion (e.g., % iris), frontality, m– Wasserman 2006 (sensor quality), 

2006, Valencia 2007
Sample ICE 2006 iris subject 

Right Eye

Left Eye

LG EOU 2200

LG EOU 2200

LG EOU 2200 was industry recommended at the incepti
ICE 2006 data acquisition metho

LG EOU2200

• Take a shot of 3 iris images• If one or more is of sufficient quality, s

Sufficient 

Quality
ICE 2006 Image Quality Repo

Performer’s

Image Qualit

Module

Input Iris 

Image
ICE2006 Quality data

• Three competitive ICE 

2006 performers (SagemIridian, Cambridge and Iritech) (de-identified henceforth)

• 59,558 iris images• Each image has three 

quality scores (one per performer)
Mining Quality: Ge

properties
Should quality measures proddifferent algorithms be correla

One algorithm’s quality measure

Another algorithm’s quality measure

?
Quality measure scatter plot

Algorithm A 

Algorithm B

Algorithm C
Correlation of Quality score

Algorithms Pearson’s r SpeaA vs. B 0.122 0.13A vs. C 0.349 0.34B vs. C 0.120 0.10
Quality Score Correlation BeEyes
Quality effects 

matching perform
Step 1: Compute Image Qual

Performer’s

Image Qualit

Module

Input Iris 

Image
Step 2: Compute Quality Mat

5

3

3

524025561

6

1

2

5

4

5

2

5

0
Step 3: Compute global thrematching score

2

3

1

Compute threshold 

λ that yields FA

Complete similarity matrix
Step 4: Prune matching 

scores by quality

1

3

2

3

1

2

Fused quality threshold values: 5, 10, 

• 20 sub-experiments with nested sscores)• Compute FAR, FRR from global th

Complete Similarity matrix Subset by quality 
Calculation of FAR and FRR

• From unpruned set, compute thr

yields FAR = 0.001 (ICE 2006 op

• Let 

QF (g) and 

QF (p) be the quali

and query samples g and 

p

• Using 

λ, calculate FAR and FRR 

match pairs (g’, p’) with min{QF (g
Performance by Quality
Performance by Quality
Covariate Analyses

• Response of quality algorithms to subsets• Response of vendor matchers to subsets
Quality Covariate Study• Race and eye color• Three covariates

– East Asian– Caucasian w/Light Eyes– Caucasian w/Dark Eyes

• Quality scores normalized

– Empirical CDF
Quality Covariate Study
Classic Scoring of Results

Matching Scoring

FRR FAR 

Matching Scoring FRR 

FAR 

OO

FAR

FRRIs th

sig

Single Performance Point
Re-sampling Techniques

List 

of 

Performanc

Points

Matching Scoring

Global Threshold 

λ

@ FAR = 0.001 
Error Ellipse

FAR

FRR
Ellipse

• Is the level of difference significant?

Case 1: Yes Case 
Performance Covariate Study• Race, eye color, eye• Covariates

– East Asian– Caucasian w/Light Eyes– Left eye– Right eye

• Measure effect

– FAR– FRR
Performance variations by combmatcher and demographic• For each matcher

– Compute similarity threshold that yi

for entire data set

– For each demographic category in {

Caucasian Light eyes}

• Divide match pairs with target and que

category into 60 equal-sized subsets o

• For each subset

– Compute and plot FAR, FRR for each 

threshold
Performance Covariate Study• First look 

– East Asian– Caucasian w/Light Eyes

• Four groupings

– Left eye -- East Asian– Right eye -- East Asian– Left eye -- Caucasian w/Light Eyes– Right eye -- Caucasian w/Light Eye


Performance Covariate Study• Next look 

– Left eye– Right eye

• Four groupings

– Left eye -- East Asian– Right eye -- East Asian– Left eye -- Caucasian w/Light Eyes– Right eye -- Caucasian w/Light Eye


Quartile Quality Study• Effect of Quality

– FAR– FRR

• Bracket Quality by Quartile

– High Quality Quartile

• 25% highest quality samples

– Low Quality Quartile

• 25% lowest quality samples

– Disjoint quality intervals; no match
Algorithm B and Quality Measure 
Algorithm A and Quality Measur
Error estimation: Data-imposlimitations• Number of Non-Matches (impostors) in ICE 20

1:1000 

Number of false accepts

562,301 

60 partitions 9400 

Eye, race, eye color

1000

False ac

Expected false accepts
Observations and Conclusio

• Initial examination of ICE 2006 q• Iris image quality affects perform

(general trends, from aggregateperformance data)

– FAR decays with quality @ fixed F– FRR nearly invariant for a range o

after an initial drop, at fixed FAR

• Also:

– Demographic effects for quality me– Demographic effects on FAR

• Non-match distribution affected 

demographics (not presented he
Conclusions (contd.)

• Iris image quality measurement 

research and thorough testing

– Lack of correlation between three I

responders suggests that they werdifferent aspects of quality, or meadifferent degrees of accuracy

– Opportunities:

• for further research• Fusion

• Quality is not in the eye of the be

the recognition performance figu
ICE Mining• Should enable development 

structural models, with speciaanalyses

– e.g., Generalized Linear Mixed 
Thank Yo
Robert Frost, Harper’s Magaz

Some say the world will end Some say in ice.From what I've tasted of desI hold with those who favor fBut if it had to perish twice,I think I know enough of hateTo know that for destruction Is also greatAnd would suffice.