e and Iris Evaluation Activities 

at NIST 

Dr. P. Jonathon Phillips - NIST 

3 May 2006 

CardTechSecurTech 2006 
C, FRVT 2006 & ICE Sponsors 

ting Agency 

Director of National Intelligence Intelligence Technology Innovation Center 

ITIC

oring Agencies 

– Science & Technology Directorate – Transportation Security Administration 
C and ICE Team 

ogram Manager for FRGC and ICE

P. Jonathon Phillips — NIST

aluation Team

Todd Scruggs — SAICMatt Sharpe — SAICWilliam Worek — SIACKevin Bowyer — University of Notre DamePatrick Flynn — University of Notre DameRoss Beveridge — Colorado State UniversityAlice O’Toole — University of Texas at Dallas

GC and ICE Liaison

Cathy Schott — Schafer Corp
ine

e Recognition Grand Challenge (FRGC) 

http://face.nist.gov/frgc 

us of the Face Recognition Vendor Test (FRVT) 

6 

http://face.nist.gov/frvt2006 

mparison of Human and Computer Performance 

http://face.nist.gov/frgc 

Challenge Evaluation (ICE) 2005 and 2006 

http://iris.nist.gov/ice 
ce Recognition Grand Challenge 

Overview 
GC and FRVT 2006

hat is the difference between FRGC

d FRVT 2006?

FRGC (May 2004 – March 2006)

– Still and 3D face recognition algorithm

development project

FRVT 2006 (30 January 2006)

• Independent government evaluation of face

recognition systems

– Measure progress since FRVT 2002
GC Background 

newed interest in developing new methods 

automatic face recognition 

Fueled by advances in 

• Computer vision techniques • Computer design • Sensor design • Interest in fielding face recognition systems 

w techniques have potential to significantly 

uce error rates 
uly 2002

 Oct 2005 –

 Dec 2006 

 May 2004 – 

Mar 2006 

ckground 

aseline 

Independent 

Evaluation

Technology 

Development 
GC Objective 

he primary objective of the FRGC is to: 

velop still and 3D algorithms to improve 

performance an order of magnitude over 

FRVT 2002 
ect Point to Measure 

fication rate at : 

alse accept rate = 0.1% 

y 2002: 

0% error rate (80% verification rate) 

al: 

% error rate (98% verification rate) 
mages

MATCH

3D Face 

Recognition 

Algorithm 

3D Sensor 

Shape

Texture

D Image 
GC Modes Examined

3D Full Face

Multiple Stills

gle Still

3D Single

view

Outdoor/ 

Uncontrolled 
GC Experiments

 Controlled indoor still versus indoor still  Multiple still versus multiple still 

 3d versus 3D 

3t - Texture only 3s - Shape only 

+ 

 Uncontrolled still versus indoor still 

= 
Number 

FRGC Participation 

116

52

21

0

50

100150

Participation 

Access to v1.0aAccess to v2.0Results Submitted

GC Participation 
Exp 1 Exp 2 Exp 3 Exp 3t Exp 3s Exp 4 

E

Experiment Experiment

11

xp 5 Exp 6

GC Progress 

University 

Jan 2005 

0.0 0.2 0.4 0.6 0.8 1.0 

Verification rate @ FAR = 0.001 

Baseline Median Max 

University 

Aug 2005 

17 11 10

4

5 12
GCv2 Exp. 1 

Exp 1 Composite Performance 

095 096 09098 099 00 

ROC I ROC III 

Algorithm 
R = 0.1%

 Evaluations

andard)

Performance Goals and Progress

Point 80%

ed in

002
I 200s 

Performance Goals and Progress 

Independent Evaluations 

(Gola �standard) 

98 % ___

To be measured 

by FRVT 2006 

____,.

Starting Point 80%

FR � Measured in

VT g FRVT 2002

r ==i· 

FAR= 0.1% 
Point 80%

ed in

002

R = 0.1%

 Evaluations

andard)

Performance Goals and Progress

al 98%

asured

T 2006

99.99% Multi-Still

(Mar 06)

98% Three-Dimensional

(Mar 06)

Face Recognition Grand Challenge

(Qualified Results)

99% High Resolution Still

(Mar 06)

* First set of results after 4 months in a 12 month period
Point 80%

ed in

002

R = 0.1%

 Evaluations

andard)

Performance Goals and Progress

al 98%

asured

T 2006

99.99% Multi-Still

(Mar 06)

98% Three-Dimensional

(Mar 06)

Face Recognition Grand Challenge

(Qualified Results)

99% High Resolution Still

(Mar 06)

* First set of results after 4 months in a 12 month period
mmary

ce Recognition Grand Challenge

Order of magnitude increase in

performance

Systematically investigate still and 3D

Formulate series of challenge problems

Face Recognition Grand Challenge

Completion March 2006
VT 2006

est in a series of large scale independent

aluations for face recognition systems

Previous evaluations in the series were the

FERET, FRVT2000, and FRVT 2002

mary goal is to

Measure progress of prototype systems/algorithms

and commercial face recognition systems sinceFRVT 2002

Conduct comparison across modalitiesCompare performance with FRGC goals
VT 2006 Status Update

e Face Recognition Vendor Test

RVT) 2006

Began on 30 January 2006Currently underway

• Testing executables at this time

22 Participants

• 10 countries• 30% of Participants are from Academia
Human-Computer Comparison 

, Phillips, Jiang, Penard, Ayyad, Abdi 2005 
blem

e face recognition algorithms ready for

plications?

enormous improvements over last decadeaccuracy of algorithms tested intensively

ow accurate do they have to be to be useful?

meet or exceed human performance
man-Machine Comparisons 

ame image pairs from Exp. 4

even state-of-the-art algorithms

– 4 from industry– 3 from academic institutions

Comparisons

– 120 difficult face pairs– 120 easy face pairs
mpling

mogeneous

caucasian males/females 20-30 yrscomparisons made on identity not

• age, race, sex

muli

240 pairs of faces

• 120 male pairs

– 60 easy– 60 difficult

• 120 female pairs

– 60 easy– 60 difficult
1

2345

cedure

man subject raters respond...

. sure they are the same person. think they are the same person. not sure. think they are not the same person. sure they are not the same person
Identity Matching for Difficult Face Pairs 

 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 

False Accept Rate 

NJIT CMU Viisage Human Performance Algorithm A Algorithm B Algorithm C Algorithm D Chance Performance 
ults Summary 

3 algorithms surpass humans!

– NJIT (Liu, IEEE: PAMI, in press

)

– CMU (Xie et al., 2005)– Viisage (Husken et al., 2005)

 less accurate than humans
Identity Matching for Easy Face Pairs 

 

 

6 

  

0 

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 

False Accept Rate 

CMU NJIT Algorithm B Algorithm D Viisage Algorithm C Algorithm A Human Performance Chance Performance 
clusions

orithms compete favorably with humans on

 difficult task of matching faces across

anges in illumination

ome algorithms are better than humans on “difficult”

ace pairs

early all are better than humans on “easy” face pairs
Challenge Evaluation 

erview 
Goals 

ad Goals

acilitate iris recognition technology

evelopment

echnology assessment of iris recognition

deled after FRGC/FRVT 2005

RGC (Face Recognition Grand Challenge)RVT 2006 (Face Recognition Vendor Test

006)
Algorithm Similarity Matrix

y Automatic 

Input 

Output 

Target Set 

Query Set 
ge Quality

Image Quality

Module

Scoring

Real

Number

Quality

Measure

put Iris

mage

Similarity 

Matrix 
2005 and 2006 

hat is the difference between ICE 

ase I 2005 and ICE Phase II 2006? 

CE 2005 – Technology Development 

– Iris recognition challenge problems – Iris data set 

CE 2006 - Evaluation 

– Independent government technology evaluation – Sequestered data 
 2005 Challenge Problems 
ne Experiments 

Exp 1 Exp 2 

Right Eye Left Eye 

24

425 Iris Images 

Individuals 120

1528 Iris Images 

Individuals 

112 Overlapping Individuals 

132 Total Individuals 
2005 

llenge Problem 

Open book 

a Released September 2005 

is images 

xperiments 

Ground truth 

ilarity Matrices Submitted March 2006 

Generated by participants Scored by NIST 

T an independent Evaluation 

NO sequestered data 
ICE Participation 

70 .,.--_!6_6 

60 

;;;;;;-------

'- 50 

G) 

.c 40 

§ 30

z 20

10 

o�-

42 

9 

Participation 

■ Interested Groups I 

■ ICE 2005 Data 

Results Submitted I 

ICE Participation 
ult Submissions 

ults submitted:

 Groups

5 Algorithms + 1 irisBEE Baseline

 Countries

 Phase I Participants:

Cambridge University (Cam 1, Cam 2)Carnegie Mellon University (CMU)

Chinese Academy of Sciences, Center for InformationScience (CAS 1, CAS 2, CAS 3)

ndiana University, Purdue University, Indianapolis (IUPUI)

itech (IritchA, IritchB, IrtchC, IritchD)

PELCO (Pelco)SAGEM - Iridian (SAGEM)West Virginia University (WVU)Yamataki Corp / Tohoku University (Tohoku)
C Results - Fully Automatic 

Exp 1 Exp 2 

esults from Open Book Challenge Problem 

NOT Independent Evaluation 
C Results 

Exp 1 Exp 2 

esults from Open Book Challenge Problem 

NOT Independent Evaluation 
Plot Performance Results 

y Automatic, FAR=0.001 

esults from Open Book Challenge Problem 

NOT Independent Evaluation 
Plot Performance Results 

y Automatic, FAR=0.001 

esults from Open Book Challenge Problem 

NOT Independent Evaluation 
 Independence 

pose:

xamine relationship between left & right iris

hod:

or each subject, compute mean match score

• Right and left iris

or each subject, compute mean non-match score

• Right and left iris

Scatter plot of right verses left iris

• Mean match score• Mean non-match score
CAS 1

Iritech D

F

 Independence - Iritech 
CAS 1

Iritech D

F

 Independence-CASIA 
 Independence-Summary
lity Measures 
2006 Schedule 

pril 2006

CE 2006 Protocol released

June 2006

Executables submission deadline

CE 2006 evaluation begins

cember 2006

CE 2006 Final Report released