FRGC and ICE Workshop 

Dr. P. Jonathon Phillips - NIST 

March 22-23, 2006 

NRECA Conference Facility 

Arlington, Virginia 
 

 

 

 

 

FRGC, FRVT 2005 & ICE Sponsors 

Executing Agency 

Sponsoring Agencies 

Director of National Intelligence 

Intelligence Technology Innovation Center 

ITIC 

– Science & Technology Directorate 

– Transportation Security Administration 
 

 

 

 

 

 

 

 

 

FRGC and ICE Team 

• Program Manager for FRGC and ICE 

• P. Jonathon Phillips — NIST 

• Evaluation Team 

• Todd Scruggs — SAIC 

• Matt Sharpe — SAIC 

• William Worek — SIAC 

• Kevin Bowyer — University of Notre Dame 

• Patrick Flynn — University of Notre Dame 

• Ross Beveridge — Colorado State University 

• Alice O’Toole — University of Texas at Dallas 

• FRGC and ICE Liaison 

• Cathy Schott — Schafer Corp 
ICE Participation 

ICE Participation

66

42

9

0

10

20

30

40

50

60

70

Signed Up Obtained

Data

Submitted

Results

ICE Participants

Number

Number of

Participants
 

ICE Participation 

• Results received on ver1.0 in March 2006 
 

 

FRVT 2006 Update 

• The Face Recognition Vendor Test 

(FRVT) 2006 

– Began on 30 January 2006 

– Currently underway 

• Testing executables at this time 

– 22 Participants 

• 10 countries 

• 30% of Participants are from Academia 
Iris Challenge Evaluation 

Overview 
 

 

 

 

 

ICE Goals 

• Broad Goals 

– Facilitate iris recognition technology 

development 

– Technology assessment of iris recognition 

• Modeled after FRGC/FRVT 2005 

– FRGC (Face Recognition Grand Challenge) 

– FRVT 2006 (Face Recognition Vendor Test 

2006) 
Questions Examined 

Target Set 

Query Set 

Input 

Algorithm Similarity 

Matrix 

Output 

Recognition 

Image Quality 
 

 

 

 

ICE 2005 and 2006 

• What is the difference between ICE 

Phase I 2005 and ICE Phase II 2006? 

– ICE 2005 – Technology Development 

– Iris recognition challenge problems 

– Iris data set 

– ICE 2006 - Evaluation 

– Independent government technology evaluation 

– Sequestered data 
ICE 2005 Challenge Problems 
Define Experiments 

Exp 1 Exp 2 

Right Eye Left Eye 

1425 Iris Images 

124 Individuals 

1528 Iris Images 

120 Individuals 

112 Overlapping Individuals 

132 Total Individuals 
 

 

 

 

 

Define Experiments 

• Exp 3 and 4 

– Right iris verses left iris 

– Left iris verses right iris 

• Purpose 

– Examine right-left iris independence 

– Analysis not included in today’s 

presentation 
Iris Challenge Evaluation 

• Fully Automatic 

• Quality Metric 
Fully Automatic 

Target Set 

Query Set 

Input 

Algorithm Similarity

Matrix 

Output

Image 

Image 
Image Quality 

Image Quality 

Module 

Scoring 

Real 

Number 

Quality 

Measure 

Input Iris 

Image 

Similarity 

Matrix 
ICE 2005 Results 
 

 

 

ICE 2005 

• Challenge Problem 

– Open book 

• Data Released September 2005 

– Iris images 

– Experiments 

– Ground truth 

• Similarity Matrices Submitted March 2006 

– Generated by participants 

– Scored by NIST 

• NOT an independent Evaluation 

– NO sequestered data 
 

 

 

 

 

 

 

 

Result Submissions 

• Results submitted: 

– 9 Groups 

– 15 Algorithms + 1 irisBEE Baseline 

– 6 Countries 

• ICE Phase I Participants: 

– Cambridge University (Cam 1, Cam 2) 

– Carnegie Mellon University (CMU) 

– Chinese Academy of Sciences, Center for Information 

Science (CAS 1, CAS 2, CAS 3) 

– Indiana University, Purdue University, Indianapolis (IUPUI) 

– Iritech (IritchA, IritchB, IrtchC, IritchD) 

– PELCO (Pelco) 

– SAGEM - Iridian (SAGEM) 

– West Virginia University (WVU) 

– Yamataki Corp / Tohoku University (Tohoku) 
 

 

 

 

Hidden Test 

• Find all mislabeled irises 

246240.tiff 

• Accidentally included in Exp 2 

• Error corrected in Exp 2 mask matrix 

1 Error in 2953 image!! 
ROC Results - Fully Automatic 

Exp 1 Exp 2 

Results from Open Book Challenge Problem 

NOT Independent Evaluation 
ROC Results 

Exp 1 Exp 2 

Results from Open Book Challenge Problem 

NOT Independent Evaluation 
Bar Plot Performance Results 

Fully Automatic, FAR=0.001 

Results from Open Book Challenge Problem 

NOT Independent Evaluation 
Bar Plot Performance Results 

Fully Automatic, FAR=0.001 

Results from Open Book Challenge Problem 

NOT Independent Evaluation 
Bar Plot Performance Results 

Fully Automatic, FAR=0.0001 

Results from Open Book Challenge Problem 

NOT Independent Evaluation 
Bar Plot Performance Results 

Fully Automatic, FAR=0.0001 

Results from Open Book Challenge Problem 

NOT Independent Evaluation 
Bar Plot Performance Results 

Manual Intervention, FAR=0.001

Results from Open Book Challenge Problem 

NOT Independent Evaluation 
 

 

 

 

Eye Independence 

• Purpose: 

– Examine relationship between left & right iris 

• Method: 

– For each subject, compute mean match score 

• Right and left iris 

– For each subject, compute mean non-match score 

• Right and left iris 

– Scatter plot of right verses left iris 

• Mean match score 

• Mean non-match score 
CAS 1

Iritech D

F

Eye Independence - Iritech 
Eye Independence-CASIA 

CAS 1 

Iritech D 

F 
Eye Independence-Summary 
Quality Measures 
 

 

 

 

ICE 2006 Schedule 

• Today 

– Key points in afternoon talk 

• 1 April 2006 

– ICE 2006 Protocol released 

• 15 June 2006 

– Executables submission deadline 

– ICE 2006 evaluation begins 

• December 2006 

– ICE 2006 Final Report released 
 

 

 

 

 

Conclusion 

• ICE - Technology Development 

• ICE 2006 – Independent Government 

Evaluation 

– Modeled after FRVT 2006 

• Goals 

– Facilitate technology development 

– Technology assessment of iris recognition