Welcome to the Third FRGC W

Dr. P. Jonathon Phillip

February 16, 2005
Sponsors

• Department of Homeland Securit• Federal Bureau of Investigation (• Intelligence Technology Innovatio

(ITIC)

• National Institute of Justice (NI• Technical Support Working Group 
Participation

• Results received from ver1.0a 

deadline

– 10 Participants– 32 Experiments

• Results received from ver2.0 

deadline

– 17 Participants– 53 Experiments

Participants and Exp

0

204060

Ver1.0a 

FRGC Vers

Number
3rd FRGC Workshop Agend

• FRGC Overview and Experiment Results• Introduction to FRVT 2005• Break • Guest Speakers

– Ross Beveridge, Colorado State University– David Kriegman, KBVT– Alice O’Toole, University of Texas at Dallas– Klaus Keus and Christoph Busch, German Fa

Testing

• Lunch• Technical Presentations• Adjourn
FRGC and FRVT 2005
Grand Challenge Team

• P. Jonathon Phillips—Director, F• Patrick Flynn—Notre Dame• Todd Scruggs—SAIC• Joe Marques—Mitre• Kevin Bowyer—Notre Dame• Jin Chang—Notre Dame• Kevin Hoffman—SAIC• Jaesik Min—Notre Dame• William Worek—SIAC
Outline

• Overview of Face Recognition 

Challenge (FRGC)

• Overview and Results of FRG• Introduction to the Face Rec

Vendor Test (FRVT) 2005
FRGC and FRVT 2005

• What is the difference betw

and FRVT 2005?

– FRGC (May 2004 – August 200

– Still and 3D face recognition algo

development project

– FRVT 2005 (August/Septembe

• Independent government evaluati

recognition systems

– Measure progress since FRVT 20
FRGC Background

• Renewed interest in developing ne

for automatic face recognition 

– Fueled by advances in 

• Computer vision techniques• Computer design• Sensor design• Interest in fielding face recognition sys

• New techniques have potential to 

reduce error rates 
FRGC Goal

• The primary goal of the FRGC 

Promote and advance face rec

technology designed to supporface recognition efforts in the Government 
FRGC Primary Objective

Develop still and 3D algorit

improve performance an o

magnitude over FRVT 2
Select Point to Measure

• Verification rate at :

– False accept rate = 0.1%

• Current:

– 20% error rate (80% verificatio

• Goal: 

– 2% error rate (98% verification 
Measuring Accuracy w/Error 

Rate of 2%

• Non-match scores: 

– Sufficient

• Match scores: 

– Need to design collection for suff

• Allows for error ellipses

• Minimal demographic analysis

1,000 match scores = ~ 20 err

10,000 match scores = ~ 200 er

50,000 match scores = ~ 1,000 er

50,000 match scores = ~ 1,000 er
Data Collection

Fall Semester

(Gallery)

15 Weeks

Spring 

(Pr

15 

200 

People

All match scores ~ 50,0
Modes Examined

Single Still Outdoor/

Uncontrolled

3D Si

Multiple Stills view
3D Images

3D Sensor

Shape

Textur

e

3D Image

3D Fa

Recogn

Algorit
Measure Progress on:

• Indoor cooperative face reco• Outdoor cooperative face rec• Comparison of still & 3D face 

recognition

• Effect of multiple images• Effect of High Dynamic Rang

on outdoor face recognition

• Comparison between human a

performance
Grand Challenge Architect

ComparisonAlgorithms/Systems

Modes

Accuracy of:

3D Sensors

3D from stills

Preprocessing/ReconstructionCompression

Meta data• eye coordinates• pose• gender

Image QualityMeasures
Grand Challenge Schedule

Date Activity

Aug-05 Face Recognition Grand Challenge 

21-Jun-05 FRGC Workshop at IEEE CVPR

Apr-05 Release Challenge Problem v2.X

Third FRGC Workshop

- Participants Present Results from v2.0 Feb-05

- Announce Challenge Problem v2.X- Introduce FRVT 2005

14-Jan-05 Results to Challenge Problem v227-Sep-04 Release Challenge Problem v2.0 

Second FRGC Workshop

10-Sep-04 - Participants Present Results from v1.0

- Explain Challenge Problem v2.0

5-May-04 Release Challenge Problem v1.0

First FRGC Workshop

mpleted5-May-04 - Explain challenge problem v1.0 in detail Co
Challenge Problem

• Experimental Data set

– Training set– Validation set

• Set of Experiments

– Target & Query sets

• Biometric Experimentation Enviro

– Infrastructure for Experiments

• Scoring Routines• Baseline Algorithms
Three Challenge Problems

• Ver1.0a

– Released 5 May 2004– 275 Subjects; 943 Subject ses

Recordings

• Ver2.0

– Released 27 September 2004– 466 Subjects; 4,007 Subject s

32,056 Recordings

• Ver2.X

– To be released April 2005
FRGC Challenge Problems

• FRGC consists of a series of 

progressively difficult challeproblems 

– Ver1.0a: small data set to intr

problem area

– Ver2.0: large data set designe

improving face recognition

– Ver2.X: Additional data and ex
FRGC Challenge Problems

• Ver2.X: 

– Additional data

• Samples from AY 2004-05 data c• Compression

– New Experiments– Covariate analysis– Normalization
How to Participate

• To participate in the FRGC:

– Send email request to: jonathon@nis– Once approved, obtain the two parts 

• Part 1 is the data

– Obtain data by contacting Pat Flynn at: 

signing the data license agreement

• Part 2 is the Biometrics Experimentatio

(BEE), which includes the 6 experiments

– Obtain BEE by contacting Todd Scruggs 

wendell.t.scruggs@saic.com and signing t

– Register on bulletin board for FRGC 

http://bbs.bee-biometrics.org
Getting the FRGC v2 Data

• Roughly 50 K files, 70 GB storage

– Submit ver1.0 results to Jonathon– Receive OK from Jonathon

• Get new license from FRGC ver2.0 topic Sign 

obtained from www.bee-biometrics.org

• Obtain a 120GB or larger external drive with 

(IEEE1394) or USB 2.0 interface

• Send form and disk to address on form• Receipt of disk will be acknowledged by e-ma• Disks will be shipped 1 to 2 weeks after their • Data will reside on a Linux ext2 filesystem on 

partition

• Disks will be return by UPS ground shipping c
Overview and Results 

FRGC ver1.0a
Goals of ver1.0a

• Introduce participants to FR• Provide sample of data• FRGC challenge problem• BEE

– Architecture– Baseline Algorithms
Ver1.0a Timeline

• 5 May ‘05: ver1.0a released • 10 Aug ’05: Results due for ve• 10 Sept ’05: Second FRGC W

– Summary of results for ver1.0
Example subject session

Cont

Uncon

3D 
FRGC Core Experiments

• Exp 1: Controlled indoor still versus indo• Exp 2: Indoor multi-still versus indoor m• Exp 3: 3D versus 3D• Exp 4: Controlled indoor still versus unc• Exp 5: 3D versus controlled single still• Exp 6: 3D versus uncontrolled single sti
Experimental Results Summ

15

4

5

13

Verification Rates @ 0.001 FAR Sub

1
Overview and Results of v
Outline

• Data and challenge problem• Generalized verification prot• Baseline Performance• Results from Ver2.0
Goals of ver2.0

• FRGC challenge problem 

– Test ability to run experiments 

large data set

– Challenge researchers to meet 

performance goal

• Increase FR performance levels b

magnitude
Ver 2.0 Timeline

• 27 September 2004: Release 

• 14 January 2005: Similarity 

results due

• 16 February 2005: Third FRG

Workshop

– Summary of Results
Training and Validation Partit

- Academic Year 2002-03 - Academic 

- 16,028 Co

- 8,014 Unc

- 4,007 3D 

- 12,776 Large Still Training 

Set

- 943 3D Subject Sessions

Training Va
Three Data Sets

FRGC Challenge Problem

Validation

Target

Query

Training
Demographics

(ver2.0 Validation Partition—Final)

43%

57%

MaleFemale

65%

18%

17%

122

68%

10%
Size of Faces

(ver2 On Validation)

Pixels between center of 

Mean Med

Controlled 261 26Uncontrolled 144 143D 160 16
Target / Query Sets

(ver2.0 Validation Partition—Final)

466 Subjects; 4,007 Subject sessions; 32,

Subject Sessions 2003-04

56

26

32

36

31 28 27

22 22 25

16 15 16 19 13 17 

0

102030405060

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 

Number of Subject Sessions

Validation Partition
Large Still Training Set

222 Subjects; 100 Subject sessions; 12,7

12 16 12

7

19 12

9

135

0

20406080

100120140160

9 10 11 12 13 14 15 16

Su

Subjects

Number of Subject Sessions
Generalized Verification 

Protocol• Motivation

– Increase number of match scor– Covariate analysis– Sampling of match and non-mat

distributions
Generating Match Scores

Fall Spri

1 Image per Person

5 

People

5 Images p

1 5 5 = 25 Match S
Generating Match Scores

Fall Sp

6 Image per Person 5 Images 

5 

People

6 5 5 = 150 Match 
FERET & FRVT Verification Pro

Query

Probe Set

Ta

r

ge

t

Gallery
Old vs New Method

Old

Gallery

Probe

New
Generalized Verification Pro

Query

Ta

r

ge

t
Generalized Verification Pro

• Only for verification—NO ide

Gallery

Probe Set
Similarity Matrix

Biobox / Algorithm / System

Query

T

a

r

g

e

t

Ta

Sim

Compute ROC
Three ROCs

ROC I - Within Semesters

ROC II – Within Year

ROC III – Between Semesters

∩ 
Similarity Score Normaliza

• Post processing similarity sco• Can improve verification perf
Classical Similarity Score 

Normalization

Probe Galler

Match

Similarity Scores (2.3, 4.0,..

Normaliz

Normalized Similarity Scores (-0.5, 1.1,...
Generalized Verification 

Protocol

Gallery

How do I norma
Normalization Set

Target Query No

Match Match

(3.9) (3

Normalization

1.1 Normalized S
Baseline Performance
FRGC Core Experiments

• Exp 1: Controlled indoor still versus indo• Exp 2: Indoor multi-still versus indoor m• Exp 3: 3D versus 3D• Exp 4: Controlled indoor still versus unc• Exp 5: 3D versus controlled single still• Exp 6: 3D versus uncontrolled single sti
Size of ver2.0 Experiment

Exp. Target set 

size

Query set size

1 16,028 16,028 

2 4,007 4,007 3 4,007 4,007 4

1

6,0

2

8

8,0

1

4

5 4,007 16,028 

6 4,007 8,014 
Baseline algorithm—PCA

200th2nd

1st

eigenvectors

Face

projection

Face s

• Whiten Cosine distance for classifier
Baseline Algorithms

Still versus StillExperiments 1 and 4

Still PCA 

Still

Similarity 

Score
Baseline Algorithm

Shape

PCA 

Texture

Similarity 

Score

Texture Fusion

+ 

Similarity 

Score Shape PCA 

3D versus 3DExperiments 3
Multi-still versus Multi-st

Experiment 2

Still Still Still Still

Still PCA PCA PCA PCA Still PCA PCA PCA PCA Still PCA PCA PCA PCA Still PCA PCA PCA PCA 

+

Fusi
Results of FRGC ver2.0 C

Problem
Participation

• Results received from ver1.0a 

deadline

– 10 Participants– 32 Experiments

• Results received from ver2.0 

deadline

– 17 Participants– 53 Experiments

Participants and Ex

0

204060

Ver1.0a 

FRGC Ver

Number
Ver2.0 Experiments

Grand Total Experiments 

Total Experiments Conducted

34

12 11

4

5

13

1 

0

10

20304050

1 2 3 3t 3s 4 5 

Experiments

Number Conducted 
Breakout for Exp. 3

Experiment 3 Breakout

11

4

5

05

1015

3 3t 3s

Experiments

d

r

e

e

t

b

c

m

u

Nu

Cond
Ver2.0 Baseline FAR = 0.1

88

54

0

20406080

100

Exp 2

Exp 3

Verification rate (%)
Experimental Results Summ

Exp 1 Exp 2 Exp 3 Exp 3t Exp 3s Exp 4 Exp 5 

0.0 0.2 0.4 0.6 0.8 1.0

Verification rate @ FAR = 0.001

34 12 11

1

13

5

4

Experiment
Results Summary

0.0 0.2 0.4 0.6 0.8 1.0

Verification rate @ FAR = 0.001

0.9892 1 0.9728 0.9622

0.8655

0.76

 Exp 1 Exp 2 Exp 3 Exp 3t Exp 3s Exp 4 

Experiment
Experiments 4 and 8

0.0 0.2 0.4 0.6 0.8 1.0

Verification rate @ FAR = 0.001

Algorithm
Effect of Expression on Ex

0.0 0.2 0.4 0.6 0.8 1

Verification rate @ FAR = 0.001

.0

ALLNeutral vs Neutral vs 

Algorithm

Exp 3 Performance by expression
Effect of Expression on Ex

0.0 0.2 0.4 0.6 0.8 1.0

Algorithm

Verification rate @ FAR = 0.001

ALLNeutral vs NeutralNeutral vs NonNeutral

Exp 3 Texture Performance by Expression

0.0 0.2 0.4 0.6 0.8 1.0 

Verification rate @ FAR = 0.001

Exp 3 Shap

Exp 3 Shape P
Summary of Expression

0.0 0.2 0.4 0.6 0.8 1.0

Verification rate @ FAR = 0.001

Shape TexturShape

 All Neutral vs Neutral Neutral v

Expression Covariate
Publication of FRGC Result

• Check with sponsors to de

if they want to be cited

•

Please include FRGC refer

– P. J. Phillips, P. J. Flynn, T. Scr

Bowyer, J. Chang, K. Hoffman, J. Min, W. Worek, Overview of 

Recognition Grand Challenge, I

Proceedings International Comand Pattern Recognition (CVPR) 
Database of FRGC Results

• Enroll in FRGC archive when y

is submitted or posted

– Optional: check with sponsor(s
Tentative Database Conten

• Figure and Graph Archive

– Data used to plot figures and g

• Points on a ROC and bar plot

– Plotting instructions
Tentative Contents (Cont.)

• Similarity Score Archive

– Similarity matrices– Signature sets– Mask matrices– Code for scoring– Limited access?

• New results sent to authors• Scoring code contributed to arch• Full documentation of new results
Summary

• Face Recognition Grand Chall

– Order of magnitude increase in 

performance

– Systematically investigate still – Formulate series of challenge p– Face Recognition Grand Challen

Completion August ‘05
Introduction Introduction
FRVT 2005 FRVT 2005

• Latest in a series of large scale 

evaluations for face recognition 

– Previous evaluations in the series we

FRVT2000, and FRVT 2002 

• Primary goal is to 

– Measure progress of prototype 

systems/algorithms and commercial recognition systems since FRVT 200

– Conduct comparison across modaliti– Compare performance with FRGC go
Software Development Kit (SDK) Test

• Sequestered data• Independent evaluation• Evaluation modes module• Starts in the August/September 

frame
Test Types

• FERET

– Proctored Test

• FRVT 2000 & 2002

– System brought to Governmen

• FRVT 2005

– SDK Test
SDK Test

• Deliver software SDK with c

API

• API based on evolving ISO • Consulting with Patrick Grot• Tentative Platforms

– Windows– Linux
SDK Components

• Read recordings (files)• Create samples• Preprocess samples• Write preprocessed samples• Create templates• Match templates• Similarity score normalizatio• Write similarity scores
Preprocessing Experiment

• Read recordings• Create samples• Preprocess samples• Write preprocessed samples
Preprocessing Example

Create 

Sample

Create 

Sample

Preprocess Preprocess

Write Write

Read Read

ROI Compresse

Image

Create 

Sample

Create 

Sample

Preprocess Preprocess

Write Write

Read Read
Recognition Experiment

• Read recordings• Create sample• Create templates• Match templates• Similarity score normalizatio• Write similarity scores
Recognition Experiment

Simila

Sco

Normali

Simila

Scor

Normaliz

Create 

Sample

Create 

Sample

Write 

Similarity 

Scores

Write 

Similarity 

Scores

Read 

Recordings

Read 

Recordings
FRVT 2005

• SDK specifications

– Detailed specifications forthco

• Example implementation• SDK concerns

– Time to complete experiments– Especially for 3D algorithms
FRVT 2005

• There will be surpri• NOT a test on Notr

Dame data
Summary

• FRVT 2005

– Independent government e

face recognition systems

– Measure progress since FR– Conduct comparison across – Compare performance with 

Goals

– August/September 2005 t
Next Steps

• March/April 2005

– Planned release of v2.X–

• April 2005

– Planned release of SDK

• 10 April 2005

– Deadline for submission of 

IEEE Workshop papers on FRGC experiments

• 10 June 2005

– Final papers due for IEEE 

Workshop

• 21 June 20

– IEEE W

CA

• August 20

– Submiss

FRGC ve

– End of F

• August/Se

– Start of 
Reminder

• Permission to post today’s pre

on the bbs website

– We need your permission to po

presentations

– Please send your permission or 

version of your presentations vCathy Schott at cschott@scha