Multiple Biometric Grand Chal

05 D

Second Workshop
MBGC Sponsors 

Executing Agency

Sponsoring Agencies

Director of National IntelligenceIntelligence Advanced Research Projects Activity

– Science & Technology Directorate

– Criminal Justice Informa– Operational Technology 
The MBGC Team

• NIST 

– Test Director, Dr. P. Jonathon Phillips– Hassan Sahibzada

• Colorado State University

– Prof. Ross Beveridge, Prof. Geof Givens, David Bolme, 

• SAIC 

– Dr. Todd Scruggs

• Schafer Corporation 

– J S ll ay 

Scallan

• University of Notre Dame

– Prof. Kevin Bowyer & Prof. Patrick Flynn

• U i it f T t D ll U

niversity o

f

Texas a

t

D

allas

– Prof. Alice O’Toole
Overview

• MBGC Overview MBGC Overview• Results MBGC version 1.0 Part 1

– Still Face– Video

• Talk & Lunch• R lt MBGC i 1 0 P t 2 Results MBGC version 1.0 Part 2

– Portal

• MBGC version 2.0 MBGC version 2.0• Multiple Biometric Evaluation 2009
Challenge Problems

• What are challenge problems?

– A series of experiments designed to advan

technology’s state-of-the-art

• Experiments designed• Experiments and test data distributed to resear• Researchers complete experiments and submit • S lid d d d Scores are consolidate

d an

d reporte

d

– Introduction of new technology
Ideal Challenge Problem

Today

rmance

C

h

alle

nge Proble

m Difficulty

Time

/ PerforC a e ge ob e Difficultyogress / Pr

Time / Effort / $
Technology Progress

2000

1996

2002

1993

• 17 Years• 7 E l ti 7

Evaluations

• 5 Challenge Problems 

(Technology Develo

pment

)

2009 

( gy p )• 3 Biometrics• 150,000+ Facial and Iris Images

2008

2006
Improved FR Performance

Face Recognition Error Rate
MBGC Challenge Problems

Portal Video

Video Still Face Still Face

I

Mugshot
MBGC Goal

• The main goal of the Multiple Biometric Grand Challen

(MBGC) is to

– Address face and iris recognition problems tha

relevant to those found in operational data

• Low to medium resolution face• Still and video iris• Near Infrared (NIR) & High Definition (HD) video f

portals

• Unconstrained recognition from still & video
MBGC Goal

• Programmatic method

– Sequence of challenge problems Sequence of challenge problems

• Modeled after the FRGC and ICE 2005• Challenge problems and data distributed to 

researchers

– Workshops– Multiple Biometric Evaluation 2009
Timeline

• Estimated task schedule for MBGC: Estimated task schedule for MBGC:

Schedule Task

December 2007

Formal annoucement of MBGC project Begin data collection at University of Notre Dame Design protocols, challenge problems and prepare 

April 2008 1st MBGC Workshop Release 1st challenge problem 

December 2008 2nd MBGC Workshop Results Challenge Problem Version 1

January 2009 Release Challenge Problem Version 2 

June 2009 3rd MBGC Workshop Results Challenge Problem Version 2
MBGC Version 1

• Goals

– Familiarize community with problem and d– Introduce participants to challenge protoco

Introduce participants to challenge protoco

experiment environment.

– Grow the research community that works o

Grow the research community that works o

problems.

1 – st Characterization of the state of the art Characterization of the state of the art.
MBGC Implementation

• MBGC Team

– Define challenge problems– Develop challenge problem protocols– Prepare data for distribution– S b itt d lt Score su

b

mitt

e

d results 

• Participants

– D l t h l d l ith Develop 

tec

hnology an

d algorithms

– Submit self reported similarity matrices for 

challenge problems. challenge problems.

– Submit quality scores. 
Quality Scores

• No quality scores submitted No quality scores submitted• The data reflects the natural variations 

encountered during data collection or iencountered during data collection or i

subsequent processing.

• Rep ese tat e o ope at o a data resentative of operational data.

• An appropriate mechanism for dealing 

data variation are q y ualit scores.
MBGC Participation to Date

• Organizations given access to MBGC data. 68• Organizations submitting results for MBGC version 1.0• Both industry and academic organizations participated

– Industry 7– Academic 7

• Countries

– China– France– Germany– Greece– United Kingdom– United States
MBGC Version 1 Results

• Challen

g p e 

problems released summer 2008.

• Results were submitted in November 2008.

– Portal 3 Novembe

r

– Still Face 10 November– Video 20 November

• Similarity matrices submitted.• Similarity matrices are self reported Similarity matrices are self reported.

• These matrices represent participants best ef
Challenge problem assessments

• Provides snapshot at one date Provides snapshot at one date

– MBGC v1: Beginning of November 2008

• Power from multiple results Power from multiple results• Reasonable assessment• FRGC predictive of FRVT 2006 FRGC predictive of FRVT 2006
Planned for Summer 2009