Iris Challenge Evaluation (ICE) 2006 Protocol 

30 March 2006 

Introduction 

The Iris Challenge Evaluation (ICE) 2006 is the first independent U.S. Government 

evaluation for iris recognition technology. The ICE 2006 is based on the Face 

Recognition Vendor Test (FRVT) 2006, an independent evaluation of face recognition 

algorithms on sequestered data. ICE 2006 will evaluate performance on 1-1 matching, 

with image quality as an option. Performance will be measured for single iris or both 

irises of an individual. 

The interface for the executables will be based on the Biometric Experimentation 

Environment (BEE). The BEE was used as part of the Face Recognition Grand 

Challenge (FRGC) and will be used in the Iris Challenge Evaluation (ICE) 2006. The 

interface specifications for ICE 2006 are provided in a separate document Executable 

Calling Signatures for ICE 2006. ICE 2006 is open to academic institutions, research 

institutions, and commercial companies. 

ICE 2006 Tasks 

The one task being evaluated in ICE 2006 is 1-1 Matching. The basic biometric 

verification task is comparing two biometric samples and reporting a similarity score. 

This task will measure performance when matching is based only on the two biometric 

samples being compared. Each similarity score between a target biometric sample t and 

query biometric sample q is independent of all other biometric samples in the target and 

query sets. The computation of image quality is an option for 1-1 matching. A single 

quality score is produced for each target and query image. The quality score for a 

biometric sample is an integer, with larger numbers indicating better quality. 

In ICE 2006, a biometric sample consists of either a single iris image or two iris images. 

When the biometric sample consists of a single iris image, it can be an image of either a 

right iris or left iris. When a biometric sample consists of two iris images, one image is a 

right iris and one image is a left iris of an individual. The biometric samples in target and 

query sets will consist of either all right iris images, all left iris images, or all two iris 

biometric samples. 

General Rules 

To maintain fairness, the Government will adhere to the following procedures for ICE 

2006. Information provided to any single Participant will also be provided to all 

Participants. The primary and preferred source of communications between Participants 

and ICE 2006 government personnel is the frequently asked question (FAQ) page 

(http://iris.nist.gov/ICE). The ICE Liaison will be the primary point of contact and will 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

manage communications with Participants. All e-mails sent to the ICE Liaison and 

germane to the ICE 2006 will be posted on the ICE FAQ page. Contact with the ICE 

Liaison other than via the FAQ areas, or contact with someone other than the non-ICE 

Liaison will only be allowed under extenuating circumstances (for example, major issues 

with similarity files). 

1. Evaluation Result Output. Complete results for all experiments must be 

outputted in the correct format for an executable. If complete results in the correct 

format are not provided, then results for an executable may not be released. For 1-1 

this means complete similarity matrices with floating point numbers for all entries. 

For image quality, this means an image quality for all biometric samples in a 

signature set. 

2. Format. The image format for ICE 2006 is tiff. 

3. ICE 2006 participation agreement. The ICE 2006 participation agreement must 

be signed by all participants at the time of executable submission. 

Combinations of Executables that can be Submitted 

To participate in the two-iris track, ICE 2006 participants must submit a single iris 

executable. The single iris executable must be able to process left and right irises. 

Rules for Executables 

This section governs the rules for the executables that are submitted to ICE 2006. 

• Each ICE 2006 participant will be given a naming convention for their 

executables. See the ICE 2006 Application for instructions on receiving the 

naming convention. 

• A separate executable will need to be submitted for single and two iris tasks. 

• Participants can submit multiple executables for a task. 

• Executables need to be able to be installed multiple times during ICE 2006. We 

will erase disks and reinstall the operating system and executables to insure the 

evaluation protocols are being followed. 

• Executables will need to be able to run on a stand alone machine. 

• There will be no internet access during the ICE 2006 evaluations. Executables 

will need to be installed and executed without access to the internet. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

• Executables will need to able to run in either Windows 2003 or Linux. The 

targeted versions of the operating systems are Windows 2003 service pack 1 and 

Fedora 3. 

Independence requirements 

Parts of the ICE 2006 will be run as a batch process, i.e., executables will be given a list 

of biometric samples and asked to process them. However, batch processing in ICE 2006 

is to expedite the evaluation of tasks that may not be inherently a batch process for 

potential applications of interest. To make sure the ICE 2006 results generalize to these 

applications, a set of independence rules have been established. As part of ICE 2006, 

executables will be explicitly tested to make sure that the independence rules are 

obeyed. Failure to follow the independence rules will most likely result in a 

participant being disqualified from ICE 2006. Listed below are the independence 

rules: 

• 1-1 Matching: The computation of a similarity score s(t,q) is independent of 

other biometric samples in the both target and query sets. A simple test for 

independence, a similarity score s(t,q) from a batch process must be yield the 

same results as when the target set only consist of the t, and the query set only 

consists of q. 

• Optional Image Quality: The image quality rating IQ(t) for target sample t, must 

be independent of all other biometric samples in the target signature set. A simple 

test for independence, IQ(t) from a batch process must be the same as when the 

target set consists of a single target sample t. 

Training 

ICE 2006 does not dictate or require specific training or tuning sets. ICE 2006 

participants are free to train and tune their algorithm/system on the data set(s) of their 

choice. Many ICE 2006 participants are likely to be ICE 2005 participants. ICE 2006 

participants are free to train on all the data provided as part of ICE 2005. All algorithms 

and systems must be submitted completely trained and tuned. 

Experiment Information Provided 

The experimental protocol will provide information about each experiment. The 

information will reveal whether the experiment is single or double iris, and will be 

provided in the sigset files that accompanies each experiment. 
 

 

 

 

 

 

 

 

 

 

 

Notation: 

The definition of target, query, gallery, and probe sets is based on Face Recognition 

Vendor Test 2002 Performance Metrics, by P.J. Grother, R.J. Micheals and P. J. Phillips. 

Proceedings for the 4th International Conference on Audio Visual Based Person 

Authentication, 2003 (http://www.frvt.org/DLs/Avbpa_2003_evaluation_metrics.pdf). 

Below is a list of notations used in the document: 

• t target biometric sample 

• q query biometric sample 

• T target set 

• Q query set 

• G gallery 

• s(t,q) similarity score between target biometric t and query biometric sample q. 

• s(G,q) similarity scores between Gallery G and query biometric sample q. 

• s(G,Q) the complete set of similarity scores between Gallery G and query set Q. 

• IQ(t) image quality measure for biometric sample t.