1 

2 

3 

4 

5 

6 

7 

8 

9 

10

11

12

13

14

15

16

17

IREX III 

ONE-TO-MANY IRIS IDENTIFICATION EVALUATION

CONCEPT, EVALUATION PLAN, AND API

VERSION 0.4

Patrick Grother

Image Group 

Information Access Division 

Information Technology Laboratory 

National Institute of Standards and Technology 

January 6, 2011

18

19

20

21
IREX III 

NIST Concept, Evaluation Plan and API Page 2 of 34

1

2 Status of this Document 

3

4 This document is the second public version. The entire content is open for comment. A revised version will be 

5 released as scheduled below - it is intended to be final. Important changes will be color-coded. 

6 Comments and questions should be submitted to irex@nist.gov.

7

8

9 Timeline of the IREX III Evaluation 

10

11 Table 1 - Dates and milestones 

August 4, 2011 IREX IV – Provisional commencement of NIST's IREX evaluation. This activity will include 1:1 

matching. It may extend to 1:N and other areas. The API in this document includes 1:1 

functions.

August 4, 2011 Decision on whether to conduct a further round of IREX III. Announce timeline for that.

July 1, 2011 NIST releases first public report. This report will attribute biometric error rate and 

processing speed estimates to the names of IREX III participants.

May 26, 2011 Window for 1:N participation closes. Anything received after this deadline will be ignored.

March 31, 2011 Hard deadline for a participants first submission of a 1:N algorithms. If a 1:N algorithm is 

not received by this date, the provider is excluded from all 1:N participation. This 

milestone, in the middle of the participation window, is intended to ensure that 

participants do not wait until the last minute to submit.

February 1, 2011 Window for 1:N participation opens. Shaded green below.

February 1, 2011 Deadline for participants to indicate intent to participate. Please send email to 

irex@nist.gov indicating a non-binding date when you expect to send an email.

January 21, 2011 NIST releases final API

January 19, 2011 Comments due on revised API

January 6, 2011 NIST releases revised API

January 4, 2011 Comments due on initial API

December 16, 2010 NIST releases initial API 

November 20, 2010 NIST announces IREX III

12

13

14

January 2011 February 2011 March 2011 April 2011 May 2011

Su Mo Tu We Th Fr Sa

 1

2 3 4 5 6 7 8

9 10 11 12 13 14 15

16 17 18 19 20 21 22

23 24 25 26 27 28 29

30 31

Su Mo Tu We Th Fr Sa

 01 2 3 4 5

6 7 8 9 10 11 12

13 14 15 16 17 18 19

20 21 22 23 24 25 26

27 28

Su Mo Tu We Th Fr Sa

 1 2 3 4 5

6 7 8 9 10 11 12

13 14 15 16 17 18 19

20 21 22 23 24 25 26

27 28 29 30 31 

Su Mo Tu We Th Fr Sa

 1 2

3 4 5 6 7 8 9

10 11 12 13 14 15 16

17 18 19 20 21 22 23

24 25 26 27 28 29 30

Su Mo Tu We Th Fr Sa

1 2 3 4 5 6 7

8 9 10 11 12 13 14

15 16 17 18 19 20 21

22 23 24 25 26 27 28

29 30 31

June 2011 July 2011 August 2011 September 2011 October 2011

Su Mo Tu We Th Fr Sa

 1 2 3 4

5 6 7 8 9 10 11

12 13 14 15 16 17 18

19 20 21 22 23 24 25

26 27 28 29 30

Su Mo Tu We Th Fr Sa

 1 2

3 4 5 6 7 8 9

10 11 12 13 14 15 16

17 18 19 20 21 22 23

24 25 26 27 28 29 30

31

Su Mo Tu We Th Fr Sa

 1 2 3 4 5 6

7 8 9 10 11 12 13

14 15 16 17 18 19 20

21 22 23 24 25 26 27

28 29 30 31

Su Mo Tu We Th Fr Sa

 1 2 3

4 5 6 7 8 9 10

11 12 13 14 15 16 17

18 19 20 21 22 23 24

25 26 27 28 29 30 

Su Mo Tu We Th Fr Sa

 1

2 3 4 5 6 7 8

9 10 11 12 13 14 15

16 17 18 19 20 21 22

23 24 25 26 27 28 29

30 31

15 

16

17
IREX III 

NIST Concept, Evaluation Plan and API Page 3 of 34

1 Table of Contents

2 1.1. Scope .............................................................................................................................................................. 6

3 1.2. Audience ......................................................................................................................................................... 6

4 1.3. Purpose and market drivers ............................................................................................................................ 6

5 1.4. Offline testing ................................................................................................................................................. 6

6 1.5. Phased testing .................................................................................................................................................7

7 1.6. Application scenarios .......................................................................................................................................7

8 1.7. Options for participation ..................................................................................................................................7

9 1.8. Interim reports ................................................................................................................................................ 8

10 1.9. Final reports .................................................................................................................................................... 8

11 1.10. Notes on images ............................................................................................................................................. 8

12 1.11. Use of multiple images per person ................................................................................................................. 9

13 1.12. Identification ................................................................................................................................................... 9

14 1.13. Quality based exclusion..................................................................................................................................10

15 1.14. Reporting of failure to enroll, acquire, process ...............................................................................................11

16 1.15. Matching of empty, broken and missing templates ........................................................................................11

17 1.16. Reporting of template size..............................................................................................................................11

18 1.17. Reporting of runtime memory usage ..............................................................................................................11

19 1.18. Reporting of computational efficiency ...........................................................................................................11

20 1.19. Exploring the accuracy-speed trade-space ..................................................................................................... 12

21 1.20. Hardware specification .................................................................................................................................. 12

22 1.21. Operating system and compilation environment ........................................................................................... 12

23 1.22. Threaded computations ................................................................................................................................. 13

24 1.23. Time limits ...................................................................................................................................................... 13

25 1.24. Ground truth integrity .................................................................................................................................... 13

26 2. Data structures and constants supporting the API ................................................................................................. 14

27 2.1. Overview ............................................................................................................Error! Bookmark not defined.

28 2.2. Identification of cameras ...............................................................................................................................14

29 2.3. Iris image sets ................................................................................................................................................ 15

30 2.4. Datatype for ancillary data from a template generation ................................................................................ 15

31 2.5. Data type for distance scores .........................................................................................................................16

32 2.6. File structures for enrolled template collection .............................................................................................16

33 2.7. Data structure for result of an identification search ......................................................................................16

34 3. API Specification ..................................................................................................................................................... 17

35 3.1. Implementation identifiers ............................................................................................................................ 17

36 3.2. Maximum template size ................................................................................................................................. 17

37 3.3. Quality support ..............................................................................................................................................18

38 3.4. API for 1:1 Verification.....................................................................................................................................18

39 3.5. 1:N Identification ............................................................................................................................................18

40 3.6. Software and Documentation ........................................................................................................................24

41 3.7. Runtime behavior ...........................................................................................................................................26

42 4. References ..............................................................................................................................................................26

43 Annex A Submission of Implementations to IREX III ......................................................................................................28

44 A.1 Submission of implementations to NIST ........................................................................................................28

45 A.2 How to participate .........................................................................................................................................28

46 A.3 Implementation validation .............................................................................................................................28

47

48 List of Figures 

49 Figure 1 – Current extent of the IREX Program ................................................................................................................ 6

50

51 List of Tables 

52 Table 1 - Dates and milestones .......................................................................................................................................... 2
IREX III 

NIST Concept, Evaluation Plan and API Page 4 of 34

1 Table 2 – Abbreviations .....................................................................................................................................................5

2 Table 3 – Subtests supported under the IREX III still-iris activity .......................................................................................7

3 Table 4 – Definition of True Positive Identification Rate .................................................................................................. 9

4 Table 5 – Definition of False Positive Identification Rate ................................................................................................. 9

5 Table 6 – Definition of Reliability ....................................................................................................................................10

6 Table 7 – Definition of Selectivity ....................................................................................................................................10

7 Table 8 – Definitions of Type I error rates .......................................................................................................................10

8 Table 9 – Definitions of Type II error rates ......................................................................................................................10

9 Table 10 – Identification Performance characteristics ....................................................................................................10

10 Table 11 – Number of threads allowed for each application ............................................................................................ 13

11 Table 12 – Processing time limits in milliseconds ............................................................................................................. 13

12 Table 13 – Kind flags indicating standardized properties ................................................................................................14

13 Table 7 – Sensor identifiers .............................................................................................................................................14

14 Table 14 – Structure for a single iris, with metadata ....................................................................................................... 15

15 Table 15 – Structure for a set of images from a single person ......................................................................................... 15

16 Table 16 – Data structure for ancillary data from a template generation function ......................................................... 15

17 Table 17 – Structure for a set of images from a single person .........................................................................................16

18 Table 18 – Enrollment dataset template manifest...........................................................................................................16

19 Table 19 – Structure for a single candidate .....................................................................................................................16

20 Table 20 – Implementation identifiers ............................................................................................................................ 17

21 Table 21 – Implementation template size requirements ................................................................................................. 17

22 Table 22 – Implementation template size requirements ................................................................................................. 18

23 Table 23 – Procedural overview of the identification test ...............................................................................................18

24 Table 24 – Enrollment initialization ................................................................................................................................. 19

25 Table 25 – Enrollment feature extraction........................................................................................................................20

26 Table 26 – Enrollment finalization ................................................................................................................................... 21

27 Table 27 – Identification feature extraction initialization ................................................................................................22

28 Table 28 – Identification feature extraction ....................................................................................................................22

29 Table 29 – Identification initialization .............................................................................................................................23

30 Table 30 – Identification search ......................................................................................................................................23

31 Table 31 – Implementation library filename convention ................................................................................................. 24

32

33
IREX III 

NIST Concept, Evaluation Plan and API Page 5 of 34

1

2 Terms and definitions 

3 The abbreviations and acronyms of Table 2 are used in many parts of this document. 

4 Table 2 – Abbreviations 

API Application Programming Interface

PACS Physical access control system

UID Unique Identity (program in India, aka. Aadhaar).

FNIR False negative identification rate

FPIR False positive identification rate

FMR False match rate

FNMR False non-match rate

Reliability Measure of how many searches for which an enrolled mate exists are successful.

Selectivity Measure of how many non-matches are returned in a search when in fact no mate is enrolled.

DET Detection error tradeoff characteristic: For identification this is a plot of Reliability vs. Selectivity.

INCITS InterNational Committee on Information Technology Standards

ISO/IEC 19794 ISO/IEC 19794-6: Information technology — Biometric data interchange formats — Part 6: Iris image 

data.

First edition: 2005-06-15. (See Bibliography entry).

Second edition: expected mid 2011, replacing 2005.

I379 INCITS 379:2004 - U.S. precursor to the 19794-5:2005 international standard. Now defunct.

ANSI/NIST Type 17 The most common container for iris images in the law enforcement world. 

IREX NIST's IRIS EXCHANGE program supporting standards-based iris biometrics

NIST National Institute of Standards and Technology

PIV Personal Identity Verification

SC 37 Subcommittee 37 of Joint Technical Committee 1 – developer of biometric standards

SDK The term Software Development Kit refers to any library software submitted to NIST. This is used 

synonymously with the terms "implementation" and "implementation under test".

5

6

7
IREX III 

NIST Concept, Evaluation Plan and API Page 6 of 34

1 IREX III 

2 1.1. Scope 

3 This document establishes a concept of operations and an application programming interface (API) for evaluation of 

4 iris identification implementations submitted to NIST's IREX III evaluation. This document covers only the recognition 

5 of two-dimensional still infrared images. See http://iris.nist.gov/irex for all IREX documentation. 

6

IREX 

IREX I 

Support for ISO/IEC 

19794-6:2011 esp. 

compact formats and 

compression thereof 

IREX III 

Assessment of 1+2 eye 

iris identification 

performance, (1:N, 1 ≤ 

N  107

). Support for 

ANSI/NIST Type 17 

standard. 

IREX II 

IQCE - Iris Quality 

Calibration and 

Evaluation - Support 

for ISO/IEC 29794-6 Iris 

Image Quality, 

expected c. 2012 

IREX IV (Planned) 

Assessment of 1+2 eye 

iris verification 

performance. Support 

for ISO/IEC 19794-

6:2011 and PIV. 

7 Figure 1 – Current extent of the IREX Program 

8 1.2. Audience 

9 Universities and commercial entities with an ability to implement a large scale one-to-many iris identification 

10 algorithm are invited to participate in the IREX III still-iris test. Organizations with only a one-to-one interest or 

11 capability should wait for the IREX IV activity. 

12 Organizations will need to implement the API defined in this document. Participation is open worldwide. There is no 

13 charge for participation. While NIST intends to evaluate technologies that could be readily made operational, the test 

14 is also open to experimental, prototype and other technologies. 

15 1.3. Purpose and market drivers 

16 This test is intended to support a plural marketplace of iris recognition systems. More specifically, the test is intended 

17 to assess 1:N identification performance in as large a population as possible, thereby testing the long-posited promise 

18 of iris as a very powerful biometric. 

19 While the largest applications, in terms of revenue, have been for one-to-many search for border control and war 

20 zone identity management, the use of iris for planetary-scale de-duplication (likely in combination with fingerprints) in 

21 the India's Aadhaar program is occurring now. 

22 The test is planned against the backdrop of an expanding marketplace of iris cameras designed to operate in a variety 

23 of applications beyond just 1:N de-duplication of an enrollment database. For example: 

24 ― some standoff-capture cameras can rapidly image and verify (in a one-to-many mode) high volumes of people; 

25 ― some mobile cameras can be preloaded with templates and firmware-based segmentation and identification 

26 capability for rapid 1:N watchlist. 

27 These applications are differentiated by the population size and other variables. 

28 1.4. Offline testing 

29 While this set of tests is intended as much as possible to mimic operational reality, this remains an offline test 

30 executed on databases of images. The intent is to assess the core capability of iris recognition algorithms. This test 
IREX III 

NIST Concept, Evaluation Plan and API Page 7 of 34

1 will be conducted purely offline - it does not include a live human-presents-to-camera component. Offline testing is 

2 attractive because it allows uniform, fair, repeatable, and efficient evaluation of the underlying technologies. Testing 

3 of implementations under a fixed API allows for a detailed set of performance related parameters to be measured. 

4 Human-in-the-loop testing is necessary to evaluate the overall system performance in an operationally realistic 

5 application. 

6 1.5. Phased testing 

7 To support research and development efforts, this testing activity will embed multiple rounds of testing. These test 

8 rounds are intended to support improved performance. Once the test commences, NIST will test implementations on 

9 a first-come-first-served basis and will return results to providers as expeditiously as possible. Providers may submit 

10 revised implementations to NIST only after NIST provides results for the prior implementation. The frequency with 

11 which a provider may submit implementations to NIST will depend on the times needed for vendor preparation, 

12 transmission to NIST, validation, execution and scoring at NIST, and vendor review and decision processes. 

13 For the number of implementations that may be submitted to NIST see section 1.7. 

14 1.6. Application scenarios 

The test will evaluate one-to-many identification implementations115 . As described in Table 3, the test is intended to 

16 represent close-to-operational use of iris recognition technologies in identification applications in which the enrolled 

17 dataset could contain images from up to ten million persons. 

18 Table 3 – Subtests supported under the IREX III still-iris activity 

# A B C

1. Aspect 1:N identification 1:N identification Reverse 1:N identification

2. Fast Slow Very slow.

Maximum effort to localize and 

match difficult irises.

3. Enrollment dataset Enrollment images are intended to be of reasonable to good 

quality, or at least of operationally representative quality.

Enrollment images are

challenging, possibly acquired 

under adverse circumstances and

with defects.

4. Search dataset Similar to the enrollment set, as in a de-duplication task. Perhaps of enrollment quality

usually without adverse effects.

5. Example application Open-set identification of an image against a central database,

e.g. de-duplication, or a search of a mugshot against a database

of known criminals.

Forensic identification, a nonideal iris image collected without

normal controls, cooperation

and illumination.

6. Intended number of

subjects, N

Up to O(10

7

) but dependence on

N will be computed from O(10

2

) 

upwards.

Up to O(10

7

) but dependence

on N will be computed from

O(10

2

) upwards.

O(10

4

) . From O(10

2

) upwards. 

SDK shall not implement explicit

limits on N.

7. Identification metrics Threshold based. Threshold and rank based.

8. Prior NIST test

references

See IREX at http://iris.nist.gov/irex

See MBE face recognition for metrics, analyses at http://face.nist.gov/mbe

9. Minimum number of

SDKs required

1 1 0

10. Deadline for submission

of first instance

March 31, 2011 May 28, 2011 April 29, 2011

19 1.7. Options for participation 

20 The following rules apply: 

 

1

 NIST has previously only modeled identification scenarios. The simplest simulation mimics a 1:N search by conducting N 1:1 

comparisons. 
IREX III

NIST Concept, Evaluation Plan and API Page 8 of 34

1 ― A participant must properly follow, complete and submit the AnBnex A Participation Agreement. This must be 

2 done once. It is not necessary to do this for each submitted implementation.

3 ― All participants shall submit at least one class A implementations labeled fast.

4 ― All participants shall submit at least one class B implementations labeled slow.

5 ― Submission of class C implementation is optional.

6 ― Any implementation shall implement exactly one of the functionalities defined in clause 3.

7 ― At any point in time, the maximum number of implementations undergoing testing at NIST will be two. This is the

8 total of classes A + B + C. NIST will invite submission of revised implementations when testing results for each 

9 prior implementation have been released.

10 ― A provider of an implementation may ask NIST not to repeat feature extraction and enrollment processes. This 

11 may expedite testing of an implementation because NIST can proceed directly to identification trials. NIST 

12 cannot conduct surveys over runtime parameters - NIST must limit the extent to which participants are able to 

13 train on the test data.

14 1.8. Interim reports

15 The performance of each implementation will be reported in a "score-card". This will be provided to the participant. 

16 While the score cards may be used by the provider for arbitrary purposes, they are intended to promote

17 development. The score cards will 

18  be machine generated (i.e. scripted),

19  be provided to participants with identification of their implementation,

20  include timing, accuracy and other performance results,

21  include results from other participants implementations, but will not identify the other providers,

22  be expanded and modified as revised implementations are tested, and as analyses are implemented,

23  be generated and released asynchronously with implementation submissions,

24  be produced independently of the status of other providers’ implementations,

25  be regenerated on-the-fly, primarily whenever any implementation completes testing, or when new analysis is 

26 added.

27 NIST does not intend to release these test reports publicly. NIST may release such information to the U.S. 

28 Government test sponsors. While these reports are not intended to be made public, NIST can only request that 

29 sponsoring agencies not release this content.

30 1.9. Final reports

31 Once NIST terminates the testing rounds, one or more final public reports will be released. NIST may publish

32  Reports (typically as numbered NIST Interagency Reports),

33  Publications in the academic literature,

34  Presentations (typically PowerPoint).

35 Our intention is that the final test reports will publish results for each and every implementation submitted by each 

36 participant. The intention is to report results for the most capable implementations (see section 1.12, on metrics). 

37 Other results may be included (e.g. in appendices) to show, for example, examples of progress or tradeoffs. 

38 IMPORTANT: Results will be attributed to the providers.

39 1.10. Notes on images

40 ― Images are likely to have dimensions of 640x480 pixels.

41 ― Images will all be collected in the near infra-red.
IREX III

NIST Concept, Evaluation Plan and API Page 9 of 34

1 ― Images from more than one sensor will be included.

2 ― Some persons will have images from more than one sensor.

3 ― Some images are of poor quality. NIST will target the natural population. NIST will secondarily attempt to control 

4 for non-ideal variations.

5 ― Some images were collected outdoors. Pupil radius may be small.

6 EDITOR'S NOTE :: More information in second edition of this document.

7 1.11. Use of multiple images per person

8 Some tests will proceed with

9 ― K = 1 image per person. This could be labeled as a left eye (L), a right eye (R), or unknown (U).

10 ― K = 2 images per person. These might be labeled as any combination of L, R, and U.

11 ― K = a random number of images per person. These might be labeled as any combination of L, R and U.

12 1.12. Identification

13 For identification testing, the test will target open-universe applications such as de-duplication and watch-lists. 

14 Open-set applications require estimation of two error rates: Type I errors are those in which a person's biometric data 

15 is incorrectly not associated with its enrolled mate; Type II errors are those in which a person's biometric data is 

16 associated with other enrollees' data. Table 8 defines metrics for Type I identification errors used in this report, and 

17 notes various synonyms and complementary terms. 

18 Table 9 defines metrics for Type II errors. Plots of the two error rates, parametric on threshold, will be the primary 

19 reporting mechanism.

20 While some one-to-many applications operate on the assumption that a candidate list of identities will be reviewed by 

21 a human examiner, for which rank-based metrics are relevant, this test will primarily target lights-out identification i.e. 

22 the iris identification system operates on its own, making decisions against some threshold. However, the analysis 

23 might be extended to also include a rank criterion.

24 The test will not address the closed-set task because it is operationally uncommon. In a closed-set application, all 

25 searches have an enrolled mate. Operationally closed-universe applications are rare. One example is a cruise ship in 

26 which all passengers are enrolled and all searches should produce one, and only one, identity. Another example is 

27 forensic identification of dental records from an aircraft crash. Most practical applications of biometric identification 

28 are open-set problems. 

29 In summary, IREX III will test only open-set identification tasks. This means that some fraction of searches will have 

30 no enrolled mate. This is operationally typical: some subjects have not been issued a visa or drivers license before; 

31 some law enforcement searches are from first-time offenders. Operationally searches for these people should return 

32 zero identities.

33 Table 4 – Definition of True Positive Identification Rate

TPIR (R,T,L) =

Num. searches with enrolled mate reported as candidate with distance ≤ threshold, T, and rank ≤ 

R on a candidate list of length L Equation 3

Num. searches with enrolled mate

34

35 Table 5 – Definition of False Positive Identification Rate

FPIR (T,L) =

Num. searches without enrolled mate yielding one or more candidates with distance ≤ threshold, 

T when candidate list is of length L Equation 4

Num. searches without enrolled mate

36
IREX III

NIST Concept, Evaluation Plan and API Page 10 of 34

1 Table 6 – Definition of Reliability

REL (T,L) = TPIR(N,T, L) where N is the size of the enrolled population Equation 5

2

3 Table 7 – Definition of Selectivity

SEL (T,L) =

Num. candidates with score ≤ threshold, T produced in searches without enrolled mate, when 

candidate list is of length L Equation 6

Num. searches without enrolled mate

4

5 Table 8 – Definitions of Type I error rates

Metric Measured over Definition Related terms

True Positive 

Identification 

Rate (TPIR)

Searches for 

which a mate is 

present in the 

enrolled dataset.

Table 4. Fraction of identification searches for which the enrolled mate is 

present on the candidate list with rank less than or equal to R, and distance

less than or equal to threshold, T.

Special cases:

1. The rank requirement can be set to be difficult, i.e. R = 1, or absent 

(i.e. R = N, where N is the number of enrolled identities) or any 

value in between.

2. The threshold requirement can be difficult (i.e. high value of T), or 

absent (i.e. T = 0), or any value in between.

Hit Rate and 

Reliability of 

synonyms

FNIR and miss 

rate are 

synonyms for the 

complement 1 –

FNIR

FNIR See TPIR FNIR = 1 – TPIR(R, T, L) FNIR

Miss Rate See TPIR FNIR(R, T, L) FNIR

Hit Rate See TPIR TPIR(R, T, L) FNIR

6

7 Table 9 – Definitions of Type II error rates

Metric Measured over Definition Related terms

False Positive 

Identification Rate 

(FPIR)

Searches for which a mate 

is not present in the 

enrolled dataset.

Table 5. Fraction of identification searches for which any (i.e. 

one or more) enrolled identities on a candidate list of length L 

are returned with distance less than or equal to threshold T.

Selectivity

Selectivity See FPIR Table 7. The mean, over a set of searches, of the number of 

candidates returned for which the distance is less than or 

equal to a threshold, T.

False positive 

identification 

rate

8

9 From these metrics the primary performance characteristics are defined in Table 10.

10 Table 10 – Identification Performance characteristics

Metric Measured over Definition

CMC Searches with mates The cumulative match characteristic is a plot of 1 - FNIR(R, 0, L) vs. R, with 1 ≤ R ≤ L

ROC Searches with and without mates The receiver operating characteristic is a plot of REL(T,L) vs. SEL(T,L)

11

12 1.13. Quality based exclusion

13 NIST will examine the effectiveness of iris image quality scores. These are computed from input images during 

14 feature extraction. The planned analyses relate to accuracy prediction:

15 ― The default method will be the error vs. reject analysis document in P. Grother and E. Tabassi, Performance of 

16 biometric quality measures, IEEE Trans. PAMI, 29:531–543, 2007.
IREX III

NIST Concept, Evaluation Plan and API Page 11 of 34

1 ― NIST will survey over an additional parameter, , the fraction of images excluded from a subsequent computation 

2 of the ROC characteristic. The images excluded will be those with the lowest scalar quality value reported by the 

3 implementation during template generation. Quality-based exclusion is valuable in multimodal applications, 

4 where an alternative biometric can be used when an iris is automatically judged to be poor.

5 ― We will include a ROC for  = 0, as a baseline.

6 The primary target application will be 1:N de-duplication of a large database.

7 Analyses other than for the default case may be conducted.

8 1.14. Reporting of failure to enroll, acquire, process

9 FTA and FTE have different meanings in offline tests such as IREX III versus online tests such as those conducted in 

10 access control scenario test with humans interacting with biometric readers.

11 In IREX III, soft failures, where the algorithm elects to not produce a template (e.g. on image quality grounds) shall be 

12 treated identically to hard failures, where the algorithm crashes or hangs. In IREX III, any failure to convert images 

13 into templates shall be counted as a "failure to enroll" (FTE) and reported as such.

14 1.15. Matching of empty, broken and missing templates

15 After a soft failure, the template generator may return an empty (0 byte) template or a short one (a few bytes). In all 

16 cases NIST will pass the template to the matching / searching functions which must handle such templates 

17 transparently.

18 After a hard failure, the absence of an output template generator will cause NIST will pass to an empty (0 bytes) 

19 template to the matching / searching functions which must handle such templates transparently. 

20 When 1 or more of the templates passed to a one-to-many search are empty or short, the matching / searching 

21 functions shall produce a value of -1, and return the appropriate non-zero error code. Such events will be included in 

22 the reported performance estimates, as follows:

23 ― In a negative identification system, where a person claims not to be enrolled (e.g. a border crossing watchlist 

24 system equipped designed to detect and reject previously deported travelers), the one-to-many search should 

25 flag an empty input template (from a traveler wearing patterned contact lenses, for example). NIST will simulate 

26 this outcome by setting the distance to a low (i.e. genuine) value to force a hit on the database. Such an 

27 occurrence would prompt secondary inspection at a border crossing. NOTE: If the image actually had an 

28 enrolled mate then this will benefit the accuracy estimate of the implementation under test. If the image did not 

29 have an enrolled mate then this will penalize the implementation.

30 ― In a positive access control application e.g. gymnasium access without any claim of identity, a correct one-to31 many search should result in a rejection of an empty input template – NIST will simulate this outcome by setting 

32 the distance to a high value for searches in which a non-zero error is reported.

33 1.16. Reporting of template size

34 Because template size is influential on storage requirements and computational efficiency, this API supports 

35 measurement of template size. NIST will report statistics on the actual sizes of templates produced by iris recognition 

36 implementations submitted to IREX III. Template sizes were reported in [IREX}.

37 1.17. Reporting of runtime memory usage

38 NIST will report the amount of memory used during one-to-many searches. That is NIST will not rely on the naïve first 

39 order estimate of this, i.e. N times the enrollment template size, plus the size of the search template).

40 1.18. Reporting of computational efficiency

41 As with other tests, NIST will compute and report recognition accuracy. In addition, NIST will also report timing 

42 statistics for all core functions of the submitted implementation implementations. This includes feature extraction, 

43 and 1:1 and 1:N recognition. For an example of how efficiency can be reported, see [IREX].
IREX III

NIST Concept, Evaluation Plan and API Page 12 of 34

NIST will plot 1:N search duration as a function of N. Some face identification implementations [MBE] scale as Nb1 with 

2 b < 1. It is not clear whether the indexing approaches proposed for iris recognition [HAO, UVW] offer such behavior.

3 EDITOR'S QUESTION TO PROVIDERS: Is there any need to support measurement of speed of K > 1 simultaneous, 

4 "batched" searches against an N template enrollment. There have been claims that the time taken to execute a 1:N 

5 search of 20 templates (e.g. from K = 20 persons) in a single function invocation is less than 20 times the duration of 

6 searching 1.

7 No decision yet. Two providers asked for it. Another indicated essentially that this should be an iris recognition test, 

8 not a nifty-assembly-language-coding demonstration. Emphasis should be primarily on natural efficiency of the core 

9 algorithms and secondarily on scaling properties (with N). Batch mode acceleration adds API complexity.

10 1.19. Exploring the accuracy-speed trade-space

11 The requirement to submit both class A “fast-less-accurate" and class B "slow-more-accurate” variants with perhaps a

factor of three between the speeds

2

12 is intended to demonstrate a capability to trade accuracy for speed. Speed will 

13 be reported alongside some discussion that iris recognition algorithms can run on back-office blade clusters, and on 

14 embedded devices such as hand held cameras. Participants are cautioned that the final report will note that 

15 algorithms that are slow on blades will be even slower on embedded devices.

16 IREX III will be conducted entirely on the blades described below, not on low power embedded platforms. Further,

17 NIST cannot require that class A and B submissions are actually "fast" and "slow" variants - participants can always

18 instead choose to submit variants on some other axis e.g. "A = mature" vs. "B = experimental". We'll test them 

19 regardless.

20 1.20. Hardware specification

21 NIST intends to support high performance by specifying the runtime hardware beforehand. NIST will execute the test 

22 on high-end PC-class computers. These machines have 4-cpus, each of which has 4 cores. These blades are labeled 

Dell M905 equipped with 4x Quad Core AMD Opteron 8376HE processors323 running at 2.3GHz. Each CPU has 512K 

24 cache. The bus runs at 667 Mhz. The main memory is 192 GB Memory as 24 8GB modules. Sixteen processes can be

25 run without time slicing.

26 NIST is requiring use of 64 bit implementations throughout. This will support large memory allocation to support 1:N 

27 identification task with image counts in the millions. If all templates were to be held in memory, the 192GB capacity

28 implies a limit of 20KB per template, for a 10 million image enrollment. Note that while the API allows read access of 

29 the disk during the 1:N search, the disk is, of course, relatively slow.

30 Some of the section 3 API calls allow the implementation to write persistent data to hard disk. The amount of data 

31 shall not exceed 200 kilobytes per enrolled image. 

32 NIST will respond to prospective participants' questions on the hardware, by amending this section.

33 1.21. Operating system and compilation environment

34 All submitted implementations shall run on CentOS 5.5 which runs Linux kernel 2.6.18-194. http://www.centos.org/ 

35 NIST will link the provided library file(s) to our ISO 98/99 C/C++ language test drivers. Participants are required to 

provide their library in a format that is linkable using gcc version 4.1.2436 . The standard libraries are: 

 

2

See the FNMR vs. time plots in Figure 18 of [IREX].

3

cat /proc/cpuinfo returns fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht 

syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3wext 3dnow constant_tsc nonstop_tsc pni cx16 popcnt lahf_lm cmp_legacy svm 

extapic cr8_legacy altmovcr8 abm sse4a misalignsse 3dnowprefetch osvw

4

The command "gcc –v" gives the following output

Using built-in specs. Target: x86_64-redhat-linux

Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enablechecking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,objc++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic --

host=x86_64-redhat-linux

Thread model: posix gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)
IREX III

NIST Concept, Evaluation Plan and API Page 13 of 34

1 /usr/lib64/libstdc++.so.6.0.8 lib/libc.so.6 -> libc-2.5.so /lib/libm.so.6 -> libm-2.5.so

2 A typical link line might be

3 gcc -I. -Wall -m64 -o i3test i3test.c -L. -lirex_Enron_A_07 -lpthread

4 1.22. Threaded computations

5 Table 11 shows the limits on the numbers of threads an iris recognition implementation may use. In many prior tests

6 threading has not been permitted (i.e. T=1) because NIST will parallelize the test by dividing the workload across many 

7 cores and many machines. For the functions where we allow multi-threading, e.g. in the 1:N test, NIST requires the 

8 provider to disclose the maximum number of threads to us. If that number is T, NIST will run the largest integer 

9 number of processes, P, in parallel such that TP  C, where C is the number of cores on the machine, in this case 16.

10 Table 11 – Number of threads allowed for each application

Function Number of threads

Feature extraction 1

Finalize enrollment (before 1:1 or 1:N) 1  T  C

Identification 1  T  C

11

12 NIST will not run an implementation from participant X and an implementation from participant Y on the same 

13 machine at the same time.

14 For single-threaded libraries, NIST will run up to 16 processes concurrently. NIST's calling applications are single15 threaded. In a prior face test [MBE] this issue led to some contention: In order to compare speeds, the duration of 

16 searches from single threaded algorithms was divided by C on the assumption that either the algorithm could be 

17 threaded, or that C searches would be in a transaction queue on a sustained basis.

18 For IREX III, the duration will again be multiplied by T/C.

19 1.23. Time limits

20 The elemental functions of the implementations shall execute under the time constraints of Table 12. These times 

21 limits apply to the function call invocations defined in section 3. Assuming the times are random variables, NIST 

22 cannot regulate the maximum value, so the time limits are 90-th percentiles. This means that 90% of all operations 

23 should take less than the identified duration.

24 The time limits apply per image. When K images of a person are present (e.g. one image of each eye), the time limits 

25 shall be increased by a factor K.

26 Table 12 – Processing time limits in milliseconds

Function Class A Class B Class C

Feature extraction for enrollment of a 640x480 pixel image 500 (1 core) 1500 (1 core) 10000 (1 core)

Feature extraction for identification of a 640x480 pixel image 500 (1 core) 1500 (1 core) 10000 (1 core)

Finalization of a 1 million template enrollment database 7,200,000 (16 cores)

Identification search No time limit, but speed will be reported.

27 1.24. Ground truth integrity

28 Some of the test databases will be derived from operational systems. They may contain ground truth errors in which

29 ― a single person is present under two different identifiers, or

30 ― two persons are present under one identifier, or

31 ― in which no iris is present in the image,

32 ― left and right eyes are mislabeled as right or left.
IREX III

NIST Concept, Evaluation Plan and API Page 14 of 34

1 If these errors are detected, they will be removed or repaired. NIST will use aberrant scores (high impostor scores, 

2 low genuine scores) to detect such errors. This process will be imperfect, and residual errors are likely. For 

3 comparative testing, identical datasets will be used and the presence of errors should give an additive increment to all 

4 error rates. For very accurate implementations this will dominate the error rate. NIST intends to attach appropriate 

5 caveats to the accuracy results. For prediction of operational performance, the presence of errors gives incorrect 

6 estimates of performance.

7 2. Data structures and constants supporting the API

8 2.1. Overview

9 This section describes data structures and constant values that are used the API of clause 3.

10 2.2. Image types

11 In this iris recognition test, an individual is represented by K  1 two-dimensional iris images each of which is 

12 accompanied by a left-right-unknown eye label and a camera identifier. The images used in this test may be 

13 conformant to the ISO/IEC 19794-6:2011 and ANSI/NIST ITL 1-2011 Standards. Both standards include the types of 

14 images specified in Table 13 – these differ primarily in their geometric specifications. The image type 7 is specialized in 

15 that it includes eyelid and sclera regions that are masked.

16 Table 13 – Kind flags indicating standardized properties

Image

Type 

(integer)

ISO/IEC 

19794-6:2011 

Presence

Meaning (R is iris radius) IREX presence

0 No Unconstrained – image has no geometric or other requirements associated with 

standardized image

IREX III

1 Yes Image has iris margin requirements greater than or equal to (0.2R, 0.6R) in y and 

x directions.

2 Yes As for k1 but images are 640 x 480 IREX III

3 Yes Images are centered and have strict margin requirements identical to (0.2R, 

0.6R).

IREX IV, NOT IREX III

7 Yes As for k3 but the eyelid and sclera are painted over with a fixed pixel value, prior 

to lossless or lossy compression.

IREX IV, NOT IREX III

17 2.3. Identification of cameras

18 When known, cameras will be identified using the 2 byte integer codes of Table XX. In many cases the camera ID may 

19 not be known and the code 0x0000 will be passed to the SDK.

20 Table 14 – Sensor identifiers

# Sensor Manufacturer and Model Identifier 

1 IrisID LG 2200 0x2A16

2 IrisID LG 3000 0x2A1E

3 IrisID LG 4000 0x2A26

4 IrisID TD100 0x2A40

5 Crossmatch SEEK 0x1800

6 Crossmatch I SCAN 2 0x1801

7 L1 / Securimetrics PIER 0x1A03

8 L1 / Mobile - Eyes 0x1A10

9 L1 / HIIDE 0x1A11

10 Cogent Fusion 0x1700

11 AOptix Insight 0x4700
IREX III

NIST Concept, Evaluation Plan and API Page 15 of 34

12 Hoyos Group HBOX 0x9800

13 HoyosGroup EyeSwipe 0x9801

14 Unknown or unspecified 0x0000

1 2.4. Iris image sets

2 IREX III allows enrollment of multiple iris images into a single template. The structure of Table 15 defines the container 

3 for a single iris image, and the structure of Table 16 then defines a linear array of these. For IREX III, the number of 

4 irises (numirises, Table 16) will be one or two.

5 Table 15 – Structure for a single iris, with metadata

"C" code fragment Remarks

1. typedef struct siris

2. {

3. uint8_t eye; Eye labels as in the ISO standard. SUBJECT_EYE_UNDEF = 0 (00Hex), 

SUBJECT_EYE_RIGHT = 1 (01Hex) and SUBJECT_EYE_LEFT = 2 (02Hex)

4. uint16_t image_width; Number of pixels horizontally

5. uint16_t image_height; Number of pixels vertically

6. uint8_t image_type; Image type integer code per Table 13

7. uint16_t camera; The camera per Table 14

 uint8_t *data; Pointer to WH pixels of raster scanned intensity data, 8 bits per pixel.

8. } ONEIRIS;

6 Table 16 – Structure for a set of images from a single person

"C" code fragment Remarks

1. typedef struct miris

2. {

3. uint32_t numirises; The number of accessible pointers, F, such that the last element is F-1

4. ONEIRIS **irises; Pointers to F pre-allocated iris images of the same person.

5. } MULTIIRIS;

7 2.5. Datatype for ancillary data from a template generation

8 When an input iris image is converted into a template, the implementation shall populate a data structure identical to 

9 that in Table 17. To support multiple-image multiple-eye usage, the linear array of Table 18 shall be used.

10 Table 17 – Data structure for ancillary data from a template generation function

"C" code fragment Remarks

1. typedef struct osiris

2. {

3. double iris_radius; Estimate of iris radius, in pixels

4. uint16_t iris_center_x; Estimate of the horizontal coordinate of the iris center

5. uint16_t iris_center_y; Estimate of the vertical coordinate of the iris center

6. double pupil_radius; Estimate of pupil radius, in pixels

7. uint16_t pupil_center_x; Estimate of the horizontal coordinate of the pupil center

8. uint16_t pupil_center_y; Estimate of the vertical coordinate of the pupil center

9. uint8_t quality; An assessment of image quality. The legal values are:

[0,100] - The value should have a monotonic decreasing relationship with false nonmatch rate anticipated for this sample if it was compared with a pristine 

image of the same person. So, a low value indicates high expected FNMR.

254 - This value indicates the value was not assigned.

255 - This value indicates a failed attempt to calculate a quality score.

 uint8_t failed; 0 means iris was successfully segmented and the other fields have been assigned.

1 means iris could not be segmented and the other fields should be ignored.

10. } ONESEGMENTATION;
IREX III

NIST Concept, Evaluation Plan and API Page 16 of 34

1

2 Table 18 – Structure for a set of images from a single person

"C" code fragment Remarks

1. typedef struct omiris

2. {

3. uint32_t numirises; The number of accessible pointers, F, such that the last element is F-1

4. ONESEGMENTATION **irises; Pointers to F pre-allocated iris images of the same person.

5. } MULTISEGMENTATION;

3

4 2.6. Data type for distance scores

5 Identification and verification functions shall return a measure of the distance between the irises data contained in 

6 the two templates. The datatype shall be an eight byte double precision real. The legal range is [0, DBL_MAX], where 

7 the DBL_MAX constant is larger than practically needed and defined in the <limits.h> include file. Smaller values 

8 indicate more likelihood that the two samples are from the same person.

9 Providers are cautioned that algorithms that natively produce few unique values (e.g. integers on [0,127]) will be 

10 strongly disadvantaged by the inability to set a threshold precisely, as might be required to attain a false match rate of 

11 exactly 0.0001, for example.

12 2.7. File structures for enrolled template collection

13 An implementation converts iris images into a template (using the "convert_multiiris_to_enrollment_template" 

14 function of section 3.5.3. To support the one-to-many identification NIST will concatenate enrollment templates into 

15 a single large file. This file is called the EDB (for enrollment database). The EDB is a simple binary concatenation of 

16 proprietary templates. There is no header. There are no delimiters. The EDB may extend to hundreds of gigabytes in 

17 length.

18 This file will be accompanied by a manifest; this is an ASCII text file documenting the contents of the EDB. The 

19 manifest has the format shown as an example in Table 19. If the EDB contains N templates, the manifest will contain 

20 N lines. The fields are space (ASCII decimal 32) delimited. There are three fields, all containing numeric integers. 

21 Strictly speaking, the third column is redundant.

22 Table 19 – Enrollment dataset template manifest

Field name Template ID Template 

Length

Position of first 

byte in EDB

Datatype required Unsigned decimal integer, not necessarily consecutive, nor 

starting at 0, nor in any particular order.

Unsigned 

decimal integer

Unsigned 

decimal integer

Datatype length required 4 bytes 4 bytes 8 bytes

Example lines of a 

manifest file appear to 

the right. Lines 1, 2, 3 and 

N appear.

90201744 1024 0

16323202 1536 1024

7456433 512 2560

...

183838 1024 307200000

23

24 2.8. Data structure for result of an identification search

25 All identification searches shall return a candidate list of a NIST-specified length. The list shall be sorted in ascending 

26 order of the distance score – i.e. the most similar matching entries are listed first with lowest rank. The data structure 

27 shall be that of Table 20.

28 Table 20 – Structure for a single candidate

"C" code fragment Remarks

1. typedef struct candidate
IREX III

NIST Concept, Evaluation Plan and API Page 17 of 34

2. {

3. uint8_t failed; If the candidate computation failed, this value is set on [1,255]. If the candidate is valid it 

should be set to 0.

4. uint32_t template_id; The Template ID integer from the enrollment database manifest defined in clause 2.7.

5. double distance_score; Required measure of distance between the identification template and the enrolled candidate. 

Lower scores mean more likelihood that the samples are of the same person.

An algorithm is free to assign any non-negative value to a candidate. The distribution of values 

will have an impact on the appearance of a plot of false-negative and false-positive 

identification rates.

6. double probability; Required estimate of the probability that the biometric data and candidate belong to different

persons, i.e. the probability that a score this small would be observed given that the pair of 

images are from different people = P(DISTANCE | IMPOSTOR). This value shall be on [0:1]. This 

is the integral of the expected impostor distribution from 0 to the distance_score, i.e. the 

expected one-to-one false match rate.

7. } CANDIDATE;

1

2 3. API Specification

3 3.1. Implementation identifiers

4 All implementations shall support the self-identification function of Table 21. This function supports NIST book5 keeping. The version numbers should be distinct between all submit implementations.

6 Table 21 – Implementation identifiers

Prototype int32_t get_pid(

char *sdk_identifier, A participant-assigned ID. This shall be different for each submitted implementation. 

char *email_address); Output

Description This function retrieves a point-of-contact email address from the implementation under test.

Output 

Parameters

sdk_identifier Version ID code as hexadecimal integer printed to null terminated ASCII string. NIST 

will allocate exactly 5 bytes for this. This will be used to identify the implementation in 

the results reports. This value should be changed every time any implementation is 

submitted to NIST. The value is vendor assigned - format is not regulated by NIST. 

EXAMPLE: "011A"

email_address Point of contact email address as null terminated ASCII string. NIST will allocate at 

least 64 bytes for this. The implementation shall not allocate.

Return Value 0 Success

Other Vendor-defined failure

7 3.2. Maximum template size

8 All implementations shall report the maximum expected template sizes. These values will be used by the NIST test 

9 harnesses to pre-allocate space for template data. The values should apply to a single image. For a MULTIIRIS

10 containing K images, NIST will allocate K times the value returned. The function call is given in Table 22.

11 Table 22 – Implementation template size requirements

Prototype int32_t get_max_template_sizes( uint32_t *max_enrollment_template_size, Output

 uint32_t *max_recognition_template_size) Output

Description This function retrieves the maximum template size needed by the feature extraction routines.

Output

Parameters

max_enrollment_template_size The maximum possible size, in bytes, of the memory needed to store feature 

data from a single enrollment image.

max_recognition_template_size The maximum possible size, in bytes, of the memory needed to store feature 

data from a single verification or identification image.

Return Value 0 Success
IREX III 

NIST Concept, Evaluation Plan and API Page 18 of 34

Other Vendor-defined failure

1 3.3. Quality support 

2 Section 1.13 conceives of each template being accompanied by an assessment of the image quality, with quality being 

3 an integer scalar summary of expected utility of the image in a subsequent search. The function call given in Table 22

4 indicates whether or not NIST should universally ignore those values. Note, because we expect to state in the final 

5 report that meaningful quality values are operationally valuable, a value of zero from this function call is undesirable. 

6 Table 23 – Implementation template size requirements 

Prototype int32_t is_quality_assessment_supported()

Description This function indicates whether the implementation is capable of computing meaningful quality values

Return Value 0 Quality assessment is not supported. All template generation 

function calls will return quality values = 254 in all ONESEGMENTATION

structures of clause 2.5.

Other Meaningful quantitative quality assessment is supported. Template 

generation function calls will typically return values in {0-100,255}.

7 3.4. API for 1:1 Verification 

8 EDITOR's NOTE – 1:1 verification has been dropped from IREX III to simplify the test, and expedite implementation. 

9 Verification is likely to be tested in a future IREX IV with an API derived from IREX I. 

10 3.5. 1:N Identification 

11 3.5.1. Scope 

12 The 1:N application proceeds in two phases, enrollment and identification. The identification phase includes separate 

13 pre-search feature extraction stage, and a search stage. 

14 The design reflects the following testing objectives for 1:N implementations. 

 support distributed enrollment on multiple machines, with multiple processes running in parallel 

 allow recovery after a fatal exception, and measure the number of occurrences 

 allow NIST to copy enrollment data onto many machines to support parallel testing 

 respect the black-box nature of biometric templates 

 extend complete freedom to the provider to use arbitrary algorithms 

 support measurement of duration of core function calls 

 support measurement of template size 

15 Table 24 – Procedural overview of the identification test Phase

# Name Description Performance Metrics to be reported 

by NIST

Enrollment

E1 Initialization Give the implementation advance notice of the number of 

individuals and images that will be enrolled.

Give the implementation the name of a directory where any 

provider-supplied configuration data will have been placed by 

NIST. This location will otherwise be empty.

The implementation is permitted read-write-delete access to the 

enrollment directory during this phase. The implementation is 

permitted read-only access to the configuration directory.

After enrollment, NIST may rename and relocate the enrollment 

directory - the implementation should not depend on the name 

of the enrollment directory.
IREX III

NIST Concept, Evaluation Plan and API Page 19 of 34

E2 Parallel

Enrollment

For each of N individuals, pass multiple images of the individual 

to the implementation for conversion to a combined template. 

The implementation will return a template to the calling 

application.

The implementation is permitted read-only access to the 

enrollment directory during this phase. NIST's calling application 

will be responsible for storing all templates as binary files. These 

will not be available to the implementation during this enrollment

phase.

Multiple instances of the calling application may run 

simultaneously or sequentially. These may be executing on 

different computers. The same person will not be enrolled twice.

Statistics of the times needed to 

enroll an individual.

Statistics of the sizes of created 

templates.

The incidence of failed template 

creations.

E3 Finalization Permanently finalize the enrollment directory. This supports, for 

example, adaptation of the image-processing functions, 

adaptation of the representation, writing of a manifest, indexing,

and computation of statistical information over the enrollment 

dataset.

The implementation is permitted read-write-delete access to the 

enrollment directory during this phase.

Size of the enrollment database as a 

function of population size N and 

the number of images.

Duration of this operation. The time 

needed to execute this function 

shall be reported with the preceding 

enrollment times.

Pre-search

S1 Initialization Tell the implementation the location of an enrollment directory. 

The implementation could look at the enrollment data.

The implementation is permitted read-only access to the 

enrollment directory during this phase.

Statistics of the time needed for this 

operation.

Statistics of the time needed for this 

operation.

S2 Template 

preparation

For each probe, create a template from a set of input images. 

This operation will generally be conducted in a separate process

invocation to step S2.

The implementation is permitted no access to the enrollment 

directory during this phase.

The result of this step is a search template.

Statistics of the time needed for this 

operation.

Statistics of the size of the search 

template.

Search

S3 Initialization Tell the implementation the location of an enrollment directory. 

The implementation should read all or some of the enrolled data 

into main memory, so that searches can commence.

The implementation is permitted read-only access to the 

enrollment directory during this phase.

Statistics of the time needed for this 

operation.

S4 Search A template is searched against the enrollment database. 

The implementation is permitted read-only access to the 

enrollment directory during this phase.

Statistics of the time needed for this 

operation.

Accuracy metrics - Type I + II error 

rates.

Failure rates.

1

2 3.5.2. Initialization of the enrollment session

3 Before any enrollment feature extraction calls are made, the NIST test harness will call the initialization function of

4 Table 25.

5 Table 25 – Enrollment initialization 

Prototype int32_t initialize_enrollment_session(

const char *configuration_location, Input

const char *enrollment_directory, Input

const uint32_t num_persons, Input

const uint32_t num_images) Input

Description This function initializes the implementation under test and sets all needed parameters. This function will be called 
IREX III

NIST Concept, Evaluation Plan and API Page 20 of 34

N=1 times by the NIST application immediately before any M  1 calls to convert_multiiris_to_enrollment_template. 

The implementation should tolerate execution of P > 1 processes on the same machine each of which may be 

reading and writing to the enrollment directory. This function may be called P times and these may be running 

simultaneously and in parallel.

Input 

Parameters

configuration_location A read-only directory containing any vendor-supplied configuration parameters or runtime data files.

enrollment_directory The directory will be initially empty, but may have been initialized and populated by 

separate invocations of the enrollment process. When this function is called, the

implementation may populate this folder in any manner it sees fit. Permissions will be 

read-write-delete.

num_persons The number of persons who will be enrolled 0 ≤ N ≤ 232- 1 (e.g. 1million)

num_images The total number of images that will be enrolled, summed over all identities 0 ≤ M ≤ 232- 1 

(e.g. 1.8 million)

Output

Parameters

none

Return Value 0 Success

2 The configuration data is missing, unreadable, or in an unexpected format.

4 An operation on the enrollment directory failed (e.g. permission, space).

6 The implementation cannot support the number of persons or images.

8 The descriptions are unexpected, or unusable.

Other Vendor-defined failure

1

2 3.5.3. Enrollment

3 A MULTIIRIS is converted to a single enrollment template using the function of Table 26.

4 Table 26 – Enrollment feature extraction

Prototypes int32_t convert_multiiris_to_enrollment_template(

const MULTIIRIS *input_irises, Input

MULTISEGMENTATION *output_properties, Output

uint32_t *template_size, Output

uint8_t *proprietary_template); Output

Description This function takes a MULTIIRIS, and outputs a proprietary template. The memory for the output template is 

allocated by the NIST test harness before the call i.e. the implementation shall not allocate memory for the result. 

If the function executes correctly (i.e. returns a zero exit status), the NIST calling application will store the

template. The NIST application will concatenate the templates and pass the result to the enrollment finalization 

function (see section 3.5.4).

If the function gives a non-zero exit status:

 If the exit status is 8, NIST will debug, otherwise

 the test driver will ignore the output template (the template may have any size including zero)

 the event will be counted as a failure to enroll. Such an event means that this person can never be identified 

correctly.

IMPORTANT. NIST's application writes the template to disk. The implementation must not attempt writes to the 

enrollment directory (nor to other resources). Any data needed during subsequent searches should be included in 

the template, or created from the templates during the enrollment finalization function of section 3.5.4.

Input

Parameters

input_irises An instance of a Table 16 structure. Implementations must alter their behavior according to 

the number of images contained in the structure.

Output 

Parameters

output_properties For each input image in the MULTIIRIS the function shall return the estimated iris and pupil

centers, and image qualities. The calling application will pre-allocate the correct number of 

ONESEGMENTATION structures (i.e. one for each image in the MULTIIRIS). The calling 

application will NOT initialize this memory. The implementation must guarantee sensible 

values on return.

template_size The size, in bytes, of the output template
IREX III

NIST Concept, Evaluation Plan and API Page 21 of 34

proprietary_template The format is entirely unregulated. NIST will allocate a KT byte buffer for this template: The 

value K is the number of images in the MULTIIRIS; the value T is output by the maximum 

enrollment template size function of Table 22.

Return Value 0 Success

2 Elective refusal to process this kind of MULTIIRIS

4 Involuntary failure to extract features (e.g. could not find iris in the input-image)

6 Elective refusal to produce a template (e.g. insufficient pixels between the eyes)

8 Cannot parse input data (i.e. assertion that input record is non-conformant)

Other Vendor-defined failure. Failure codes must be documented and communicated to NIST 

with the submission of the implementation under test.

1

2 3.5.4. Finalize enrollment

3 NIST will write an application around the sole function call of Table 27. Implementations shall not require calls to any 

4 other (initialization) functions.

5 After all templates have been created, in prior processes, the function of Table 27 will be called. This freezes the 

6 enrollment data. After this call the enrollment dataset will be forever read-only. This API does not support 

7 interleaved enrollment and search phases.

8 The function allows the implementation to conduct, for example, statistical processing of the feature data, indexing 

9 and data re-organization. The function may alter the file structure. It may increase or decrease the size of the stored 

10 data. No output is expected from this function, except a return code. 

11 Table 27 – Enrollment finalization

Prototypes int32_t finalize_enrollment (

const char *enrollment directory, Input

const char *edb_name, Input

const char *edb_manifest_name); Input

Description This function takes the name of the top-level directory where enrollment database (EDB) and its manifest have 

been stored. These are described in section 2.7. The enrollment directory permissions will be read + write. 

The function supports post-enrollment vendor-optional book-keeping operations and statistical processing. The 

function will generally be called in a separate process after all the enrollment processes are complete.

This function should be tolerant of being called two or more times. Second and third invocations should probably 

do nothing.

Input

Parameters

enrollment_directory The top-level directory in which enrollment data was placed. This variable allows an 

implementation to locate any private initialization data it elected to place in the directory.

edb_name The name of a single file containing concatenated templates, i.e. the EDB of section 2.7.

While the file will have read-write-delete permission, the implementation should only alter 

the file if it preserves the necessary content, in other files for example.

The file may be opened directly. It is not necessary to prepend a directory name.

edb_manifest_name The name of a single file containing the EDB manifest of section 2.7.

The file may be opened directly. It is not necessary to prepend a directory name.

Output 

Parameters

None

Return Value 0 Success

2 Cannot locate the input data - the input files or names seem incorrect.

4 An operation on the enrollment directory failed (e.g. permission, space).

6 One or more template files are in an incorrect format.

Other Vendor-defined failure. Failure codes must be documented and communicated to NIST 

with the submission of the implementation under test.

12

13
IREX III

NIST Concept, Evaluation Plan and API Page 22 of 34

1 3.5.5. Pre-search feature extraction

2 3.5.5.1. Scope

3 This section defines the API for production of templates from search images. Templates produced during enrollment 

4 will not be used during search. This allows role-specific asymmetric templates. NIST will write an application around

5 function calls of 3.5.5.2 and 3.5.5.3. Implementations shall not require calls to any other (initialization) functions.

6 3.5.5.2. Initialization

7 Before a MULTIIRIS is sent to the identification feature extraction function, the test harness will call the initialization 

8 function in Table 28.

9 Table 28 – Identification feature extraction initialization 

Prototype int32_t initialize_feature_extraction_session(

const char * configuration_location, Input

const char * enrollment directory); Input

Description This function initializes the implementation under test and sets all needed parameters. This function will be 

called N=1 times by the NIST application immediately before any M  1 calls to 

convert_multiiris_to_identification_template. The implementation should tolerate execution of P > 1 processes 

on the same machine each of which can read the configuration directory. This function may be called P times 

and these may be running simultaneously and in parallel.

The implementation has read-only access to its prior enrollment data.

Input 

Parameters

configuration_location A read-only directory containing any vendor-supplied configuration parameters or 

run-time data files.

enrollment_directory The top-level directory in which enrollment data was placed and then finalized by the 

implementation. The implementation can parameterize subsequent template 

production on the basis of the enrolled dataset.

Output

Parameters

none

Return Value 0 Success

2 The configuration data is missing, unreadable, or in an unexpected format.

4 An operation on the enrollment directory failed (e.g. permission).

Other Vendor-defined failure

10

11 3.5.5.3. Feature extraction

12 A MULTIIRIS is converted to an atomic identification template using the function of Table 29. The result may be 

13 stored by NIST, or used immediately. The implementation shall not attempt to store any data.

14 Table 29 – Identification feature extraction

Prototypes int32_t convert_multiiris_to_identification_template(

const MULTIIRIS *input_irises, Input

MULTISEGMENTATION *output_properties, Output

uint32_t *template_size, Output

uint8_t *identification_template); Output

Description This function takes a MULTIIRIS, and outputs a proprietary template. The memory for the output template is 

allocated by the NIST test harness before the call i.e. the implementation shall not allocate memory for the result. 

If the function executes correctly, it returns a zero exit status. The NIST calling application may commit the template 

to permanent storage, or may keep it only in memory (the vendor implementation does not need to know). If the 

function returns a non-zero exit status, the output template will be not be used in subsequent search operations.

The function shall not have access to the enrollment data, nor shall it attempt access.

Input

Parameters

input_irises An instance of a Table 16 structure. Implementations must alter their behavior according to 

the number of images contained in the structure.

Output output_properties For each input image in the MULTIIRIS the function shall return the estimated iris and pupil
IREX III

NIST Concept, Evaluation Plan and API Page 23 of 34

Parameters centers, and image qualities. The calling application will pre-allocate the correct number of 

ONESEGMENTATION structures (i.e. one for each image in the MULTIIRIS). The calling 

application will NOT initialize this memory. The implementation must guarantee sensible 

values on return.

template_size The size, in bytes, of the output template

identification_template The output template for a subsequent identification search. The format is entirely 

unregulated. NIST will allocate a KT byte buffer for this template: The value K is the number 

of images in the input MULTIIRIS; the value T is output by the maximum enrollment 

template size function of Table 22.

Return 

Value

0 Success

2 Elective refusal to process this kind of MULTIIRIS

4 Involuntary failure to extract features (e.g. could not find iris in the input-image)

6 Elective refusal to produce a template (e.g. insufficient pixels between the eyes)

8 Cannot parse input data (i.e. assertion that input record is non-conformant)

Other Vendor-defined failure. Failure codes must be documented and communicated to NIST with 

the submission of the implementation under test.

1

2 3.5.6. Search

3 3.5.6.1. Scope

4 Once search templates have been produced, they may be searched against an enrollment database. NIST will write 

5 an application around function calls of 3.5.6.2 and 3.5.6.3. Implementations shall not require calls to any other 

6 (initialization) functions.

7 3.5.6.2. Initialization

8 The function of Table 30 will be called once prior to one or more calls of the searching function of Table 31. The 

9 function might set static internal variables so that the enrollment database is available to the subsequent 

10 identification searches.

11 Table 30 – Identification initialization

Prototype int32_t initialize_identification_session(

const char *configuration_location, Input

const char *enrollment_directory); Input

Description This function reads whatever content is present in the enrollment_directory, for example a manifest placed 

there by the finalize_enrollment function.

Input 

Parameters

configuration_location A read-only directory containing any vendor-supplied configuration parameters or runtime data files.

enrollment_directory The top-level directory in which enrollment data was placed.

Return Value 0 Success

Other Vendor-defined failure

12

13 3.5.6.3. Search

14 The function of Table 31 compares a proprietary identification template against the enrollment data and returns a 

15 candidate list.

16 Table 31 – Identification search

Prototype int32_t identify_template(

const uint8_t *identification_template, Input

const uint32_t identification_template_size, Input

const uint32_t candidate_list_length, Input

CANDIDATE * const *candidate_list); Output
IREX III

NIST Concept, Evaluation Plan and API Page 24 of 34

Description This function searches a template against the enrollment set, and outputs a list of candidates.

NIST will allocate memory for the candidates before the call.

Input 

Parameters

identification_template A template from convert_multiiris_to_identification_template() - If the value 

returned by that function was non-zero the contents of identification_template 

will not be used and this function (i.e. identify_template) will not be called.

identification_template_size The size, in bytes, of the input identification template 0 ≤ N ≤ 232- 1

candidate_list_length The number of candidates the search should return

Output

Parameters

candidate_list An array of "candidate_list_length" pointers to candidates. The datatype is 

defined in section 2.8. Each candidate shall be populated by the 

implementation. The candidates shall appear in ascending order of distance

score - i.e. most similar entries appear first. The calling application will allocate 

memory for the candidates before this call. The calling application will NOT 

initialize this memory. The implementation must guarantee sensible values on 

return.

Return Value 0 Success

2 The input template was defective.

Other Vendor-defined failure

1

2 NOTE: Ordinarily the calling application will set the input candidate list length to operationally typical values, say 0 

3 L  100, and L << N. We may therefore extend the candidate list length such that L approaches N. 

4 4. Software and Documentation

5 4.1. Implementation library and platform requirements

6 Participants shall provide NIST with binary code only (i.e. no source code). Header files ( “.h”) should not be 

7 necessary. They are allowed, but these shall not contain intellectual property of the company nor any material that is 

8 otherwise proprietary. It is preferred that the implementation be submitted in the form of a single static library. 

9 However, dynamic and shared library files are permitted.

10 The core library shall be named according to Table 32. If necessary additional dynamic or shared library files may be 

11 submitted that support this “core” library file (i.e. the “core” library file may have dependencies implemented in 

12 these other libraries).

13 Intel IPP libraries are not permitted, and will not be supplied.

14 Access to any GPUs is not permitted.

15 Table 32 – Implementation library filename convention

Form libIREX_provider_class_sequence.ending

Underscore 

delimited parts of 

the filename

libIREX provider classes sequence ending

Description First part of the 

name, required to 

be this.

Single word name of 

the main provider

EXAMPLE: Acme

Functional class in 

Table 3.

EXAMPLE: A

A two digit decimal 

identifier to start at 00 

and increment by 1 

every time any 

implementation is sent 

to NIST. EXAMPLE: 07

Either .so or .a

Example libIREX_Enron_A_07.a

16

17 NIST will report the size of the supplied libraries.
IREX III

NIST Concept, Evaluation Plan and API Page 25 of 34

1 4.2. Configuration and vendor-defined data

2 The implementation under test may be supplied with configuration files and supporting data files. The total size of 

3 the implementation, that is all libraries, include files, data files and initialization files shall be less than or equal to 1 073 

741 824 bytes = 102434 bytes.

5 NIST will report the size of the supplied configuration files.

6 4.3. Linking

7 On request, NIST will allow use of "g++" for linking, but the API must have "C" linkage. The Standard C++ library is 

available58 . The prototypes of this document will be written to a file "irex.h" which will be included via 

extern "C"

{

#include <irex.h>

}

9 NIST will handle all input of images via NETPBM or PNG libraries..

10 All compilation and testing will be performed on x86 platforms. Thus, participants are strongly advised to verify 

11 library-level compatibility with gcc (on an equivalent platform) prior to submitting their software to NIST to avoid 

12 linkage problems later on (e.g. symbol name and calling convention mismatches, incorrect binary file formats, etc.).

13 NIST will not allow or support Intel Integrated Performance Primitives (Intel IPP) and "icc" compiled libraries. See the 

14 processor specifications in section 1.20.

15 For this test, Windows machines will not be used. Windows-compiled libraries are not permitted. All software must 

16 run under LINUX.

17 Dependencies on external dynamic/shared libraries such as compiler-specific development environment libraries are 

18 discouraged. If absolutely necessary, external libraries must be provided to NIST upon prior approval by the Test 

19 Liaison.

20 4.4. Installation and Usage

21 The implementation must install easily (i.e. one installation step with no participant interaction required) to be tested, 

22 and shall be executable on any number of machines without requiring additional machine-specific license control 

23 procedures or activation.

24 The implementation shall be installable using simple file copy methods. It shall not require the use of a separate 

25 installation program. 

26 The implementation shall neither implement nor enforce any usage controls or limits based on licenses, number of 

27 executions, presence of temporary files, etc. The implementations shall remain operable until October 31, 2012.

28 Hardware (e.g. USB) activation dongles are not acceptable.

29 4.5. Hard disk space

30 IREX III participants should inform NIST if their implementations require more than 100K of persistent storage, per 

31 enrolled image on average.

 

5

This includes the compiler that installs with RedHat, which is Target: x86_64-redhat-linux configured with: ../configure --prefix=/usr --

mandir=/usr/share/man -- infodir=/u sr/share/info --enable-shared --enable-threads=posix --enable- checking=release - -with-system-zlib --enable-

__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --

disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic --host=x86_64-redhat-linux Thread model: posix 

gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)

The libraries are what shipped with RH 5.1: /usr/lib64/libstdc++.so.6.0.8 lib/libc.so.6 -> libc-2.5.so /lib/libm.so.6 -> libm-2.5.so
IREX III

NIST Concept, Evaluation Plan and API Page 26 of 34

1 4.6. Documentation

2 Participants shall provide complete documentation of the implementation and detail any additional functionality or 

3 behavior beyond that specified here. The documentation must define all (non-zero) vendor-defined error or warning 

4 return codes.

5 4.7. Modes of operation

6 Individual implementations provided shall not include multiple “modes” of operation, or algorithm variations. No 

7 switches or options will be tolerated within one library. For example, the use of two different “coders” by an feature

8 extractor must be split across two separate implementation libraries, and two separate submissions.

9 4.8. Runtime behavior

10 4.8.1. Interactive behavior

11 The implementation will be tested in non-interactive “batch” mode (i.e. without terminal support). Thus, the 

12 submitted library shall not use any interactive functions such as graphical user interface (GUI) calls, or any other calls 

13 which require terminal interaction e.g. reads from “standard input”.

14 4.8.2. Error codes and status messages

15 The implementation will be tested in non-interactive “batch” mod, without terminal support. Thus, the submitted 

16 library shall run quietly, i.e. it should not write messages to "standard error" and shall not write to “standard output”. 

17 An implementation may write debugging messages to a log file - the name of the file must be declared in 

18 documentation.

19 4.8.3. Exception Handling

20 The application should include error/exception handling so that in the case of a fatal error, the return code is still 

21 provided to the calling application.

22 4.8.4. External communication

23 Processes running on NIST hosts shall not side-effect the runtime environment in any manner, except for memory 

24 allocation and release. Implementations shall not write any data to external resource (e.g. server, file, connection, or 

25 other process). Implementations shall not attempt to read any resource other than those explicitly allowed in this 

26 document. If detected, NIST reserves the right to cease evaluation of all implementations from the supplier, 

27 notification to the provider, and documentation of the activity in published reports.

28 4.8.5. Stateful behavior

29 All components in this test shall be stateless, except as noted. This applies to iris detection, feature extraction and 

30 matching. Thus, all functions should give identical output, for a given input, independent of the runtime history. 

31 NIST will institute appropriate tests to detect stateful behavior. If detected, NIST reserves the right to cease 

32 evaluation all implementations from the supplier, notification to the provider, and documentation of the activity in 

33 published reports.

34 5. References

MBE P. Grother, G .W. Quinn, and P. J. Phillips, Multiple-Biometric Evaluation (MBE) 2010, Report on the Evaluation of 2D StillImage Face Recognition Algorithms, NIST Interagency Report 7709, Released June 22, 2010. Revised August 23, 2010. 

http://face.nist.gov/mbe

HAO F. Hao, J. Daugman, and Z. Piotr. A fast search algorithm for a large fuzzy database. IEEE Transactions on Information

Forensics and Security, 3(2):203–212, June 2008.

IREX P. Grother, E. Tabassi, G. W. Quinn, W. Salamon, Iris Exchange I (IREX I), Performance of Iris Recognition Algorithms on 

Standard Images, NIST Interagency Report 7629, October 22, 2009. http://iris.nist.gov/irex

PERFSTD

INTEROP

ISO/IEC 19795-4 — Biometric Performance Testing and Reporting — Part 4: Interoperability Performance Testing. 

Posted as document 37N2370. The standard was published in 2007. It can be purchased from ANSI at 
IREX III

NIST Concept, Evaluation Plan and API Page 27 of 34

http://webstore.ansi.org/.

ISO

STD11

ISO/IEC 19794-5:2011 — Information technology — Biometric data interchange formats — Part 5: Irisimage data. The 

standard is expected to be completed in January 2011, and formally published in Summer 2011. It will replace the 

original ISO standard published in 2005. The standard can be purchased from ANSI at http://webstore.ansi.org/

UVW Rajiv Mukherjee and Arun Ross, Indexing Iris Images, in Proc. of International Conference on Pattern Recognition (ICPR), 

(Tampa, USA), December 2008.

1

2
IREX III

NIST Concept, Evaluation Plan and API Page 28 of 34

1 Annex A

2 Submission of Implementations to IREX III 

3 A.1 Confidentiality and integrity protection

4 NIST requires that all software, data and configuration files submitted by the participants be signed and encrypted. 

5 Signing is done with the participant's private key, and encryption is done with the NIST public key. The detailed 

6 commands for signing and encrypting are given here: http://iris.nist.gov/irex/crypto_protection.pdf [Link is correct 

7 Jan 28 2010].

8 NIST will validate all submitted materials using the participant's public key, and the authenticity of that key will be 

9 verified using the key fingerprint. This fingerprint must be submitted to NIST by writing it on the signed participation 

10 agreement.

11 By encrypting the submissions, we ensure privacy; by signing the submission, we ensure authenticity (the software 

12 actually belongs to the submitter). NIST will not accept into IREX III any submission that is not signed and encrypted. 

13 NIST accepts no responsibility for anything that is transmitted to NIST that is not signed and encrypted with the 

14 NIST public key.

15 A.2 How to participate

16 Those wishing to participate in IREX III testing must do all of the following, on the schedule listed on Page 2.

17 ― IMPORTANT: Follow the instructions for cryptographic protection of clause A.1

18 ― Send a signed and fully completed copy of the Application to Participate in the IREX III Evaluation (linked from 

19 http://iris.nist.gov/irex under IREX III). This must identify, and include signatures from, the Responsible Parties as 

20 defined in section XX. The properly signed IREX III Application to Participate shall be sent to NIST as a signed then 

21 scanned PDF file.

22 ― Provide an implementation library which complies with the API (Application Programmer Interface) specified in 

23 this document.

24  Encrypted data and implementations below 20MB can be emailed to NIST at irex@nist.gov

25  Encrypted data and implementations above 20MB shall be

 Made available as a file.zip.gpg or file.zip.asc download from a generic webserver626 , or:

27  Mailed as a file.zip.gpg or file.zip.asc on CD / DVD to NIST at this address:

IREX III Test Liaison (A203)

100 Bureau Drive

A203/Tech225/Stop 8940

NIST

Gaithersburg, MD 20899-8940

USA

In cases where a courier needs a phone number please use 

NIST shipping and handling on: 301 -- 975 -- 6296.

28

29 A.3 Implementation validation

30 Registered Participants will be provided with small validation dataset and programs available on the website 

31 http://iris.nist.gov/irex at XXX (TBA).

32 The validation test programs shall be compiled by the provider. The output of these programs shall be submitted to 

33 NIST.

 

6 NIST will not register, or establish any kind of membership, on the provided website.
IREX III

NIST Concept, Evaluation Plan and API Page 29 of 34

1 Prior to submission of the implementation and validation data, the Participant must verify that their software 

2 executes on the validation images, and produces correct distance scores and templates.

3 Software submitted shall implement the IREX III API Specification as detailed in the body of this document.

4 Upon receipt of the implementation and validation output, NIST will attempt to reproduce the same output by 

5 executing the implementation on the validation imagery, using a NIST computer. In the event of disagreement in the 

6 output, or other difficulties, the Participant will be notified.