Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

CMT/1 Timeline

p.4, A.3.1

ge Schedule requires “Participant” status prior to access to

Validation Dataset. This means a vendor must commit to

participating prior to seeing any validation images.

Allow access to Validation Dataset upon submission of

“Intention to Participate” so vendors can make final

decision about participation after preliminary testing

with Validation Dataset

For procedural consistency with

other NIST tests, will keep

schedule as is.

NIST hopes to make or facilitate

making some iris imagery public

for research purposes.

L1/1 1. bullet list ge We assumed that NIST plans to compile or freshly collect 

test image databases that can be given to the 

participants. 

If this is the case add corresponding task to the 

itemized list. 

Noted

Fresh data collection by NIST

has never been part of the plan.

NIST has issued call for iris

imagery and hopes iris

community will share their iris

images with NIST.

Iritech/1 1 Table 1 te Iris boundary shape quality metric might be split into iris‐pupil

boundary shape and iris‐sclera boundary shape. However,

visible iris‐sclera boundary shape is closely related to sight

direction quality and may not play own role.

Iris boundary shape quality metric might be split into

iris‐pupil boundary shape and iris‐sclera boundary

shape.

Accept in principle

See L1/6

This has been added to Table 4

and not Table 1, since Table 1

just lists quality metrics

mentioned in SC 37 N3331. One

of the planned outcome of IQCE

is to refine the list of quality

metrics in N3331.

Note that evaluating these will

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 1 of 1 

1 
 

 

 

 

 

 

 

 

 

 

 

 

           

   

         

           

       

               

           

                 

                   

                   

                   

             

               

         

     

       

           

     

         

      

       

       

      

         

       

           

       

     

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

be difficult at least because it

requires ground‐truth boundary

shape.

NIST will report the interaction

of iris shape and gaze (sight)

direction quality metrics. (see

CAM/4).

Iritech/2 1 Table 1 te It seems there are no detection qualities among quality

metrics. Of course, detection qualities are algorithm‐

dependent and also are product of other “primary” image

qualities like sharpness or signal‐to‐noise ratio, but we are sure

some of them will be useful for estimating matching error

rates. Thus we propose to include such measures to quality

metrics.

Add “Pupil detection quality”, “Iris detection quality”

and “occlusion detection quality” to the third column

of the table

Reject

1. Agree that segmentation

accuracy greatly impacts

matching performance.

Several image properties such

as iris size (e.g. very large

irises), margin (e.g. off‐centre

or cropped irises) could cause

segmentation failure. Some

algorithms might be more

sensitive to such image

impairments than others.

NIST will relate failure to create

(proprietary) template to image

quality features such as iris size,

margin, and sharpness and

signal‐to‐noise ratio, as

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 2 of 2 

1 
 

 

 

 

1 2 (3) 4 5 (6) (7)

 # Clause Paragraph/ Type Comment (justification for change) Proposed change NIST Proposed Disposition 

No./ Figure/Table/ of 

Subclause Note comment1 No./ (e.g. Table 1) 

Annex 

(e.g.3, 3.1,

Annex A, 

A.1) 

suggested by the comment.  

2. Vendors can choose to

compute segmentation‐

accuracy as a proprietary

quality metric in position 33‐64 

The disjunction CAM/1 1 P. 7.26, te

sense, as these

Table 1 concepts. Why

of their IQAA.

"Noise (or camera sensitivity)" does not make Accept in principle. are two quite different and independent

the ("or") disjunction? Table 1 lists quality metrics as

appeared in N3331. Will make

a comment to N3331 to

separate them.

Q: Should a new entry be added

to Table 4?

L1/2 1. Table 1, te A distinction between motion blur due to camera shake Introduce a comment on this fact. Accept in principle.

column “Iris and motion b lur due to eye/eyelid movement may be or Add the following NOTE after Acquisition useful. The two types of motion blur give rise to 

Covariates” Table 4. potentially different methods of compensation. Replace the cell “Motion Blur” by two cells “camera 

shake” and “eye and eyelid motio n blur”. NOTE Motion blur occurs

because either the iris is

moving (e.g., subject or

subject’s eye moving), or the

device is moving     (e.g., active

tracking devices).     These two

causes of motion blur, give rise

 

 

 

 

 

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 3 of 3 

1 
 

 

 

 

 

 

 

 

 

 

 

 

       

         

         

           

       

       

 

 

 

 

 

    

     

               

                     

           

           

       

           

         

     

     

           

   

                                 

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

to potentially different methods

of compensation. Participants

can choose to submit two

distinct metrics for motion blur,

one at position 10 and another

one in vendor‐defined portion

of IQAA output (i.e. positions

33‐64).

L1/3 1. Table 1, 

column “Iris 

Acquisition 

Covariates” 

te Magnification does not directly depend on resolution and 

should be included in the list of Iris Acquisition 

Covariates. It correlates with the quality metric “iris size 

in pixels” 

Add another cell “Magnification”. Accept in principle.

Added magnification to Table 4.

(Table 1 is a list of covariates in

N3331.)

Iritech/4 1 Table 1 te Interlace may be added to quality metrics. Interlacing artifacts,

even if detected, cannot be fully compensated.

Add “Interlace” to the third column Accept in principle.

Added interlace to Table 4.

Interlace is bad and shall be

avoided as most cameras do

nowadays.

While 19794‐6 shall ban

interlace, 29794‐6 shall include

it as a q‐metric because of

legacy data.

Iritech/5 1 Table 1 ed Iris size item is in smaller font Enlarge font for “Iris size” item in the third column of

the table

Accept

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 4 of 4 

1 
 

 

 

 

 

 

 

 

 

 

 

 

                   

              

             

               

       

             

         

   

 

        

     

          

       

             

         

       

           

        

               

               

       

                 

         

   

     

   

   

         

        

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

CMT/2 2 te A primary output is listed as measurements of effectiveness of

IQAAs in predicting false nonmatch rate. Some degradations

may impact imposter distributions in addition to genuine

distributions, so such measurements should predict the impact

on both FMR and FNMR.

A primary output should be listed as “Measurements

of effectiveness of IQAAs in predicting false‐non‐

match‐rate and/or false‐match‐rate...”

Accept

L1/4 2 te To which FRR range and algorithm do tolerance values 

refer? Do they make sense at all? 

Noted

Clause 2 cautiously mentions

“tolerance bounds when

possible”. For some of the

metrics, tolerance bounds may

not make sense or may not be

possible. For some other

metrics (e.g. pupil‐iris‐ratio, or

margin, or maybe iris size) it

might make sense.

CMT/3 5 te Motivation and background does not explicitly mention the

use of quality information for quality‐based biometric fusion,

an important use of quality.

Modify first sentence to read “...and in invocation of

quality‐directed processing of samples, including

quality‐based biometric fusion.”

Accept

Iritech/6 9.2 page11

line 39

ed Typographic error “Element 14‐16 are reserved” should be

“Element 15‐16 are reserved”

Accept in principle.

The size of q‐vector has

changed. Elements 16‐32 are

reserved.

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 5 of 5 

1 
 

 

 

 

 

 

 

 

 

 

 

 

                     

          

               

                 

               

                

           

             

                 

   

                       

                 

         

             

                 

                   

                     

                 

                   

               

          

         

   

       

     

       

     

       

                               

         

         

        

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

CMT/4 9.2 te Use of compressed images is not considered, yet in some cases

compression may introduce additional degradation.

Compression effects are probably beyond the scope of

IQCE but it should at least be acknowledged that

compression may further reduce the quality of the

image. Add an additional sentence to the first

paragraph: “Compression, if applied to image samples,

may reduce their quality but in properly designed

systems such reduction should not be great enough to

significantly affect performance”

Accept

CAM/2 9.2 IQAA

output

Table 4

P. 11.27‐29

te There may be a use case for quality assessment on

compressed images, if only to establish that canonical quality

metrics cannot_detect_compression, per se; yet obviously (as

shown in IREX I), compression does affect performance.

Moreover, in some scenarios, only a compressed image is

available (e.g. an IREX Kind=3 or Kind=7 compressed iris image

record); so the quality of compressed images clearly has a use

case.

In summary, for the above reasons, it would be

interesting to see how quality metrics predict (or fail to

predict) performance on compressed images.

Noted

Input format is only VGA and

Kind3. Kind7 is not considered

as an input format here.

Will examine how overall

quality score (i.e. not quality

metrics in positions 2‐64)

predict or fail to predict

performance on compressed

images.

IREX I showed that compression

affects performance.

CAM/3 9.2 Table 4

P. 11.41

te What is "gray scale density"? Is it just pixel value (0 ‐ 255)? Change to gray level spread.

N3331 clause 6.4.2.2 heading is

gray scale density but talks

about gray scale resolution.

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 6 of 6 

1 
 

 

 

 

 

 

 

 

 

 

 

 

       

           

           

     

 

 

 

 

          

        

       

       

   

                 

     

                

                

                      

           

       

         

       

             

               

             

     

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

6.4.2.2 has a note that suggests

use of some measure of spread

(std or kurtosis to measure gray

scale distribution.

See L1/5, and CMT/5.

L1/5 9.2 Table 4 ge The metrics “Gray scale density”, “Sight direction”, “Iris 

boundary shape” could be named more descriptively 

using “Gray level range”, “Gaze direction”, and “Iris 

circularity”. 

Consider switching to these new terms (submit 

corresponding comments towards the quality 

standard). 

Partial Accept

Changed gray level density to

gray level spread. See CAM/3.

Kept iris‐sclera‐boundary

shape. Avoided Iris circularity

because it assumes irises

should be circle.

CMT/5 9.2 Table 4 te Some of the descriptions in this table, taken from SC37‐N‐

3331, are not accurate.

Change “Gray scale density” to “Gray scale range”.

Change “Head orientation” to “Head roll angle.” (This

is consistent with face image definitions).

Partial accept

N3331 uses gray level density.

Have changed it to “gray level

spread”. See CAM/3 and L1/5.

Use “Head rotation” per clause

6.3.8 of N3331. See Iritech/3.

Iritech/3 9.2 Table 4 te Does head orientation mean head rotation? The term head

orientation seems ambiguous to us .In ISO29794‐6, head

rotation is defined rather than head orientation.

Change “Head orientation” into “Head rotation” Accept

See CMT/5.

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 7 of 7 

1 
 

 

 

 

 

 

 

 

 

 

 

 

 

         

         

                       

               

                         

                       

                 

                   

               

         

     

       

       

     

                                     

       

           

                     

                             

                   

                   

                 

             

             

   

         

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

L1/6 9.2 Table 4 te Add pupil boundary circularity”. Partial Accept

Added pupil shape to Table 4

Prefer to avoid the term

“circularity”.

See Iritech/1

CAM/4 9.2 Table 4 –

rows 6,7

P. 11.41

te 6 and 7 interact: a change in gaze direction will affine‐deform

the iris shape.

Noted

NIST will examine and report on

this interaction. (Iritech/1)

CAM/5 9.2 Table 4, row

10

P. 11.41

te (please refer back to 11.33): In order for quality vector

elements to be normalised onto a unit interval like [0, 100], it

is vastly better to refer to "pupil‐iris ratio" (which guarantees

this), than to "iris‐pupil ratio". Also, it should be clarified

whether this refers to diameter, or to area (quadratically

related). (I presume diameter ratio is intended.)

Noted.

Table 4 is using pupil‐iris‐ratio.

Diameter ratio is intended.

(N3331 defines Pupil‐iris ratio

as ratio of diameters.) 

CAM/6 9.2 Table 4, row

11

P. 11.41

Q Is "sharpness" the same thing as a focus score (as in the

literature)?

Sharpness is not just focus.

Incorrect focal length will cause

lack of sharpness, so will other

things.

CMT/6 9.2 Table 4 te N3331 describes “image scale” in terms of the margins on each

side of the iris – this is not included in Table 4 as a quality

metric but perhaps it should be, since some algorithms may

require more area around the iris to accurately locate the

Define an additional table entry called “Field of View”

that measures the extent to which the recommended

cropping margins (0.2R vertical, 0.6R horizontal) are

Accept in principle.

Used “margin” instead of “field

of view”

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 8 of 8 

1 
 

 

 

 

 

 

 

 

 

 

 

 

   

                      

                  

                       

               

                        

                     

               

                 

                       

                   

             

                     

               

                   

                  

         

           

     

   

      

           

     

       

   

     

            

         

           

         

     

      

       

       

     

         

         

      

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

limbus and eyelids. achieved.

CMT/7 9.2 Table 4 te The use of “Visible iris area” and “Occlusions” as two separate

metrics is confusing and (probably) redundant. What is most

important is how much of the iris is visible and available for

matching, i.e. not occluded by eyelids, eyelashes, reflections,

etc. As written “Visible iris area” is the iris area not occluded

by eyelids and “Occlusions” is how much of this area not

occluded by eyelids is occluded by something else.

Either (a) define one “visible iris” metric that measures

how much of the iris is not occluded by anything or (b)

define two metrics, “visible iris” as above plus a second

called “eyelid occlusion” that measures what fraction

of the total iris area is occluded by eyelids, or (c)

“visible iris” and “eyelid occlusion” as above plus

“other occlusion” for what fraction of the total iris area

is occluded by eyelids, reflections, etc. Option (a) is

preferred.

N3331 only defines usable iris

area, and does not make any

distinctions among occlusion

caused by

eyelid/eyelash/specular

reflection/etc. Per Iritech

comment, the 2nd draft of IQCE

introduced distinction between

eyelid occlusion and other

occlusions (e.g. specular

reflection, eyelash, eye water, 

etc.) Visible iris was meant to

be the ratio (percentage) of iris

that is not occluded by eyelids,

and occlusion the percentage of

visible iris occluded by eyelash,

specular reflection, etc.

CrossMatch and CSC have

commented that this distinction

is not useful (because

ultimately it is non‐occluded iris

area that matters) and might

even be confusing.

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 9 of 9 

1 
 

 

 

 

 

 

 

 

 

 

 

 

         

           

             

           

    

       

       

       

         

     

   

   

 

 

 

 

 

 

       

         

       

       

             

               

              

             

                 

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

Therefore, the third draft of

IQCE will only include usable iris

area as a quality metric in Table

4 and delete visible iris and

occlusion. Participants are

welcome to measure and

report visible area and

occlusion separately in the

vendor defined portion of IQAA

output (e.g. positions 33‐64).

Iritech/7 9.2.2 page 13

line 4

ed Typographic error “positions 2‐13” should be

“positions 2‐14”

Partial Accept.

Change to 2‐17.

L1/7 9.2.2 Te A quality metric predictive of the false non-match rate 

may be better represented in the logarithmic space to 

avoid rescaling when the accuracy of the matching 

algorithm improves. 

Explore this concept in IREX II. Possibly mention it 

in the test spec. 

Will explore the logarithmic

space concept, not sure how

about relation with rescaling.

More detailed what‐to‐do is

much appreciated.

L1/8 9.3 ed plural required strategy • strategies Accept

CMT/8 9.4 Para 2 ed Lacks clarity. “Furthermore, NIST will examine how the quality

metrics for both enrolment and recognition images can

be evaluated jointly to better predict performance.

For example, the disparity between pupil‐iris diameter

ratio of enrolment and recognition images may be a

Accept

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 10 of 10 

1 
 

 

 

 

 

 

 

 

 

 

 

 

             

                    

                   

                   

               

                   

       

         

                   

               

             

             

                   

             

           

             

                 

     

       

         

     

   

     

   

                     

                     

         

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

better indicator of match performance than the pupil‐

iris ratio of either sample alone. Similarly, in the case

of iris occlusion, it may be more accurate to consider

the total occlusion produced by the union of the sets

of occluded pixels in the enrolment and recognition

images, rather than the occlusion (or visible iris) of just

one of the images.”

CAM/7 9.4

Measure

performan

ce

P. 13.42‐44 te These lines are difficult to parse.

Would this example be consistent with the intention of 9.4:

"For example, determine whether iris images with similar

pupil‐iris diameter ratios give more consistent matching

performance than images with dissimilar pupil‐iris diameter

ratios, and, if they do, quantify the difference in matching

performance over varying pupil‐iris diameter ratios." If not,

perhaps this point could use further clarification.

Comment: it is better, throughout, to refer to "pupil‐iris‐ratio"

(which always maps to the unit interval) rather than "iris‐pupil‐

ratio" (which does not).

The intent is as the comment

stated.

The paragraph is reworded per

CMT/8.

Sorry for poor wording.

Instances of iris‐pupil‐ratio

were mistakes. They are

corrected to pupil‐iris‐ratio.

CAM/8 9.6.

Measure

robustnes

P. 14.5

Q Will the fraction of images failing to produce output for each

IQAA be broken down by return value (e.g., elective refusal vs.

involuntary failure)?

YES

CAM/9 13. IQAA

execution

time

P. 14.34 ed Typographic error 100 milliseconds should be 50 Accept

See Iritech/8

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 11 of 11 

1 
 

 

 

 

 

 

 

 

 

 

 

 

       

   

   

         

                     

                   

                   

         

                   

   

             

       

         

       

       

                 

                    

                 

           

               

       

         

    

       

         

       

     

         

    

     

   

         

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

Iritech/8 13 page 14

line 34

ed Typographic error “The 100 milliseconds” should be

“The 50 milliseconds”

Accept

See CAM/9

Iritech/9 14.2.1 page 15

line 3

ed Typographic error “IREX participants” should be

“IQCE participants”

Partial Accept.

Change to IREX II IQCE

participants

CAM/10 14.2.2.

Sensor

identifiers

P. 15.5‐11 Q If IQCE uses atypical images from some of these sensors (e.g.

mystery Dalsa camera), can participants be made aware of the

likely range of variation of important image variables like iris

radius in pixels, and image dimensions?

Could the range of such critical image variables be made

available more generally?

NIST will aim to post CDF of

image variables like iris

diameters in pixel, and image

dimension for iris image

datasets being used for IREXII

IQCE on its FAQ page.

CMT/9 14.2.3 te Effects of image alteration or manipulation should not be

included in quality scores. They are more properly reflected in

the ability of specific matchers to compensate for certain

degradations, making them more robust to quality

impairment.

Effects of image alteration or manipulation should not

be included in quality scores.

Quality should be computed on

unaltered images.

Add to the test plan:

SDKs can choose to alter

images. Their success in

compensating for certain image

impairments will be reflected in

matching accuracy.

NOTE Effect of

alteration/enhancement on

other quality metrics is another

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 12 of 12 

1 
 

 

 

 

 

 

 

 

 

 

 

 

          

   

         

             

                 

             

             

                 

     

               

                   

                   

                     

                       

                 

                     

             

         

           

         

     

                           

   

                     

               

                 

               

           

     

   

     

                       

                                           

     

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

interesting topic. To keep this

study manageable (and

complete it in a reasonable

time), we will include this in our

future plan.

Iritech/1 14.2.3 Request for ge Image enhancement might give a problem in quality We suggest no enhancement should be done. See disposition to CMT/9

6 comment interoperability. Image enhancement by one vendor might

actually degrade the matching performance of other vendor.

CAM/11 14.2.3

Geometric,

photometr

ic or other

alterations

to images

P. 15.20‐21 ge Quality scores for altered/enhanced images could usefully be

made available, perhaps as optional fields in the quality vector.

If matching is done on a template/"record" generated from an

altered image, then the quality of the altered image should be

known.

For example, if a correction for deviated gaze is carried out, it

would be interesting to know how successfully that alteration

corrects the defect and what the effects on other measures of

quality are. Effects of alterations/enhancements on quality

and performance would be interesting in themselves.

Effects of alterations/enhancements on quality and

performance would be interesting in themselves.

See disposition to CMT/9

CAM/12 14.2.4.

Proprietary

record

creation

P. 15.22 and

throughout

te In IREX I, what are being called "records" in this document

were called "templates."

I would prefer consistency with IREX I, which would mean that

these data would be called "templates" here as well.

I would prefer consistency with IREX I, which would

mean that these data would be called "templates"

here as well.

Accept.

Replaced all instances of

proprietary record to

proprietary template.

CAM/13 14.2.4,

14.2.7

P. 15.29,

"Table 8" and

te Are input flags K1 and K3 passed to the functions described in

Tables 8 and 10? If so, how? They are not included in the

Pass input flags K1 and K3 to the functions described in

Tables 8 and 10

Accept

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 13 of 13 

1 
 

 

 

 

 

 

 

 

 

 

 

 

   

                   

               

                   

               

           

               

               

                 

                   

                 

             

             

   

                   

                 

                   

                     

                 

                 

                   

                 

                     

         

                       

                 

             

                   

       

       

         

         

     

         

       

       

     

           

   

         

         

             

       

             

     

         

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

P. 17.14,

"Table 10

"Prototype."

Iritech/1 14.2.4 Table 8 Te The proprietary record creation function in Table 8 may also Our suggestion is that it is better to force any SDKs in Reject

0 compute the quality vector during the template generation.

Our question is about the relation between the quality vector

computed during the template generation and the one

computed by the function

"compute_quality_from_image_data." A class X SDK may

support only the quality computation function without the

proprietary record creation functionality. However, a class Y

SDK should support both the proprietary record creation and

the quality computation. In this case, does NIST give the

participants the option to choose where to implement their

quality computation algorithm? Or does NIST want the

participant to implement their quality computation algorithm

at both "convert_image_to_proprietary_record" and

"compute_quality_from_image_data"?

If the participant should put their quality algorithm to both

functions, NIST need to test whether both functions produce

the same quality vector. On the other hand, if the participants

have the freedom to choose the location to put their quality

computation algorithm, NIST need to design two different test

procedures: one for the SDKs which compute the quality

vector during the template creation, and one for the SDKs

which compute the quality vector using a separate function. A

more serious problem is that if a SDK computes the quality

vector only on "convert_image_toproprietary_record" and

class X or Y to implement the separate quality

computation algorithm. And it should be explicitly

stated that the speed of IQAA will be measured for

"compute_quality_from_image."

Quality computation as part of

template (or standard record)

generation is a very relevant

operational case.

We will measure and report

computation time for

standalone IQAAs (class X) and

quality computation as part of

template generation (class Z)

separately.

To clarify class participations,

add the following to clause 8

after Table 3:

Class X SDKs will compute

quality vector from an image.

Input is an image, and output a

64‐byte quality vector. The

output of Class X SDKs will be

evaluated using comparison

scores computed by class Z

matchers.

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 14 of 14 

1 
 

 

 

 

 

 

 

 

 

 

 

 

               

                     

               

           

   

     

            

           

        

       

     

     

       

         

     

       

      

         

           

          

       

         

       

     

            

         

   

         

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

does not support a separate quality computation function,

NIST will have no way to measure the speed of quality

computation excluding the time spent for the template

generation.

Class Y is submission of quality

algorithm, a proprietary‐

template‐generator and a

matcher. A class Y matcher will

only be used to evaluate its

mated IQAA. Vendors can

submit a stand‐alone quality

measurement algorithm or

quality computation could be

part of their proprietary

template generation.

Class Z is submission of a 

proprietary template generator

(with or without quality

computation) and a matcher.

Class Z template generators and

matchers will be used for class

X IQAAs evaluation. Class Z

participants can choose to

compute quality scores as part

of their class Z proprietary‐

template‐generator, or submit

a stand‐alone IQAA as a class X.

Quality scores computed by a

Class Z proprietary‐template‐

generator will be evaluated by

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 15 of 15 

1 
 

 

 

 

 

 

 

 

 

 

 

 

     

                     

                   

             

                     

                   

       

                   

                       

             

                   

                 

               

   

         

                   

             

           

                         

                     

                 

     

               

     

                         

                     

                 

     

               

     

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

other Class Z matchers.

Iritech/1

1

14.2.4 Table 8 te The input parameter "output_type" in Table 8 shall be used to

distinguish the role of proprietary templates. The types will be

generic, verification, or enrollment. This parameter has not

Define the numeric values for output_type.

For instance, GENERIC = 0, ENROLL = 1, VERIFY = 2

Accept.

See CAM/14

been used in the previous IREX project. In IREX, NIST used

different prototypes for each type of templates as shown in

Table 11 of IREX API.

We think either approach is fine. However, current version of

IQCE API is lack of the clear definition of the values for

"output_type." The input parameter "which_eye" is clearly

defined to have the values among EYE_UNDEF = 0, EYE_RIGHT

= 1, EYE_LEFT =2. Likewise, for the input parameter

"output_type", the values for GENERIC, ENROLL, VERIFY should

be clearly defined.

CAM/14 14.2.4 Table 8

P. 15.29

What values of input parameter "output_type" denote

"generic" and "enrollment and verification"?

Define these values Accept.

See Iritech/11

Iritech/1 14.2.4 Table 8 te We advice to add some “technical” error codes to the list. It is Add “‐2 – NULL pointer” and “‐4 – bad arguments” to Accept

2 unlikely to happen, but what should the function do, if NULL the “Return values” category

pointer or invalid parameter (like unknown image format) is

passed from the caller?

Iritech/1 14.2.5 Table 9 te We advice to add some “technical” error codes to the list. It is Add “‐2 – NULL pointer” and “‐4 – bad arguments” to Accept

3 unlikely to happen, but what should the function do, if NULL the “Return values” category

pointer or invalid parameter (like unknown image format) is

passed from the caller?

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 16 of 16 

1 
 

 

 

 

 

 

 

 

 

 

 

 

         

         

                 

               

                       

                       

           

                         

                     

                 

     

               

     

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

Iritech/1

4

14.2.6 Table 10 te The prototype of the function

"compute_quality_from_image_data" has the input parameter

Remove the input parameter “allocated_bytes” from

Table 10

Accept

"allocated_byte." But it is not mentioned in the input

parameter section in the same table. Throughout the

document, the quality vector is assumed to be a vector of 32

integers. So it is unnecessary to pass the allocated bytes to this

function

Iritech/1 14.2.6 Table 10 te We advice to add some “technical” error codes to the list. It is Add “‐2 – NULL pointer” and “‐4 – bad arguments” to Accept

5 unlikely to happen, but what should the function do, if NULL the “Return values” category

pointer or invalid parameter (like unknown image format) is

passed from the caller?

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 17 of 17 

1 
 

 

 

 

 

 

 

 

 

 

 

 

                         

                       

                   

             

             

                           

                         

                     

                 

                     

   

                         

                 

                      

         

                         

       

                       

                       

         

                     

             

                 

       

             

         

                   

                 

                       

   

     

     

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

Iritech/1 9.2 Table 4 Te In Table 4, pupil iris ratio is a quality score which tells whether Modify Note 3 in 9.2 and line 13‐14 in page 13 so that Accept

7 image is good or not. However, in the second paragraph of 9.4, the pupil iris ratio is the actual ratio of pupil ratio over

pupil‐iris‐ratio seems the ratio of the pupil diameter over the iris diameter.

iris diameter (multiplied by 100 for percentage

representation). What does this pupil‐iris‐ratio actually mean?

Why pupil‐iris‐ratio = 0.5 is the

best?

If pupil iris ratio is a kind of quality score, then it should be

close to 100 when the ratio of the pupil diameter over the iris

diameter is around, for example, 0.5. So, Note 3 in 9.2 doesn’t

make any sense regardless of taking transformation. Also, in

this case, we cannot deduce the actual pupil size from this

pupil iris ratio.

If pupil iris ratio is the actual ratio of pupil diameter over the

iris diameter (multiplied by 100), this value doesn’t represent

image quality as stated in 9.2.2. In other words, neither 100

means best nor 0 does worst.

Same argument may apply to iris size. It should be a pixel size

rather than a quality score.

CMT/10 14.2.6 te Use of a dissimilarity measure to report match results may be Match score between 0 and 100 is more appropriate Reject

confusing and is not consistent with earlier NIST tests. and vendor‐neutral.

IREX I used dissimilarity score.

CAM/15 14.2.8 P. 18.6,

"Table 11"

te I don't quite understand how "get_quality_description()"

should use the input parameter "nist_assigned_identifier". Is

this actually meant to be an input rather than output

parameter? I would have thought that the SDK should never

be presented with an identifier other than its own. If it can,

Clarify. Accept in principle.

That was a mistake.

nist_assigned_identifier is not

need.

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 18 of 18 

1 
 

 

 

 

 

 

 

 

 

 

 

 

                   

         

Title: NIST proposed disposition to Comments on IREX II IQCE test plan + API v2 

Comments received: Iritech + CrossMatch(CMT) + Cambridge Univ. (CAM) + L1 + CSC (as 

annotated pdf, not included here). 

Date: 25 Nov 2009 Organization: NIST 

1 2 (3) 4 5 (6) (7)

 # Clause 

No./ 

Subclause 

No./ 

Annex 

(e.g.3, 3.1, 

Annex A, 

A.1) 

Paragraph/ 

Figure/Table/ 

Note 

(e.g. Table 1) 

Type 

of 

comment1 

Comment (justification for change) Proposed change NIST Proposed Disposition 

what return value should be given for an identifier other than

that belonging to the SDK interrogated?

Type of comment: ge = minor general Ge = major general te =minor technical Te =major technical ed = minor editorial Ed = major editorial. (Take care of Word changing te to Te etc. 

NOTE 1 Columns 1, 2, 4, 5 and 6 are compulsory. If column 4 is ge, use ge in column two also (and leave column 3 blank) if it applies to the whole document. 

NOTE 2 NBs should not change the width of columns, in order to make merging easier. 

Note 3 Please submit this document in .doc format and NOT .pdf 

page 19 of 19 

1