Iris Device Qualification Test (IDQT)

Workshop 

Dan Potter, Patrick Grother, Elham Tabassi, & Arun Vemury

July 9, 2013

NIST Workshop, July 9, 2013 1
Session Organization 

• Motivation and Purpose of Test

• Goals of this Presentation

• Review of IDQT

• Review submitted comments and editor’s 

disposition

• “Feel of the Room” for possible areas of 

document change

NIST Workshop, July 9, 2013 2
Motivation

• To develop an effective process for the 

evaluation and qualification of iris biometric 

cameras

• Fulfill the near term needs for the Air Exit and 

Entry Reengineering (AEER) project (see to 

slides and handout for more details)

NIST Workshop, July 9, 2013 3
Motivation 

NIST Workshop, July 9, 2013 4 

Iris Device 

Qualification Test 

Certified List 

for DHS Applications 
Iris Biometrics: A Complex Multivariate System 

Device Covariates

(recording optical signals)

 Spatial Frequency Response

 Throughput/Quantum 

Efficiency

 Illumination (photon noise)

 Dynamic Range and 

Resolution

 Field Distortion

 Capture Volume

• Ambient Light Mitigation

• Detector Noise

Human Subject 

Covariates

 Eye Gaze 

 Blinking/Squinting

 Pupil Dilation

 Ease of operation

 Subject motion

 Intrinsic signals (iris features, 

boarder contrasts and 

shapes, skin tones)

• Eye diseases

• Range of Pupil Dilation

• Habituation

Examples of Covariates which can influence iris Image Quality

Environmental 

Covariates

 Ambient Illumination

 Vibration

 Temperature/Humidity

 Sound environment

Device Covariates

(Human Factors Control)

 Gaze attractor

 Pupil dilation control

 Eyelid occlusion filter/ control

 Subject Motion Control

 Physical ergonimics of Device

 Software interface

Human Operator 

Covariates

• Past Experience with device

• Mental abilities

• Physical abilities

CONOPS

NIST Workshop, July 9, 2013 5
NIST Workshop, July 9, 2013 6

CONOPS
IDQT Rationale: Divide the Problem 

Human Subject 

Covariates 

 Eye Gaze 

 Blinking/Squinting 

 Pupil Dilation 

 Ease of operation 

 Subject motion 

 Intrinsic signals (iris features, 

boarder contrasts and 

shapes, skintones) 

• Eye diseases 

• Range of Pupil Dilation 

• Habituation 

Human Interaction aspects tested with Humans, not in the IDQT 

Environmental 

Covariates 

 Ambient Illumination 

 Vibration 

 Temperature/Humidity 

 Sound environment 

Device Covariates 

(Human Factors Control) 

 Gaze attractor 

 Pupil dilation control 

 Eyelid occlusion filter/ control 

 Subject Motion Control 

 Physical ergonimics of Device 

 Software interface 

Human Operator 

Covariates 

• Past Experience with device 

• Mental abilities 

• Physical abilities 

CONOPS 

IDQT CONSIDERS ASPECTS OF QUALITY 

INDEPENDENT OF HUMAN INTERACTION 

Device Covariates 

(recording optical signals) 

 Spatial Frequency Response 

 Throughput/Quantum 

Efficiency 

 Illumination (photon noise) 

 Dynamic Range and 

Resolution 

 Depth of Field 

 Capture Volume 

• Ambient Light Mitigation 

• Detector Noise 

NIST Workshop, July 9, 2013 7 
DHS Evaluation Process 

Market	Survey	

Device	Qualification	

Human-in-the	loop	Laboratory	

Performance	Qualification	

Pilot	Integration	

Field	trials	

Final	Integration	and	Deployment	

Human interaction issues 

are evaluated just on the 

devices which pass the IDQT 

IDQT acts as a filter so time is not wasted 

evaluating devices with human subjects 

NIST Workshop, July 9, 2013 8 
Project Goals

Develop “Appendix F-like” iris device qualification 

testing tools and procedures which:

1. Minimize biases between devices

2. Minimize modification to intended device operation on 

real human subjects 

3. Measure “peak” imaging performance... degradation 

from realistic operations should be revealed in 

subsequent evaluation stages

4. Should be simple enough to be practically conducted by 

a third party testing facility

NIST Workshop, July 9, 2013 9
Goals of this Session

• Present Overview of IDQT

• Point out areas of possible change to draft based on 

comments from industry

• Review received comments and editor’s disposition

• Discuss possible changes, get the “feel of the room”

– No contention

– Acceptable, but could be improved

– No acceptable, introduces significant bias and or would 

produce severely misleading guidance

NIST Workshop, July 9, 2013 10
Development Components

Face Foundation

 Passes face recognition 

requirements of capture devices

 Mimics light reflection from 

human skin

• Accurate, precise optical mount 

for eye targets

11

Targets/Algorithms

 Passes “eye-ness” 

requirements of capture 

devices

 Contains known patterns used 

for diagnostic measurements

 Mounts into face foundation 

Test Plan/Reporting

 Well documented procedure to 

validate test targets, collect and 

analyze data

 Standardized output of results 

for meaningful inter-device 

comparisons

Radius

Theta

NIST Workshop, July 9, 2013
Overview of Metrics Recorded in IDQT 

NIST Workshop, July 9, 2013 12

IDQT Image Quality Measurements 

1. Spatial Frequency Response 

2. Iris-like Feature SNR 

1. Pixel Scale (all targets) 

2. Greyscale Linearity 

3. Greyscale Resolution 

4. Field Distortion 

IDQT Device Characterization 

1. Illumination: Eye Safety 

2. Cornea Reflection Mitigation: Ambient 

Scene 

3. Cornea Reflection Mitigation: 

Instrument only 

1. Illumination: Wavelength 

Characterization 

2. Exposure Time Estimation 

Qualification Criteria 

Used in 

‘root cause’

estimation 

Mobile ID 

Guideline 

Environment 

Categories 
Rationale for Qualification Criteria 

Complicated approach: 

Assign individual criteria for a list of individual 

and combinations of metrics. Requires extensive 

controlled studies correlating individual metrics. 

More practical ‘bottom line’ approach:

Characterize signal used in iris biometrics, 

reproduce signal in static targets, encode and 

match features like commercial algorithms to 

define quality metric 

NIST Workshop, July 9, 2013 13
IDQT Face Design 

NIST Workshop, July 9, 

2013

14 3-D Printed Model 

Front surface 

“average” 3-D face 

IPD=63mm (average) 

Eyeball mount 

for iris targets 

Back surface 

accommodates 

 eyeball mounting 
Face Material Study: 

Search for skin-like NIR BRDF 

Near Infrared Image of Material Samples NIST Workshop, July 9, 2013 15

Goniospectrophotometer 
Corneal Reflections 

NIST Workshop, July 9, 2013 16

Primary Illuminator Reflection 

Ambient Light Reflection 

Nose 

Eye 

Light 

Nose Reflection 
Face Discussion Issues

Suggested IDQT uses average characteristics of 

face morphology and skin tone.

COMMENTS:

- Argument to incorporate multiple faces with 

different morphologies to explore extremes of 

scale (Include children and large end outliers) 

- Argument to incorporate multiple skin tones 

(e.g. test for face detection failure) 

NIST Workshop, July 9, 2013 17
NIST Workshop, July 9, 2013 18

Incident Light 

Cornea surface (Strict Fresnel) 

• Reflects ~2-3% of incident 

NIR light 

• Fish-eye de-magnification 

• Polarized 

Iris surface 

• Scatters ~10-16% of 

incident NIR Light 

• Lambertian? 

Reflected Light 

n1=1.00 

(AIR) 
Eye target Development 

NIST Workshop, July 9, 2013 19

• Diameter of iris piece=11.8mm, ball diameter=25mm 

• Lens surface provides cornea-like reflection 

 (calibrated to real human examples) 

• Index match on opaque backside for minimal back 

 surface reflection 

• Front Lens Radius of Curvature = 7.85mm 

 (human cornea is aspheric, ranges from ~7-8mm) 
Rationale: Capture Optical Traits of 

Human Eye 

0.29 0.20 

0.13 

0.16 

20

Typical CTF Target 

0.98 

0.02 

Brown Eye 

Reflectivity Numbers Overlaid 
Iris Signal Characterization 

NIST Workshop, July 9, 2013 

21

What is there: What is needed by matching algorithms? 

Features < 10 microns Features 0.2 - 2 millimeters? 
NIST Workshop, July 9, 2013 

Observed Optical Properties of the Iris: Spatially 

Varying Albedo 

Signal-to-Noise Ratio can be expressed as a function of device variables 

(assuming photon noise): 

Iris 

Texture

0.13 

0.16 
Contrast Decrease with Smaller Scale

NIST Workshop, July 9, 2013 23

Rough Fit: A(k)=Cs(k

2

)

-11/3

Iris albedo texture seems to follow a

power law distribution...

Add characteristic inner and outer scales: 

A(k)=C1(k

2+ ko2 )-11/6 exp(-k2/ki2

) ) (1-c2(k/ki))

(“bump” around 0.3mm)

Similar to 3-D Kolmogorov Turbulence Structure 

NIR Iris 3-D Kolmogorov
Target Pattern Creation 

Utilization of Carbon-based Ink with High Resolution Inkjet Printers 

BCC, September 19, 2012 24

Commercial Grade 

Star Target Carbon-based Inkjet 

Star Target 

CTF Comparisons 

16 Bit Value- Albedo Calibration 

Reflectance Standards 

Printer Output 

(16-bit Dynamic Range) 

Eye Reference 
Target Overview: Star Pattern 

MTF (Primary) 

• 60 segments : 0.8 – 3.5 lp/mm 

• 120 segments : 1.6 - 6.5 lp/mm

• Large Areas at Frequencies = 1, 2,& 3 lp/mm

• Theta variations noted versus target rotations, 

• Average over theta at given R used for Qualification Criteria 

NIST Workshop, July 9, 2013 25

13%

17%
Straightforward CTF 

+ Alternate encoder based metric 

NIST Workshop, July 9, 2013 26

Segment 

Unwrap 

XOR

Generate 

Raw Hamming Distance 

Score Elements 

Reference Collected Radius 

Theta 

Collected 

Target 

Image 

Create Binary Template 

= 

Can extract CTF here 
Target Overview: Quadrant Pattern 

• Gain Linearity 

• MTF (secondary) 

• Dynamic range resolution: 

 D Albedo__ _ 

 D greyscale increment 

• “Conventional” SNR

 in each uniform region 

 

NIST Workshop, July 9, 2013 27

8%

14%

20%

26%
Analysis Method: Detector Linearity 

• Fit line to linear model, 

statistical analysis on 

errors 

• Check systematics 

(specular reflections) by 

rotating target via test 

protocol 

a=0.26 

a=0.20 

a=0.14 

a=0.08 

NIST Workshop, July 9, 2013 28
Analysis Method: Contrast SNR 

• Establish Distribution Type (e.g. Gaussian) 

• Calculate Standard Deviation versus cell size and albedo 

• Use (hopefully) Gaussian Statistics for simplicity (i.e. 1,2,3.. Sigma Vs. feature type) 

NIST Workshop, July 9, 2013 29
Slanted edge MTF extraction (secondary) 

ISO 12233 slanted edge test 

= 

Gradient Amp. 

Rotate 1d fft NIST Workshop, July 9, 2013 30
Target Overview: Uniform Dark 

For Illumination Characterization 

• Device Illumination pattern 

– Nose/eye socket reflections 

– Primary Corneal reflection pattern 

(any overlap with iris?) 

• Ambient light Mitigation 

NIST Workshop, July 9, 2013 31

8%
Target Overview: Distortion Grid 

Used to map field distortion 

• Stated in object plane Cartesian 

coordinates 

• Measurements relative to pupil 

center coordinates with average 

pixel scale from limbus radius 

• Grid of error values relative to 

perfect model 

NIST Workshop, July 9, 2013 32

1.3mm 
Target Overview: Iris Feature Spectrum 

 

• Calibrated to have average 

albedo of ~0.16 at 800nm 

• -11/3 feature spectrum 

• A bit more power in theta 

NIST Workshop, July 9, 2013 33
Encoding Example 

Psuedo-Polar Normaized 

Encoded signal (3 Haar filters varying Spatial Freq. to make cube) 

IBPC Conference: March 6, 2012 
Binary Encoder/ HD Metric 

NIST Workshop, July 9, 2013 35

Normalized Image (PRISTINE) 

Normalized Image (Collected) 

Template (PRISTINE) 

Template (Collected) 

XOR RESULT 

High Frequency Middle Frequency 
Discussion: Target Patterns

• Any obvious sources of biases? 

• Realism of the Iris Texture Target – Method of the 

-11/3 power law 

• Definition of “pristine” template

• Is one iris texture target enough?

• No explicit measurement of the Phase Transfer 

Function

• No 3-D surface topology taken into account –

(illumination angle matters)

NIST Workshop, July 9, 2013 36
Other Measurements

• Exposure Time Test

– Ring of fiber fed LEDs mounted in eye target, blinking 

in series with 5ms pulses. Exposure time is estimated 

by the number of lit fibers seen in an image.

• Eye Safety 

– Calibrated Irradiance meter (1 KHz large area photodiode) embedded in eye target

• Wavelength Characterization

– Multiple captures with fiber fed USB spectrometer 

with probe mounted in eye target

NIST Workshop, July 9, 2013 37
Discussion: Other Measurements

• Wavelength Guidelines in NIST Mobile ID Best 

practices Document is not backed by available 

study

• Wavelengths used may be trade secret

• Reference to use for Eye Safety

• No allowance for wavelengths other than 

700nm-900nm

NIST Workshop, July 9, 2013 38
Best Case MTF with Typical Sampling

NIST Workshop, July 9, 2013 39

Examples from Best Case Diffraction Limited Conventional Optics (No Deconvolution)

50% mod @ ~2.5 lp/mm 50% mod @ ~5 lp/mm
MTF “Controlled” Study

NIST Workshop, July 9, 2013 40

Degraded University of Bath Images: 

• Convolution with Gaussian and Airy Function Blur Kernels 

• Scaled relative to iris diameter 

• 30 samples, ranging from FWHM ~ 0.07mm – 1.3mm (~50%@ 6 - 1/3 lp/mm) 

Sharp Blurred 
Blackbox Results from NIST 

NIST Workshop, July 9, 2013 41

Enrolment MTF

Verification MTF

(1000,1001]

(1001,1002]

(1002,1003]

(1003,1004]

(1004,1005]

(1005,1006]

(1006,1007]

(1007,1008]

(1008,1009]

(1009,1010]

(1010,1011]

(1011,1012]

(1012,1013]

(1013,1014]

(1014,1015]

(1015,1016]

(1016,1017]

(1017,1018]

(1018,1019]

(1019,1020]

(1020,1021]

(1021,1022]

(1022,1023]

(1023,1024]

(1024,1025]

(1025,1026]

(1026,1027]

(1027,1028]

(1028,1029]

(1000,1001](1001,1002](1002,1003](1003,1004](1004,1005](1005,1006](1006,1007](1007,1008](1008,1009](1009,1010](1010,1011](1011,1012](1012,1013](1013,1014](1014,1015](1015,1016](1016,1017](1017,1018](1018,1019](1019,1020](1020,1021](1021,1022](1022,1023](1023,1024](1024,1025](1025,1026](1026,1027](1027,1028](1028,1029]

0.0

0.2

0.4

0.6

0.8

Enrolment MTF

Verification MTF

(1000,1001]

(1001,1002]

(1002,1003]

(1003,1004]

(1004,1005]

(1005,1006]

(1006,1007]

(1007,1008]

(1008,1009]

(1009,1010]

(1010,1011]

(1011,1012]

(1012,1013]

(1013,1014]

(1014,1015]

(1015,1016]

(1016,1017]

(1017,1018]

(1018,1019]

(1019,1020]

(1020,1021]

(1021,1022]

(1022,1023]

(1023,1024]

(1024,1025]

(1025,1026]

(1026,1027]

(1027,1028]

(1028,1029]

(1000,1001](1001,1002](1002,1003](1003,1004](1004,1005](1005,1006](1006,1007](1007,1008](1008,1009](1009,1010](1010,1011](1011,1012](1012,1013](1013,1014](1014,1015](1015,1016](1016,1017](1017,1018](1018,1019](1019,1020](1020,1021](1021,1022](1022,1023](1023,1024](1024,1025](1025,1026](1026,1027](1027,1028](1028,1029]

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Enrolment MTF

Verification MTF

(1000,1001]

(1001,1002]

(1002,1003]

(1003,1004]

(1004,1005]

(1005,1006]

(1006,1007]

(1007,1008]

(1008,1009]

(1009,1010]

(1010,1011]

(1011,1012]

(1012,1013]

(1013,1014]

(1014,1015]

(1015,1016]

(1016,1017]

(1017,1018]

(1018,1019]

(1019,1020]

(1020,1021]

(1021,1022]

(1022,1023]

(1023,1024]

(1024,1025]

(1025,1026]

(1026,1027]

(1027,1028]

(1028,1029]

(1000,1001](1001,1002](1002,1003](1003,1004](1004,1005](1005,1006](1006,1007](1007,1008](1008,1009](1009,1010](1010,1011](1011,1012](1012,1013](1013,1014](1014,1015](1015,1016](1016,1017](1017,1018](1018,1019](1019,1020](1020,1021](1021,1022](1022,1023](1023,1024](1024,1025](1025,1026](1026,1027](1027,1028](1028,1029]

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

1lp/mm 

2lp/mm 

3lp/mm 

1lp/mm 

2lp/mm 

3lp/mm 

1lp/mm 

2lp/mm 

3lp/mm 

Arrows indicate rough 50% modulation at 1,2,3 lp/mm 

Color indicates score averages in bins (green match, blue mismatch) 
3 Qualification Levels 

Level 1 (Opens up applications for Small N, 1-1)

• Measured MTF of 50% at 1 lp/mm using the IDQT targets.

• HD of 0.1 or less using 0.75mm feature encoders to the pristine reference 

template for at least 95% of the collected images, >90% pass mask

Level 2 (Similar to old guideline, suitable for large N)

• Must pass level 1, and 50% modulation @2 lp/mm, feature size of 0.38mm.

Level 3 (Placeholder for Future*, indicates very high SNR for level 1 

and 2 feature sizes)

• Must pass level 1,2, and 50% mod @3 lp/mm, feature size of 0.25mm.

NOTE: Other metrics still reported, and used to assess the potential root 

cause of a possible failure. All levels must be eye safe.

*studies not published, still we have confidence that information density is high at 0.25mm scales.

NIST Workshop, July 9, 2013 42
Three Ambient Light Levels 

Proof of concept finished 

Process currently being worked 

out to make this test practical... 

Contrast structure of 

surrounding scene structure 

makes a difference. NIST Workshop, July 9, 2013 43
Other tests possible (but needed?): 

Influence of photon Noise

Bright illumination

Dim illumination

High Frequency Low Frequency

NIST Workshop, July 9, 2013 44
List of “nice to have” studies

• Ultimate: large, diverse human subject collection 

with multiple devices, multiple wavelengths, and 

manually controlled device to enable global 

exploration of all likely important device related 

covariates

• Multi-wavelength data collection with many 

narrowband samples within the 700-900 nm 

region for meaningful interoperability guideline

• Effect of illumination angle: 3-D structures

NIST Workshop, July 9, 2013 45
Discussion: Qualification Criteria

• Should Qualification include specific criteria 

on more than iris feature spectrum and MTF 

targets?

• Are the 3 levels 1,2,3 lp/mm too closely 

spaced in spatial frequency response, to 

broad?

• Why chose 0.1 for the Hamming distance 

criteria?

NIST Workshop, July 9, 2013 46
Acknowledgements

Work Supported by DHS S&T...

NIST Workshop, July 9, 2013 47