1	

IREX	II IQCE

Performance evaluation of iris quality measures

Elham Tabassi

Informa0on Technology Laboratory

Na0onal Ins0tute of Standards and Technology

US Department	of Commerce

September 28, 2011

BCC :: NIST session
NIST%Iris%Exchange%(IREX)%program%%

http://iris.nist.gov/irex 

IREX%I%

Δ!!Formats,!cropping,!masking!

Δ!!Compression!limits!

Δ!!Geometry,!Margins,!Radius!

Δ!!Dilation,!concentricity!

Δ!!Concluded!mid!2009!

Δ!!Supported!ISO/IEC!19794E6!

IREX%III%

Δ!!1:N!with!N!in!the!millions!

Δ!!One!and!two!eyes!

Δ!!Cross!camera!interop.!

Δ!!Timeline!

Δ!!!!!!!!Started!February!2011!

Δ!!!!!!!!Report!Fall!2011!

IREX%II%%(IQCE)%

Δ!!Iris!Image!Quality!

Δ!!!!!!!Definition!

Δ!!!!!!!Evaluation!

Δ!!!!!!!Calibration!

Δ!!!Supporting!ISO/IEC!29794E6!

Δ!!!Evaluation!Report!NIST!IR!909622!!

2&
Iris%Quality%Calibration%and%Evaluation%(IQCE)%is%...%

⨱

⨱

The%2nd%activity%under%%%

⨱

3&

Funded%by%DHS%S+T%

– Project!“Radical!improvement!in!iris!quality!assessment!and!maturing!

multimodal!biometric!utilization”!

An%evaluation%based%program%for%development%of%clear,%

implementable,%and%%interoperable%iris%quality%standard%ISO/IEC%

29694Q6.

- To!establish!requirements!on!software!or!hardware!capturing!iris!image!

⨱ A!refined!list!of!image!properties!affecting!iris!recognition!performance!

- To!established!requirements!on!iris!image!covariates
4&

Outcomes)

Evaluation%Report%

NIST%IR%7820%

http://www.nist.gov/itl/iad/ig/irexii.cfm%

Iris%Quality%Reading%%

http://www.nist.gov/itl/iad/ig

iris_image_qual_reading.cfm%
Motivation%::%Failure%mode%analysis%

Identifying%samples%that%contribute%to%recognition%error

Image!false!match!rate!(iFMR)!

Image!false!nonmatch!rate!(iFNMR)!

blue%goat%

black%ice%

clear%ice% blue%wolf%

Elevated!false!nonEmatch!

Elevated!false!match!

Elevated!false!nonEmatch!!

AND!false!match!

E.!Tabassi,!Image!specific!error!rate:!A!biometric!performance!metric,!ICPR,!pp.1124E1127,!2010!20th!Intl.!Conf.!on!Pattern!

Recognition,!2010.! 5!
IQCE%%

Cropped!

raw!image!! VENDOR SUPPLIED IMAGE

QUALITY ASSESSMENT

ALGORITHM (IQAA) 

QUALITY

VECTOR

1 Scalar quality 

2 .. 17 Defined (standard) quality 

metrics 

18..32 Reserved 

33..64 Vendor-defined quality 

metrics 

%Q%

FNMR%

6&
IQCE%submissions%

15 To keep the number of graphs somehow manageable, results for only one submission per organization is included 

16 this document. If multiple submissions from an organization, the IQAA with the better performance was chosen. Th

17 are referred to as the “primary” submissions. Results for the submissions that are not included here are available up

18 request.

19 The detection error trade-off (DET ) curves of Figure 1. show the accuracy of the IQCE comparators. Table 5 summari

20 the role and functionality of IQCE submissions.

No. Q Component IQAA submissions

1 scalar quality A2a A2f B3 C4x C4s C4zf D3 E2a E2f F1 G1 G2 Hx I1

2 gray level spread A2a A2f D3 E2a E2f F1 G1 G2 Hx I1

3 iris size A2a A2f C4x C4s C4zf D3 E2a E2f I1

4 pupil iris ratio A2a A2f B3 C4x C4s C4zf D3 E2a E2f F1 G1 G2 Hx I1

5 usable iris B3 C4x C4s C4zf D3 E2a E2f F1 G1 G2 Hx I1

6 iris-sclera contrast A2a A2f C4x C4s C4zf D3 E2a E2f F1 G1 G2 Hx I1

7 iris-pupil contrast A2a A2f C4x C4s C4zf D3 E2a E2f F1 G1 G2 Hx I1

8 iris shape A2a C4x C4s C4zf D3 E2a E2f Hx I1

9 pupil shape A2f C4x C4s C4zf D3 E2a E2f F1 Hx

10 margin A2a A2f D3 E2a E2f

11 sharpness A2a A2f B3 C4x C4s C4zf D3 E2a E2f F1 G1 G2 Hx

12 motion blur E2a E2f

13 signal to noise ratio A2a A2f D3 E2a E2f F1 G1 G2 I1

14 magnification

15 head rotation

16 gaze angle C4x C4s C4zf D3 E2a E2f Hx I1

17 interlace A2a A2f C4x C4s C4zf D3 E2a E2f Hx

Table 4: IQCE IQAA submissions, support for specific quality component measurement

No. Functionality Submissions

1 comparison algorithm A2a B3 C4s C4zf E2a E2f G1 G2 Hz1 Hz2

2 quality at template generation A2a B3 C4s C4zf E2a E2f G1 G2

3 stand alone quality A2f C4x D3 F1 Hx 

Table 5: Functionality of IQCE submissions.

7&
Data%

8&

dataset) origin) #sub #images) camera) Level)of)

ID) jects) difficulty)

OPS) Operational% 8160% 32640% Pier%2.3% easy%

ICE) Extract%of%ND% 193% 56871% LG%Iris% medium%

2005Q2006% Access%

ICE% 2200%

QFIRE) Extract%of% 136% 14165% Dalsa hard%

I05) Clarkson%U.% (blur,%

QFIRE%% illumination%

intensity)%

QFIRE) Extract%of% 135% 4365% Dalsa hard%

A05) Clarkson%U.% (offQaxis%

QFIRE%% gaze)%
Evaluation%::%metrics%

∸ Relationship%of%quality%scores%with%genuine%distribution%

∸ Relationship%of%quality%scores%with%impostor%distribution%

∸ Improvement%in%false%nonQmatch%as%poor%quality%samples%

are%rejected%

– ERC%(Error%vs%Reject%Characteristic)%Curves

∸ Recognition%performance%of%low,%mid%and%high%quality%

samples%

– Ranked%DET%(Detection%Error%Tradeoff%Characteristic)%curves%

∸ Effect%of%difference%in%quality%of%the%two%samples%being%

compared.%

– Heatmaps%(surface%plots)%%of%normalized%FNMR%

∸ Test%of%Significance:%If%the%mean%of%images%which%caused%

recognition%error%are%significantly%different%from%others%

∸ Tukey%HSD%test%of%significance%
Evaluation%::%Relationship%of%quality%scores%with%

genuine%or%impostor%%distribution

10&
Effect%of%quality%on%performance%::%ranked%DET%
Effect%of%quality%on%performance%::%ranked%DET%
How%variation%of%quality%between%two%images%being%compared%

affects%performance?Q1

Verification :: scalar quality enrolment quality :: iris size 

enrolment :: scalar quality 
How%variation%of%quality%between%two%images%being%compared%

affects%performance?Q2

Verification :: iris-pupil ratio (dilation) enrolment quality :: iris size 

enrolment!::!irisEpupil!ratio!(dilation)!!
Correlation%of%quality%components%

IQAA E2f ! Comparator E2f ! Dataset ICE2006

pupil.iris.ratio

iris.sclera.contrast

sharpness

scalar.quality

usable.iris.area

log.odds

gray.level.spread

iris.shape

motion.blur

interlace

pupil.shape

gaze.angle

iris.size

signal.to.noise.ratio

iris.pupil.contrast

margin

pupil.iris.ratio

iris.sclera.contrast

sharpness

scalar.quality

usable.iris.area

log.odds

gray.level.spread

iris.shape

motion.blur

interlace

pupil.shape

gaze.angle

iris.size

signal.to.noise.ratio

iris.pupil.contrast

margin

100 31 9 30 19 !16 !14 0 0 !5 !25 !15 !15 !16 7 4

31 100 !3 68 24 !12 19 1 1 !5 !13 !10 !1 2 !35 1

9 !3 100 41 16 1 8 1 8 !12 4 !2 !25 !11 22 8

30 68 41 100 69 !18 11 !7 4 !19 16 15 !8 5 !7 6

19 24 16 69 100 !18 !5 !18 1 !21 8 !1 !4 7 0 5

!16 !12 1 !18 !18 100 9 !1 !2 4 !7 !2 !15 !7 !6 0

!14 19 8 11 !5 9 100 1 2 6 !16 !10 5 8 !23 3

0 1 1 !7 !18 !1 1 100 0 11 !1 4 !4 0 2 !2

0 1 8 4 1 !2 2 0 100 16 0 !1 !2 1 5 3

!5 !5 !12 !19 !21 4 6 11 16 100 !7 1 2 !16 !21 !5

!25 !13 4 16 8 !7 !16 !1 0 !7 100 83 10 8 15 2

!15 !10 !2 15 !1 !2 !10 4 !1 1 83 100 8 1 9 !1

!15 !1 !25 !8 !4 !15 5 !4 !2 2 10 8 100 21 10 !5

!16 2 !11 5 7 !7 8 0 1 !16 8 1 21 100 25 7

7 !35 22 !7 0 !6 !23 2 5 !21 15 9 10 25 100 12

4 1 8 6 5 0 3 !2 3 !5 2 !1 !5 7 12 100

!1.0 !0.5 0.0 0.5 1.0

IQAA E2f ! Comparator E2f ! Dataset QFIREI5

iris.sclera.contrast

iris.pupil.contrast

margin

gaze.angle

iris.shape

interlace

motion.blur

scalar.quality

sharpness

pupil.shape

iris.size

usable.iris.area

gray.level.spread

pupil.iris.ratio

log.odds

signal.to.noise.ratio

iris.sclera.contrast

iris.pupil.contrast

margin

gaze.angle

iris.shape

interlace

motion.blur

scalar.quality

sharpness

pupil.shape

iris.size

usable.iris.area

gray.level.spread

pupil.iris.ratio

log.odds

signal.to.noise.ratio

100 93 81 89 88 86 55 85 57 !83 !80 !83 !88 !81 !3 0

93 100 78 88 87 84 54 86 60 !80 !77 !81 !86 !79 !1 !1

81 78 100 90 91 89 59 59 51 !95 !92 !94 !89 !89 1 2

89 88 90 100 93 94 60 81 65 !85 !90 !90 !95 !93 !1 !6

88 87 91 93 100 92 59 72 56 !91 !90 !91 !94 !91 !1 !1

86 84 89 94 92 100 59 72 59 !87 !88 !89 !96 !91 !1 !5

55 54 59 60 59 59 100 44 37 !58 !59 !59 !60 !58 1 8

85 86 59 81 72 72 44 100 62 !55 !57 !60 !75 !67 !4 !9

57 60 51 65 56 59 37 62 100 !44 !50 !53 !60 !56 3 !33

!83 !80 !95 !85 !91 !87 !58 !55 !44 100 95 95 89 86 !1 !7

!80 !77 !92 !90 !90 !88 !59 !57 !50 95 100 94 90 90 !1 !2

!83 !81 !94 !90 !91 !89 !59 !60 !53 95 94 100 90 89 !3 !1

!88 !86 !89 !95 !94 !96 !60 !75 !60 89 90 90 100 92 0 4

!81 !79 !89 !93 !91 !91 !58 !67 !56 86 90 89 92 100 0 4

!3 !1 1 !1 !1 !1 1 !4 3 !1 !1 !3 0 0 100 !4

0 !1 2 !6 !1 !5 8 !9 !33 !7 !2 !1 4 4 !4 100

!1.0 !0.5 0.0 0.5 1.0

15&
Evaluation%::%computation%efficiency%

quality!computation!time!(msec)!

10! 50! 100! 500! 1000!

Video

frame%

rate%

process.%

At%capture%

event%

Too%

late!!%

16&
Iris%Quality%Calibration%and%Evaluation%(IQCE)%

Summary!Results

17&

http://iris.nist.gov/irex 
Large%scale%evaluation%of%IQAAs%%

⨠ Comparative%analysis%of%core%algorithmic%capability%::%their%

ability%to%predict%recognition%performance%

≫ Evaluated%14%IQAAs%from%major%iris%technology%providers%%

⨠ Comparative%analysis%of%their%computational%efficiency%%

- Quality%computation%time%across%datasets%varies,%but%the%order%of%

IQAAs%stays%the%same.%

- Across%IQAAs%quality%computation%time%varies%by%two%order%of%

magnitudes.%The%fastest%have%an%average%30%millisecond%per%image%

across%all%four%datasets.%

⨠ Remarks%on%their%robustness%

≫ Interoperability%of%IQAA%and%iris%recognition%algorithms%

≫ Do%IQAAs%perform%better%for%their%native%iris%recognition%algorithms?%

+%Yes%for%some,%No%for%others.%

One%sentence%takeQaway%

Quality%computation%can%improve%the%performance.% 18&
Effect%of%iris%image%properties%on%performance%%

≫ Examined%14%image%covariates%(quality%components)%

≫ Vector%vs.%scalar%quality%

+%A%vector%of%quality%components%allows%for%finer%failure%analysis%and%

actionable%feedback%

Q Requires%calculation%of%a%summary%statistic%of%the%components%(e.g.%

to%prompt%for%Accept%or%Recapture)%

≫ Some%quality%components%influence%recognition%error%rates%

more%than%others.%

Q Usable%iris%area,%iris%pupil%contrast%,%pupil%shape%,%iris%sclera%contrast%,%

gaze%angle%and%sharpness%

Q IQCE%results%for%motion%blur%and%signal%to%noise%ratio%are%

inconclusive%

≫ For%some%quality%components%“sameness”%matters%

Q Iris%size,%pupil%iris%ratio%(dilation)%

19&
Recommendation%for%ISO/IEC%29794Q6%

to%be%submitted%to%ISO/IEC%JTC%1%SC%37%&WG3&

≫ A%refined%list%of%quality%components%%

≫ given%the%lack%of%implementation%support%for%head%rotation%and%

magnification%IQCE%recommends%exclusion%of%these%two%from%ISO/IEC%

29794Q6.

≫ signal%to%noise%ratio%is%a%capture%device%characteristic.%Methodology%and%

metrics%for%measurement%of%sensor%noise%is%needed.%

≫ Results%for%motion%blur%was%inconclusive.%

≫ Recommendation%on%iris%size%and%gray%level%spread%

≫ Iris%radius%>=%60%pixels%

≫ Entropy%of%pixel%intensity%values%>=%6%bits%

20&
Evaluation%::%core%algorithmic%capability%

21&
Ongoing

≫ Calibration%of%quality%scores%

≫ Evaluation%of%proprietary%quality%components%

≫ Repeat%on%(larger)%datasets%that%are%more%diverse%in%quality%

≫ Effect%of%eyewear%(eye%glass%or%contact%lenses)%on%

performance%

≫ Effect%of%illumination%intensity%

22&

≫ Repeat%on%larger%dataset%with%diverse%quality%%

≫ more%subjects%

Future
Upcoming%event%::%IBPC%2012%

23&

"#$%&'#(!)'*$%$+$,!&-

.$#'/#0/*!#'/!1,23'&(&45

)678!9:;9! 6)<=>1?)8!7>?@<?=A"8>!

1>.1)"B!8<"@>?>"8>

=#023!CDEF!9:;9

8#((!-&0!7#G,0*

").1F!"7H!#'/!@0#+'3&-,0!)BI!#''&+'2,!$3,!*,2&'/!%'$,0'#$%&'#(!2&'-,0,'2,!&'!G,0-&0J#'2,!#'/!$,*$%'4!&-!K%&J,$0%2!*5*$,J*L!!13,!

%'$,'$!%*!$&!/,$#%(!$3,!(#$,*$!/,M,(&GJ,'$*!%'!!"#!*5*$,J*!#0,!K,%'4!$,*$,/!#'/!2,0$%-%,/F!0#$3,0!$3#'!&'!$3,!(#$,*$!$,*$%'4!0,*+($*L!!13,!

-&0+J!N%((!K0%'4!$&4,$3,0!,M#(+#$&0*F!+*,0*F!$,23'&(&45!G0&M%/,0*!$&!/%*2+**!G,0-&0J#'2,D0,(#$,/!%**+,*!%'!#'5!#GG(%2#$%&'*!#'/!G0&40#J*!

$3#$!,JK,/!#!K%&J,$0%2!-+'2$%&'!&0!2&JG&','$L!!!13,!2&'-,0,'2,!N%((!#//0,**!N3#$!G,0-&0J#'2,!20%$,0%#!#0,!K,%'4!*G,2%-%,/!%'!

G0&2+0,J,'$!#'/!*,(,2$%&'!G0&2,**,*L!!13,!$#(O*!N%((!0,*G,2$!&G,0#$%&'#(!',,/*F!#'/!N%((!4%M,!,JG3#*%*!$&!N3#$!&G,0#$&0*!0,P+%0,!-0&J!

$,*$!#'/!,M#(+#$%&'!#2$%M%$%,*L!!13,!2&'-,0,'2,!#%J*!$&!%/,'$%-5!3&N!$3,!#22+J+(#$,/!,QG,0%,'2,!&-!$3,!(#*$!/,2#/,!R0,*,#023F!,M#(+#$%&'F!

/,G(&5J,'$F!&+$2&J,*S!2#'!K,!(,M,0#4,/!$&!/%0,2$!-+$+0,!K%&J,$0%2*DK#*,/!#GG(%2#$%&'*L!!13,!2&'-,0,'2,!N%((!#(*&!#//0,**!$3,!0,P+%0,/!

G0&G,0$%,*!&-!2&0,!#(4&0%$3J*!-&0!&G,0#$%&'#(!K%&J,$0%2*L

13,!2&'-,0,'2,!#%J*!$&!%/,'$%-5!'&M,(!,M#(+#$%&'!J,$3&/&(&4%,*!#'/!0,2,'$!$0,'/*!%'!$,*$%'4F!#'/!$&!/,$,0J%',!N3#$!%*!J&*$!

&G,0#$%&'#((5!0,(,M#'$!%'!$3,!2&'$,Q$!&-!$3,!2&'$,JG&0#05!#'/!,J,04%'4!J#0O,$G(#2,*L

!"#$"%&'(#&&)**++,

7#$0%2O!B0&$3,0F!>(3#J!1#K#**%F!").1F!T.

1&'5!=#'*-%,(/F!"7HF!TU

830%*$&G3!6+*23F!@0#+'3o-,0!)BIF!I>

-.*+./+/'01+%2+"0,

T*,0*F!,M#(+#$&0*F!0,*,#023!#'/!/,M,(&GJ,'$!*$#--F!

*5*$,J!#'#(5*$*F!$3,!N0%$,0*!&-!$,23'%2#(!

*G,2%-%2#$%&'*F!*$#'/#0/*!/,M,(&G,0*!#'/!#/&G$,0*L

3%"$+*'%4/)+.5+,!!

70&-,**%&'#(*!2&'2,0',/!N%$3!K%&J,$0%2!*5*$,J!

,M#(+#$%&'F!G0&2+0,J,'$F!/,G(&5J,'$F!J#%'$,'#'2,F!

/,*%4'F!2&'-%4+0#$%&'F!%'$,40#$%&'F!*$#'/#0/%V#$%&'F!

0,*,#023!#'/!/,M,(&GJ,'$L

-&1#"*%.*'/%*+0,

;;W:;W9:;;!D!.+KJ%**%&'!&-!#K*$0#2$*

;;W99W9:;;!D!"&$%-%2#$%&'!&-!#22,G$#'2,

:9W9XW9:;9!D!.+KJ%**%&'!&-!G0,*,'$#$%&'

:9W9XW9:;9!D!@%'#(!2&G5!-&0!70&2,,/%'4*

:YW:CW9:;9!D!8&'-,0,'2,!*$#0$

6#$)0*)50,

Z3,'[! ! 8&'-,0,'2,!=#023!\D]F!9:;9

! ! ! .#$,((%$,!Z&0O*3&G*!R16AS!=#023!CF!E

Z3,0,[! ! B0,,'!A+/%$&0%+J!R2#GL!Y::S

").1!B#%$3,0*K+04F!=IF!T.A

)'$,0',$[! Z%0,(,**

^&$,([! ! 16A

@,,[!! ! 16A!R,*$L!_9]:S

3#1)5'%"+%'-,'3+0*'&+*7#/0,

6,5&'/!$3,!I>1[!"&M,(!$,*$*!#'/!G,0-&0J#'2,!J,$0%2*

1,*$%'4!&-!,J,04%'4!K%&J,$0%2*

>--%2%,'$!$,*$%'4F!*$#$%*$%2#(!J,$3&/*

70&/+2$!#'/!2&JG&','$!$,*$*

)/,'$%-%2#$%&'!*5*$,J*

)'$,0&G,0#K%(%$5!$,*$%'4

8&'-&0J#'2,!$,*$%'4!R,*G,2%#((5!H,M,(!)))!W!.,J#'$%2S

1,*$%'4!&-!K%&J,$0%2!P+#(%$5!J,#*+0,*

7,0-&0J#'2,!G0&G,0$%,*!&-!K%&J,$0%2!#(4&0%$3J*

T'$,*$,/!#*G,2$*!&-!G,0-&0J#'2,F!$,*$%'4!#'/!*$#'/#0/%V#$%&'!4#G*

T*,*!#'/!#GG0&G0%#$,',**!&-!*5'$3,$%2!*#JG(,*

3#1)50'%"+%'--,''8+54")*9'%./'1"):%59'%01+5*0

>M#(+#$%&'!&-!J+($%D-#2$&0!#+$3,'$%2#$%&'

1,*$%'4!0,*%*$#'2,!$&!#2$%M,!#$$#2O*!R(%M,',**F!M+(',0#K%(%$5F!*G&&-%'4S

@,#*%K%(%$5!&-!2&JJ&'!20%$,0%#!$,*$%'4!&-!K%&J,$0%2!2&JG&','$*

I,-,#$!&-!*5*$,J*!R/,$,2$%&'!&-!,M#*%&'S

1,JG(#$,!G0&$,2$%&'!#'/!K%&J,$0%2!G*,+/&D%/,'$%-%,0*

70%M#25!,'3#'2%'4!$,23'&(&45F!/,D%/,'$%-%2#$%&'

T*#K%(%$5!W!A22,**%K%(%$5!$,*$%'4!-&0!K%&J,$0%2*

?,$0&*G,2$%M,*F!(,**&'*!(,#0',/F!3%*$&05F!(&'4D$,0J!G,0*G,2$%M,L

3#1)5'%"+%'---,';1+"%*)#.%<'%01+5*0

<G,0#$%&'#(!$,*$%'4`!2&'$%'+&+*!,M#(+#$%&'`!J&'%$&0%'4

8,0$%-%2#$%&'!20%$,0%#!#'/!2,0$%-%2#$%&'!G0&40#J*

",N!$,*$%'4!2#JG#%4'*

! A+$&J#$,/!K&0/,0!2&'$0&([!B#$,*!#'/!K#2O,'/!*5*$,J*

<M,0#((!*5*$,J!G,0-&0J#'2,

! A22+0#25F!*G,,/F!*,2+0%$5F!0,(%#K%(%$5F!+*#K%(%$5F!%'$,0&G,0#K%(%$5F!

*2#(#K%(%$5F!*$#K%(%$5F

^&N!$&!*G,2%-5!G,0-&0J#'2,!R,L4L!-&0!G0&2+0,J,'$S

1,23'&(&45!M*L!*2,'#0%&!M*L!&G,0#$%&'#(!M*L!&$3,0!$,*$*

@0&J!0,*,#023F!$&!/,*%4'F!$&!/,G(&5J,'$F!$&!-%,(/,/!&G,0#$%&'

a+#(%$5!#'/!0,(%#K%(%$5!&-!*,'*&0*!#'/!2#G$+0,!/,M%2,*

?,J&$,!$,*$%'4`!1,*$%'4!%'!$3,!8(&+/

(#.*%5*'*7+'#"$%.)=+"0,

>J#%([!!$#K#**%!A1!'%*$L4&M

3$$G[WWNNNL'%*$L4&MW%$(W%#/W%4W%KG29:;9L2-J

)678!9:;:

3$$G[WWNNNL'%*$L4&MW%$(W%#/W%4W%KG29:;:L2-J
Thank)you.)

Elham%Tabassi%

301%975%5292%

tabassi@nist.gov

www.nist.gov/itl/iad/ig/irex.cfm

http://www.nist.gov/itl/iad/ig/iris_image_qual_reading.cfm%

http://www.nist.gov/itl/iad/ig/bio_quality.cfm