Performance Evaluation of

Biometric Template Update

IBPC 2012

R. Giot1, C. Rosenberger1, B. Dorizzi2

1Université de Caen Basse-Normandie, UMR 6072 GREYC

ENSICAEN, UMR 6072 GREYC

CNRS, UMR 6072 GREYC

2Institut T ́el ́ecom; T ́el ́ecom SudParis

UMR 5157 SAMOVAR

March 8, 2012
Outline

Introduction

Available Public Databases

Differences Among Studies

Illustration

Open Questions

Conclusion

Template Update Evaluation 2 / 28
Outline

Introduction

Available Public Databases

Differences Among Studies

Illustration

Open Questions

Conclusion

Template Update Evaluation Introduction 3 / 28
Introduction

Template update Template update evaluation

• Allows to take into account • Lacks of homogeinity

intraclass variability through • Does not allow study

time comparison

• Active field of research

• Experimented on various

modalities

Template Update Evaluation Introduction 4 / 28
Outline

Introduction

Available Public Databases

Differences Among Studies

Illustration

Open Questions

Conclusion

Template Update Evaluation Available Public Databases 5 / 28
Available Public Databases

• Several databases are used in the literature

• They concern different modalities

• Morphological modalities

• 2D face

• 3D face

• Fingerprint

• Behavioral modalities

• Keystroke dynamics

• Handed signature

(EQUINOX, MORPH, UMIST, AR, VADANA)

(FRGC-EXP3)

(FVC2002)

(GREYC2009, DSL2009)

(MCYT-100)

• Few of them are specifically designed for template update (cf. next

slide)

Template Update Evaluation Available Public Databases 6 / 28
Differences Among Datasets

Database # users # samples # sessions

2D face

EQUINOX 40-50 20-100 -

MORPH 14 > 20 -

UMIST 20 25-55 -

AR 120 26 2

YOUTUBE videos 4 1200 1200

VADANA 43 ≈53 -

3D face

FRGC-EXP3 410+270 1-22 -

Fingerprint

FVC2002 110 8 1

Keystroke dynamics

GREYC2009 100 60 5

DSN2009 51 400 8

Handwritten signature

MCYT-100 100 25 5

Template Update Evaluation Available Public Databases 7 / 28
Outline

Introduction

Available Public Databases

Differences Among Studies

Illustration

Open Questions

Conclusion

Template Update Evaluation Differences Among Studies 8 / 28
..

..

..

Differences Among Studies

We can find several differences in the:

1

2

Template update system

⇒ mechanism used to update the biometric reference

This is not our subject in this presentation





Template update scenario

⇒ configuration parameters of the study

evaluation in a specific context













3 Template update evaluation

⇒ Analysis of the performance of the system

We will illustrate this point in this presentation

Template Update Evaluation Differences Among Studies 9 / 28
Scenario differences

Sessions Awareness Query Chronology

• Several sessions

• No session separation •





• No respect to chronology 



Respect to chronology

Query Presentation Order Input Size

• Global • More impostors • Genuine first

• Impostor first • More genuine

• Random presentation • Equal size • Rule (Seeger et al. 2011)

• Local (Seeger et al. 2011)

• All random

• Closest person

• Closet sample

Template Update Evaluation Differences Among Studies 10 / 28
Illustration Of The Complexity

Figure: Summary of all the possible variabilities in a template update

evaluation. Dotted nodes represent the possible configuration values, while

nodes with a straight line represent the configuration types.

Template Update Evaluation Differences Among Studies 11 / 28
Outline

Introduction

Available Public Databases

Differences Among Studies

Illustration

Open Questions

Conclusion

Template Update Evaluation Illustration 12 / 28
Experimental Protocol - Presentation

• We compute the scores of a biometric template update system

• One set of score per session

• Online evaluation

• We evaluate its performance in three different ways

Template Update System (Giot et al. 2011)

Modality Keystroke Dynamics

Authentication method Distance computing

Update decision Double-threshold semi-supervised online

Template Update Evaluation Illustration 13 / 28
Figure: Explanation of double threshold authentication

Template Update Evaluation Illustration 14 / 28
Experimental Protocol - Fixed parameters

Template Update Evaluation Parameters (Giot et al. 2011)

Sessions yes (8 sessions)

Evaluation online

Respect to chronology yes

Impostor rate 30%

Presentation orders random

Evaluation metric Equal Error Rate

(Variation of the acceptance threshold)

Template Update Evaluation Illustration 15 / 28
Experimental Protocol - Scores Computation

Enroll

test +

 adapt

test +

 adapt

test +

 adapt ...

Time Template Update Evaluation Illustration 16 / 28
Experimental Protocol - 1st evaluation

Evaluation A (Giot et al. 2011)

• Session performance is computed with the scores computed at

this session:

Ai = EER (scoresi), ∀i, 2 ≤ i ≤ # sessions (1)

• We have one EER per validation session:

A = [A2, . . . , A#sessions] (2)

Template Update Evaluation Illustration 17 / 28
i 1 ∑ ( ) Bi = EER scoresj , ∀i, 2 ≤ i ≤ # sessions i − 1

j=2

i

∑

(3)

1 = Ai i − 1

j=2

Experimental Protocol - 2nd evaluation

Evaluation B (Rattani et al. 2011)

• Session performance is computed by the mean of all the previous

sessions' performance (including the current one).

• We have one EER per validation session:

B = [B2, . . . , B#sessions] (4)

Template Update Evaluation Illustration 18 / 28
C = [ C, . . . , C ] | {z } #sessions−1

Experimental Protocol - 3rd evaluation

Evaluation C (Seeger et al. 2011)

• One global performance measure is computed (i.e., all scores of all

sessions are merged):

  # sessions

∪ C = EER  scoresi

 (5)

i=2

• We have one EER for the whole interval:

(6)

Template Update Evaluation Illustration 19 / 28
Results - One threshold configuration

2 3 4 5 6 7 8

Session

16

17

18

19

20

21

22

EER (%)

A

B

C

Figure: EER value per session, for one update threshold.

Template Update Evaluation Illustration 20 / 28
Results - Another threshold configuration

2 3 4 5 6 7 8

Session

16

18

20

22

24

26

EER (%)

A

B

C

Figure: EER value per session, for one update threshold.

Template Update Evaluation Illustration 21 / 28
Interpretation

Interpretation is different depending on the evaluation scheme

2 3 4 5 6 7 8

Session

16

17

18

19

20

21

22

EER (%)

A

B

C

2 3 4 5 6 7 8

Session

16

18

20

22

24

26

EER (%)

A

B

C

A The template update system does not perform well.

B The template update system is not too bad.

C Performance is averaged, but we cannot know if it

is because of template ageing, because of a poor

algorithm or because of a bad dataset.

In the three schemes, the scores are strictly the sames.

Template Update Evaluation Illustration 22 / 28
Interpretation

Discussion

• This difference of interpretation is problematic

• We cannot fairly compare the existing studies

• Which of these three methods is more appropriate ?

Template Update Evaluation Illustration 23 / 28
Outline

Introduction

Available Public Databases

Differences Among Studies

Illustration

Open Questions

Conclusion

Template Update Evaluation Open Questions 24 / 28
Open Questions

In addition to these problems, we can raise additional questions:

• What are the characteristics of an interesting dataset for such

kind of studies?

• What is the best evaluation procedure in order to easily compare

the systems without doing each time all the previous experiments

from scratch ?

• Is it more informative to work with datasets separated in several

sessions, or with datasets captured in a longer period without

more information ?

Template Update Evaluation Open Questions 25 / 28
Outline

Introduction

Available Public Databases

Differences Among Studies

Illustration

Open Questions

Conclusion

Template Update Evaluation Conclusion 26 / 28
..

..

..

Conclusion

Template update is an active field of research.

However, there is no common:

1 Way of evaluation or template update systems

• We have shown that the way of evaluating a system can change its

perception.

2

3

Method to create and characterize useful datasets

• Most datasets are not specifically designed for template update

Specific vocabulary

• First try in keystroke dynamics (Seeger et al. 2011)

We think that these three points must be answered in the future in

order to ease the work on such subject (especially the first one).

Template Update Evaluation Conclusion 27 / 28
Thank you for your attention

Any questions ?

Template Update Evaluation Conclusion 28 / 28