Independent Performance Evaluation of Biometric Systems

Davrondzhon Gafurov, Bian Yang, Patrick Bours and Christoph Busch 

Gjøvik University College, 

{Firstname.Lastname}@hig.no

I. INTRODUCTION 

Often in the development of a biometric product the 

evaluator of the system is the same institution who developed 

the algorithm. Furthermore, usually the test data set is also 

collected by the same developer/evaluator and in most cases 

such database will not be public. Consequently, test results 

cannot be verifed by independent institutions. Although 

this can be justifable (e.g. in the optimization phase of 

an algorithm), from the perspective of a potential customer 

it reduces trustworthiness of the developed system and 

reported performances. Therefore, for performance evaluation the availability of independent databases and desirably 

independent evaluators are very important. It is also essential 

that algorithm developers do not have access to the testing 

database and thus the risk of tuned algorithms is minimized. 

Pseudonymous identifers (PI) are complementary to 

image- or minutiae- based references and provide a level 

that is both more privacy protective and more effcient 

than symmetric or asymmetric encryption of a biometric 

reference image or minutiae template record [1]. With PI 

an individual retains complete control of its biometric data 

as multiple PI can be generated from a single biometric 

characteristic without any risk that these can be linked 

together. At the same time any of these identifers can be 

cancelled and replaced by a new one if needed. Research on 

such PI is a core objective of the TURBINE project [2]. 

This work presents the biometric performance test report 

on a fngerprint performance evaluation that has been generated in the context of the TURBINE project [2]. It is worth 

to emphasize that this performance report is results of the 

frst testing round which are not fnal results of the project. 

For the second and fnal testing round, project partners will 

submit their improved algorithms, and its results will be 

available in year 2011. Furthermore, here we only report 

the ”biometric performance” per se of the algorithms while 

the ”security performance” of the PI algorithms is evaluated 

by others in TURBINE. 

II. PERFORMANCE METRICS AND DATA SETS 

The main error types associated with any biometric performance are FMR versus FNMR, and FAR versus FRR. In this 

work we will refer to the former and latter pairs as algorithm 

and system performances (or errors), respectively. They are 

related to each other according to the below formulas: 

FAR = FMR∗(1−F T A), FRR = F T A+F NMR∗(1−F T A)

In our tests, we defne FTA using formula below. 

F T A = F T C + F T X ∗ (1 − F T C)

where F T C (Failure To Capture) and F T X (Failure To 

eXtract) are estimated as follow 

(# terminated capture attempts) + (# not suffcient quality images) F T C = Total # capture attempts 

( # not encoded templates) F T X = Total # images submitted to the template encoder 

(”#” stands for ”number of”). It is worth noting that 

F T A computation incorporates both hardware (in F T C) 

and software (in F T X) related failures. 

The (binary) fngerprint verifcation algorithms are provided by project partners, in particular Sagem Securite, 

Precise Biometrics, Philips Research Europe and University 

of Twente. An external fngerprint verifcation package by 

Neurotechnology (VeriFinger [3]) is bought and also included in the testing. The submitted PI software for the 

frst benchmark encompasses software which simulates the 

effect of a physical protection layer obtained when implementing encoding and comparison within a smartcard (oncard-comparison techniques). 

As a test database we use the GUC100 multi-scanner 

fngerprint database which consisted of fngerprint images of 

all 10 fngers from 100 subjects (almost 72000 fngerprint 

images in total) [4]. Neither project partners nor external 

parties had access to the GUC100 database or were involved 

in the testing activity. Performance evaluations were carried 

out solely by the GUC research team as an independent and 

neutral academic party in the project. 

III. PERFORMANCE RESULTS 

The focus of this work is not on comparing individual 

performances of algorithms or scanners but rather emphasizing characteristics of biometric performance evaluation 

and observing the potential performance degradation in the 

transition to the PI level. Therefore, the names of scanner 

(denoted as S1, ..., S6) and algorithm suppliers (except 

Neurotechnology) are anonymized. Test results are given in 

terms of the DET-curves. At the minutiae level curves x-axis 

are plotted in logarithmic scale. 
A. Results at the minutiae level

Figure 1 shows results of algorithm and system performance evaluations using algorithms from the aforementioned 

suppliers on the GUC100 database at minutiae level, which 

is unprotected from a data privacy perspective. Figure 2 

presents results of same performance evaluations (as in 

Figure 1) but now by taking image quality into account. We 

used NFIQ algorithm [5] for estimating fngerprint image 

quality. We selected NFIQ value 3 as a quality threshold, i.e. 

fngerprint images with NFIQ score 4 or 5 are considered 

as having insuffcient quality and counted in FTC. 

As one can observe from Figures 1 and 2 the difference 

between algorithm and system performances is insignifcant 

when ignoring image quality but very signifcant when 

taking into account image quality (see in Figures 2). 

●

●

−2 −1 0 1 2

0

5 10 15

Log10(FMR), %

FNMR, %

FMR0.1 EER

S1: EER=1.54, FNMR@FMR0.1=2.05

S3: EER=1.93, FNMR@FMR0.1=2.5

S2: EER=3.73, FNMR@FMR0.1=4.32

S4: EER=1.89, FNMR@FMR0.1=2.44

S6: EER=3.19, FNMR@FMR0.1=3.79

S5: EER=4.3, FNMR@FMR0.1=6.29

●

●

−2 −1 0 1 2

0

5 10 15

Log10(FAR), %

FRR, %

FAR0.1 EER

S1: EER=1.56, FRR@FAR0.1=2.02

S3: EER=1.93, FRR@FAR0.1=2.5

S2: EER=3.73, FRR@FAR0.1=4.32

S4: EER=1.89, FRR@FAR0.1=2.44

S6: EER=3.19, FRR@FAR0.1=3.79

S5: EER=4.3, FRR@FAR0.1=6.29

Figure 1. Algorithm and system performances without considering image 

quality. 

●

●

−2 −1 0 1 2

0 1 2

3

4

Log10(FMR), %

FNMR, %

FMR0.1

EER

S1: EER=0.71, FNMR@FMR0.1=0.74

S3: EER=0.57, FNMR@FMR0.1=0.57

S2: EER=1.8, FNMR@FMR0.1=2.04

S4: EER=1.85, FNMR@FMR0.1=2.33

S6: EER=1.58, FNMR@FMR0.1=1.92

●

●

−2 −1 0 1 2

0

5 10 15

Log10(FAR), %

FRR, %

FAR0.1 EER

S1: EER=5.57, FRR@FAR0.1=5.51

S3: EER=7.3, FRR@FAR0.1=7.83

S2: EER=5.51, FRR@FAR0.1=6.03

S4: EER=2.04, FRR@FAR0.1=2.71

S6: EER=5.93, FRR@FAR0.1=6.12

Figure 2. Algorithm and system performances with considering image 

quality. 

B. Results at the PI level

Figure 3 shows the frst phase performance evaluation 

results at the pseudonymous identifer level only using 

images from 3 scanners. At this level performance tests were 

conducted without considering the image quality metric of 

a processed sample. The fgures indicate that performance 

deterioration can occur from minutiae level to PI level. 

●

●

0 5 10 15

0

2

4

6 8 10

FMR, %

FNMR, %

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

FMR0.1 EER

S6 (pseudo−identity)

S2 (pseudo−identity)

S4 (pseudo−identity)

S6 (minutiae)

S2 (minutiae)

S4 (minutiae)

Figure 3. Supplier A - algorithm performance at the minutiae (unprotected) 

and PI (protected) levels. 

For each test scenario at the PI level the testing time 

is signifcantly higher than at the minutiae level. Thus, the 

number of points in PI level DET curves are limited (seven 

in Figure 3). Also because of small number of points, the 

single number performance indicators (e.g. EER) are not 

estimated neither. 

IV. CONCLUSION 

In this work we presented an independent report on the 

frst phase performance evaluation of fngerprint verifcation algorithms in the context of the TURBINE project. 

Performance testing was conducted both at the minutiae 

and pseudonymous identifer levels using GUC100 database 

which consisted of almost 72000 fngerprint images from 

100 individuals. Algorithm developers did not have access to 

this database or were involved in the testing activity. All the 

performance evaluation tests were conducted independently 

by a neutral academic party in the project. 

For increasing the trustworthiness of biometric performance report for potential customers, it is recommended/desired that evaluations to be conducted by an 

independent third party. 

REFERENCES 

[1] J. Breebaart, C. Busch, J. Grave, and E. Kindt. A reference 

architecture for biometric template protection based on pseudo 

identities. In BIOSIG 2008, 2008. 

[2] TURBINE project - TrUsted Revocable Biometric IdeNtitiEs. 

http://www.turbine-project.eu/index.php. 

[3] Neurotechnology’s verifnger 6.0. 

http://www.neurotechnology.com/. Last visit: 14.10.2009. 

[4] GUC100 multi-scanner fngerprint database for in-house 

(semi-public) performance and interoperability evaluation. 

http://www.nislab.no/guc100. 

[5] M.D. Garris, E. Tabassi, and C.L. Wilson. NIST Fingerprint 

evaluations and developments. Proceedings of the IEEE, 2006.