FVC-onGoing 

On-line Evaluation of 

Fingerprint Recognition Algorithms 
Outline 

• FVC: Fingerprint Verification Competitions 

– Background 

– What’s new in FVC-onGoing 

• How FVC-onGoing works 

– Architecture of the system 

– An example of evaluation 

• Benchmark areas and benchmarks 

– Fingerprint verification 

– ISO template matching 

– Fingerprint orientation Extraction 

• The next steps 

– New benchmark areas planned 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

1 
Fingerprint Verification Competitions 

• FVC: Technology Evaluations of Fingerprint Verification Algorithms 

• Since 1999, when we started organizing FVC2000: 

– Four competitions: FVC2000, FVC2002, FVC2004, FVC2006 

– A total of 179 algorithms were evaluated 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

– A total of 16 databases were collected and made available 

2 
FVC-onGoing 

• Web-based automatic evaluation of fingerprint recognition algorithms 

– Participants can be: companies, academic research groups, or 

independent developers 

– Algorithms are tested on sequestered datasets and results are reported 

using well-known performance indicators and metrics 

• Fully automated: 

1.The system automatically tests the algorithm submitted by a participant 

2.The participant sees the results in its “private area” 

3.Then the participant may decide to publish the results in the public section 

of the FVC-onGoing web site 

• Main aim: 

– Track the advances in fingerprint recognition technologies, through 

continuously updated independent testing and reporting of performances 

on given benchmarks 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

3 
What’s new in FVC-onGoing 

• Previous FVC initiatives were organized as “competitions” 

– Specific calls and Fixed time frames 

• FVC-onGoing is: 

– An “on going competition” always open to new participants 

•Datasets will remain sequestered 

– An evolving online repository of benchmarks, evaluation metrics and 

results 

•However the benchmark datasets will not evolve over time; in case new datasets 

will be added in the future, they will form a different benchmark (or a new version 

of an existing one) 

• Not only limited to fingerprint verification algorithms: 

– Ad hoc benchmarks for testing specific modules of fingerprint verification 

systems are being made available: 

•Orientation Image Extraction (already available) 

•Fingerprint indexing 

•Minutiae Extraction 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

4 
FVC-onGoing: Testing procedure 

• As in previous FVCs, the testing procedure is Strongly Supervised

– Protocol: binary executable programs compliant to a given input/output

protocol are tested on the evaluator’s hardware

– Results: generated by the evaluator from the matching

scores obtained during the test

Evaluator’s Site 

DB Results 

Match 

scores 

Generation 

of the results 

Matching 

Algorithm 

Evaluator’s hardware 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

5 

The tested algorithm is executed 

in a totally-controlled environment, 

where all input/output operations 

are strictly monitored. 
FVC-onGoing: Workflow 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

6 

Performance result

generator 

Web site: Public area 

Benchmarks, 

Protocols, ... 

Public results 

Test Engine 

Results 

Test runner 

Test output

Result published 

Registration Checked 

algorithm 

Participant Algorithm verifier 

(antivirus, protocol 

compliance, ...) 

Participant data 

Participant 

results 

Web site: Private area 

Test 

dataset 

Benchmark datasets 

Benchmark 

area 1 

DB 1 

DB 2 

... 

Benchmark 

area n 

DB 1 

DB 2 

... 

.. 

. 

Algorithm 

submitted 

To be 

evaluated 

Evaluated 

Algorithm repository Next 

algorithm 

in queue 

Evaluated algorithm 
FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

An example... 

7 
 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

8 

Benchmark areas and benchmarks 

• FVC-onGoing benchmarks are grouped into Benchmark Areas 

• All the benchmarks of a given benchmark area: 

– Address the same (sub)problem 

– Share the same evaluation protocol 

• Each benchmark is based on a sequestered dataset that will not evolve 

over time 

– In case new datasets will be added in the future, they will form a different 

benchmark (or a new version of an existing one). 

– Only results obtained on the same benchmark are comparable. 

– A participant may submit more algorithms to the same benchmark 

•But there is a minimum break (e.g. one month) between consecutive submissions 

• Currently available benchmark areas: 

– FV: fingerprint verification using proprietary templates 

– FMISO: fingerprint matching using ISO/IEC 19794-2 templates 

– FOE: fingerprint orientation extraction (orientation image) 
Area Benchmark Description 

FV-TEST A simple dataset useful to test 

algorithm compliancy with the testing protocol 

FV-STD-1.0 Fingerprint images acquired in operational 

conditions using high-quality optical scanners 

FV-HARD-1.0 Difficult cases (noisy images, distorted 

impressions, etc.): more challenging 

Currently available benchmarks 

FV 

Fingerprint 

Verification 

FMISO-TEST A simple dataset useful to test 

algorithm compliancy with the testing protocol 

FMISO-STD-1.0 Fingerprint images acquired in operational 

conditions using high-quality optical scanners 

FMISO 

Fingerprint ISO 

Template Matching 

FMISO-HARD-1.0 Difficult cases (noisy images, distorted 

impressions, etc.): more challenging 

FOE-TEST 

FOE-STD-1.0 

FOE 

Fingerprint 

Orientation Extraction 

A simple dataset useful to test 

algorithm compliancy with the testing protocol 

Orientation extraction benchmark on fingerprints

with orientation ground-truth manually labeled 

using an ad-hoc software tool. Good-quality and 

bad-quality datasets. 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

9 
Current status

Registered Participants 127 Algorithm Evaluated 388 

Academic Research 21 Groups 

Companies 46 

Independent 60 Developers 

From July 2009 

to February 2010 

Fingerprint 260 Verification 

Fingerprint ISO 128 Template Matching 

Results Published 20 

Fingerprint 11 Verification 

Fingerprint ISO 9 Template Matching 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

10 
FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

Protocols and results: on the web site 

11 

Benchmark protocols

Published results, 

Performance 

indicators, 

Graphs, 

, 

... 

http://biolab.csr.unibo.it/FVConGoing 
FV: Fingerprint Verification 

• Benchmark FV-STD-1.0: 

• Benchmark FV-HARD-1.0: 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

12 
FMISO: Fingerprint ISO Template Matching 

• Benchmark FMISO-STD-1.0: 

• Benchmark FMISO-HARD-1.0: 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

13 
New Benchmark: Fingerprint Orientation Extraction 

• Challenge: Estimation of local orientations in low-quality images 

– A fundamental step in fingerprint analysis and recognition 

Typical 

approach 

Ideal 

result 

Enhancement 

based on the 

estimated 

orientations 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

14 

Enhancement 

based on the 

true 

orientations 
Fingerprint Ground truth Extracted Errors 

Evaluating Fingerprint Orientation Extraction 

• How the benchmark works: 

– Participants’ algorithms are required to extract local orientations from 

fingerprint images and to save them into a specific format. 

– The extracted orientations are compared to the ground-truth in order to 

assess the algorithm accuracy. 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

15 
Software tool for orientation ground truth markup 

Manual adjustment of a single local 

orientation element: (a) the selected 

element, (b) the initial orientation 

proposed by the software, (c) the 

orientation selected by the user 

moving the mouse cursor (d). 

Local estimations made by the user 

(white segments), with the Delaunay 

triangulation and all the interpolated 

local orientations (grey segments). 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

16 
FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

Software tool for orientation ground truth markup 

17 

Manual adjustment of a single local 

orientation element: (a) the selected 

element, (b) the initial orientation 

proposed by the software, (c) the 

orientation selected by the user 

moving the mouse cursor (d). 

Local estimations made by the user 

(white segments), with the Delaunay 

triangulation and all the interpolated 

local orientations (grey segments). 
 

FOE: Datasets and Performance Indicators 

• Datasets: 

– The benchmarks consists of 2 datasets: a good quality dataset and a bad 

quality dataset. 

– The challenge is to obtain a good orientation extraction accuracy on the 

bad quality dataset without losing too much accuracy on the good quality 

dataset. 

•To reduce noise on low quality fingerprints, some approaches tend to oversmooth 

the orientation image and this could compromise accuracy on good quality 

fingerprints. 

• Performance indicators: 

– AvgErrBQ (Average Error on the Bad Quality Dataset) 

– AvgErrGQ (Average Error on the Good Quality Dataset) 

– Average orientation extraction time, Maximum amount of memory allocated 

– Orientation deviation histogram (over all the orientation elements) 

– Average error histogram (over all the fingerprints) 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

18 
 

FOE: Datasets and Performance Indicators 

• Datasets: 

– The benchmarks consists of 2 datasets: a good quality dataset and a bad 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

quality dataset. 

– The challenge is to obtain a good orientation extraction accuracy on the 

bad quality dataset without losing too much accuracy on the good quality 

dataset. 

•(to reduce noise on low quality fingerprints, some approaches tend to oversmooth 

the orientation image and this could compromise accuracy on good quality 

fingerprints) 

• Performance indicators: 

– AvgErrBQ (Average Error on the Bad Quality Dataset) 

– AvgErrGQ (Average Error on the Good Quality Dataset) 

– Average orientation extraction time, Maximum amount of memory allocated 

– Orientation deviation histogram (over all the orientation elements) 

– Average error histogram (over all the fingerprints) 

19 
FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

FOE: Participant’s toolkit and samples 

• Source code: C and C# skeletons are available. 

– Perform all the necessary I/O 

(including loading image and foreground, saving the orientation image, ...). 

• Sample datasets 

• Sample algorithm (Gradiend-based) and Test runner tool 

• Software viewer to display: 

– Fingerprints, 

– Ground truth, 

– Orientations extracted by an algorithm, 

– Orientation differences (errors) 

20 
FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

• Source code: C and C# skeletons are available. 

– These source files perform all the necessary I/O 

(including loading image and foreground, saving the orientation image, ...). 

• Sample datasets 

• Sample algorithm (Gradiend-based) and Test runner tool 

• Software viewer to display: 

– Fingerprints, 

– Ground truth, 

– Orientations extracted by an algorithm, 

– Orientation differences (errors) 

FOE: Participant’s toolkit and samples 

21 
FVC-onGoing: for Whom? 

Who 

Researcher 

Reviewer 

Vendor 

Developer 

End user 

Sys. Integrator 

Why 

• New algorithms can be easily compared to the 

state-of-the-art. 

• Benchmarks not only for the whole recognition 

problem, but also for sub-problems. 

• FVC-onGoing is an evolving online repository of 

evaluation metrics and results. 

• Participants can see the results before publishing. 

• The competition is always open: new algorithms 

and new versions can be submitted at any time. 

• At any time, end users and system integrators 

may ask potential providers to assess their 

performance on one or more benchmarks. 

• An evolving snapshot of the fingerprint recognition 

technology. 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

22 
The next steps 

• New benchmark areas planned 

– Fingerprint Indexing 

– Fingerprint Identification (1:N) 

– Minutiae extraction accuracy 

• New benchmarks with synthetic datasets 

– Large datasets for Fingerprint Orientation Extraction (orientation groundtruth can be automatically generated by SFinGe) 

– Datasets for Minutiae Extraction Accuracy (minutiae ground-truth 

automatically generated by SFinGe) 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

23 

SFinGe (the Italian for Sphinx, pron. sphin-je) 

A software able to synthetically (randomly) 

generate large databases of realistic fingerprint images

with ground truth data (minutiae, local orientations, ...) 
Links 

• FVC-onGoing web site: 

• http://biolab.csr.unibo.it/FVConGoing 

• ...or Google “fvc on going” and press “I’m Feeling Lucky” 

• Biometric System Laboratory web site: 

• http://biolab.csr.unibo.it 

• ...or Google “biometric system laboratory” and press “I’m Feeling Lucky” 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

24 
Thank you ! 
Benchmark Area Fingerprint Verification (FV) 

• Benchmarks: 

– FV-TEST: A simple dataset useful to test 

algorithm compliancy with the testing protocol. 

•Results cannot be published. 

– FV-STD-1.0: Fingerprint images acquired in operational 

conditions using high-quality optical scanners. 

•Results should reflect the expected accuracy in large-scale fingerprint-based 

applications. 

– FV-HARD-1.0: Contains a relevant number of difficult cases (noisy images, 

distorted impressions, etc.) that makes fingerprint verification more 

challenging. 

•Results do not necessarily reflect the expected accuracy in real applications. 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

26 
FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

• From the FVC-onGoing web site: 

Fingerprint Verification (FV): Protocol 

27 
Benchmark Area Fingerprint Matching ISO (FMISO) 

• Benchmarks: 

• FMISO-TEST: A simple dataset useful to test 

algorithm compliancy with the testing protocol 

•Results obtained on this benchmark cannot be published. 

• FMISO-STD-1.0: ISO templates created from fingerprint images 

acquired in operational conditions using high-quality optical scanners. 

•Results should reflect the expected accuracy in large-scale fingerprint-based 

applications. 

• FMISO-HARD-1.0: Contains a relevant number of difficult cases (noisy 

images, distorted impressions, etc.). 

•Results do not necessarily reflect the expected accuracy in a real applications. 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

28 
Fingerprint Matching ISO (FMISO): Protocol 

• From the FVC-onGoing web site: 

FVC-onGoing: on-line evaluation of fingerprint recognition algorithms 

29