FpVTE 2003 Evaluations Overview

Table of Contents 

FpVTE 2003 Evaluations Overview .......................................................................... 1

1 Overview........................................................................................................ 1

2 Purpose.......................................................................................................... 2

3 Who Should Participate.................................................................................. 2

4 Sponsors and Supporters ............................................................................... 2

5 Overview of Evaluations................................................................................. 2

5.1 Throughput Requirements........................................................................... 3

5.2 Matcher Results .......................................................................................... 3

5.3 Operating Points ......................................................................................... 4

6 Data Description and Restrictions .................................................................. 4

6.1 Fingerprint Types........................................................................................ 4

6.2 Fingerprint Sources..................................................................................... 4

6.3 Fingerprint File Format ............................................................................... 5

6.4 Data Restrictions......................................................................................... 5

7 Test Preparation ............................................................................................ 5

8 Final Report.......................................................................................................................... 6

1 Overview

The Fingerprint Vendor Technology Evaluation (FpVTE) 2003 was an independently 

administered technology evaluation of fingerprint matching, identification, and verification 

systems. FpVTE 2003 was conducted by the National Institute of Standards & Technology 

(NIST) on behalf of the Justice Management Division (JMD) of the U.S. Department of 

Justice.

FpVTE was designed to assess the capability of fingerprint systems to meet requirements for 

both large-scale and small-scale real world applications. FpVTE 2003 consists of multiple 

tests performed with combinations of fingers (e.g., single fingers, two index fingers, four to 

ten fingers) and different types and qualities of operational fingerprints (e.g., flat livescan 

images from visa applicants, multi-finger slap livescan images from present-day booking or 

background check systems, or rolled and flat inked fingerprints from legacy criminal 

databases).

FpVTE 2003 will serve as part of NIST's statutory mandate under section 403c of the USA 

PATRIOT Act to certify those biometric technologies that may be used in U.S. VISIT (U.S. 

Visitor and Immigrant Status Indication Technology, formerly known as the US entry-exit 

system). In addition, FpVTE results may form the basis for the design and acquisition of 

other large-scale fingerprint identification systems.

FpVTE 2003 analysis and methodologies are built on the multi-agency Face Recognition 

Vendor Test (FRVT) 2002. See http://www.frvt.org
FpVTE 2003 evaluations was conducted at the NIST facilities at Gaithersburg, MD, between 

September 29 and December 31, 2003.

Interested parties should send contact information (name, email, telephone, address) to 

FpVTE@nist.gov . 

The deadline for registration for Participants was 12 August 2003.

2 Purpose

FpVTE was intended to evaluate the state of the art / state of the practice in fingerprint 

matching systems. The Supporters and Sponsors of FpVTE are specifically interested in 

fingerprint matcher performance in these areas:

• Accuracy of very large-scale automated fingerprint identification systems (AFIS) in 

general 

• Accuracy of fingerprint identification systems using different combinations of fingers 

(e.g., single fingers, two index fingers, four to ten fingers)

• Accuracy of fingerprint identification systems using the range of different types and 

qualities of operational U.S. Government fingerprints (e.g., individual flat livescan 

images from visa applicants, multi-finger slap livescan images from present-day 

booking or background check systems, and rolled and flat inked fingerprints from 

legacy criminal databases).

• Accuracy of fingerprint verification systems

• Accuracy of fingerprint binning, screening or filtering systems (such as are used as 

processing stages in large identification systems)

3 Who Should Participate

Makers of commercially available fingerprint matching, verification, or identification (AFIS) 

systems were invited to participate in the Fingerprint Vendor Technology Evaluation 2003.

In addition, companies, research organizations, and universities that have developed 

mature prototype or research fingerprint matching, verification, or identification systems are 

invited to participate. Makers of fingerprint screening or filtering systems were also 

encouraged to participate. 

FpVTE was not evaluating image acquisition devices (fingerprint scanners).

4 Sponsors and Supporters

Sponsors

• Justice Management Division, US Department of Justice, IDENT/IAFIS Project

• National Institute of Standards and Technology

• Bureau of Immigration and Customs Enforcement

• Federal Bureau of Investigation

• U.S. Department of State

• U.S. VISIT Program

Supporters

• European Commission Services

• Office of the Chief Information Officer, U.S. Department of Justice

• Royal Canadian Mounted Police
• U.K. Police Information Technology Organisation (PITO)

• U.S. Department of Homeland Security

• U.S. Department of Justice

5 Overview of Evaluations

FpVTE was composed of three separate tests, the Small-Scale Test (SST), Medium-Scale 

Test (MST), and the Large-Scale Test (LST).

The SST and MST evaluated matching accuracy using individual fingerprint images. This 

contrasts with the LST, which evaluated matching accuracy using sets of fingerprint images.

In general, the SST and MST was of interest to makers of single-stage matchers, verification 

or small-scale identification systems, or new algorithms. Makers of fingerprint screening or 

filtering systems were also encouraged to participate in the MST. In general, the LST 

would be of interest to AFIS vendors.

The tests are designed so that the SST is a subset of the MST. LST participants were 

encouraged to participate in the MST.

If space is available, FpVTE personnel may allow a participant to enter more than one 

system in the evaluation. A second system might be submitted to demonstrate capabilities 

that can be achieved only through distinct parameter settings, an alternate system 

configuration, or different algorithms.

Click here to jump to the Test Plan for more information about SST, MST, or LST. 

5.1 Throughput Requirements

Tentative Participants were required to complete a System Throughput Questionnaire to 

qualify for the tests. This information was used to determine the final test design. The size 

and structure of the test was designed to optimize among competing analysis objectives, 

available resources, and the desire to include all qualified participants.

The System Throughput Questionnaire is located in the Participant Area, which is only 

available to registered Participants and Tentative Participants.

5.2 Matcher Results

The results from each test are participant-defined measures of similarity, which for most 

systems means matcher scores. Note that the systems do not return Match and Non-match 

determinations. FpVTE analysis is based on distributions of similarity scores for mate vs. 

non-mate comparisons. Which fingerprints are mated is, of course, not revealed.

For the SST and MST tests, matchers compare individual fingerprint images, and return a 

similarity score for each comparison. When comparing one dataset of fingerprint images 

against another, a fully-populated matrix of similarity scores is generated. This fullypopulated similarity matrix is what will be used to evaluate SST and MST Participants.

For the LST, matchers compare sets of fingerprint images (e.g. a 2-image, 4-image, or 10-

image electronic "card" of fingerprint images). Since an AFIS generally filters out many nonmates, similarity scores may not be available for every comparison. Participants 

nevertheless will be expected to generate a fully-populated matrix of scores, but may, for 

instance, fill the majority of the matrix with a default value indicating no similarity. 

Participants are advised that the choice of true scores vs. default values will affect scoring, 

and are encouraged to fill the matrix as completely as possible with true scores.
Click here to jump to the Test Plan for more information about similarity scores and 

matrices.

5.3 Operating Points

Evaluations were based on ROC (Receiver Operator Characteristic) or, equivalently, DET 

(Detection Error Trade-off) analysis based on similarity scores. Rank-based analysis will also 

be performed.

For SST, evaluations focused on the relationship between TAR (True Accept Rate: TAR = 1 -

FRR (False Reject Rate)) and FAR (False Accept Rate), when FAR is between 10-2 and 10-5.

For MST, evaluations focused on the relationship between TAR and FAR when FAR is 

between 10-2 and 10-7. In addition, evaluations will focus on the relationship between TAR 

and FAR when TAR is between 98% and 99.99%.

For LST, the relationship between TAR and FAR was analyzed, focusing solely on very low 

FAR values (down to about 10-8).

Since some algorithms do not operate equally well at a broad range of operating points, 

participants could submit two algorithms or tunings of an algorithm for the SST or MST, one 

optimized for low FAR and another for high TAR.

6 Data Description and Restrictions

The fingerprints have been collected from a range of U.S. Government sources. Some of the 

fingerprints are representative of current operational data, while others are representative 

of legacy data.

6.1 Fingerprint Types

The fingerprints are of three types:

• Flat (plain) fingerprints, individually collected

• Fingerprints segmented from slap (simultaneous plain) fingerprints (segmented by 

FpVTE personnel using a combination of automatic and manual methods).

• Rolled fingerprints

Note that the slap fingerprints have already been segmented into individual fingerprints.

6.2 Fingerprint Sources

Individual flat (plain) fingerprints were collected using a variety of single-finger livescan 

devices.

Slap fingerprints were collected using multi-finger livescan devices, as well as inked 

fingerprint cards scanned using flatbed scanners.

Rolled fingerprints were collected using rolled livescan devices, as well as inked fingerprint 

cards scanned using flatbed scanners.

The fingerprints were collected by the following devices. The listing of makes and models 

does not imply a recommendation by NIST or FpVTE personnel, but simply recognizes the

actual devices used by the variety of Federal agencies that contributed data to FpVTE.
Single-finger livescan devices:

• Identix/Identicator DFR-90

• Cross Match Verifier Model 300

• Identix TV555

Rolled and multi-finger livescan devices:

• Identix TP-600

• CrossMatch ID-1000

• DBI Tenprinter

• Heimann LS2 Check NEW

Inked cards were scanned using FBI EFTS Appendix F-certified flatbed scanners.

The source (type of scanner) of each fingerprint was not provided in the tests.

6.3 Fingerprint File Format

Each fingerprint (SST or MST) or fingerprint set (LST) was contained in an ANSI/NIST 

format file. Each file contains

• a single fingerprint image (SST and MST), or

• a set of fingerprint images (LST)

All images are WSQ compressed.

Note: The FBI’s Electronic Fingerprint Transmission Specification (EFTS) is based on 

ANSI/NIST. Fingerprint files that are EFTS compliant are necessarily ANSI/NIST compliant.

6.4 Data Restrictions

The FpVTE 2003 Evaluation Datasets will bear the legend “Notice: May contain Privacy Act 

or FOIA Protected Information” and, to the extent permitted by law, will be protected under 

the Freedom of Information Act (5 U.S.C 552) and the Privacy Act (5 U.S.C. 552a) as 

applicable.

FpVTE 2003 Evaluation Datasets and data derived from the datasets shall not be retained in 

any way or form whatsoever by Participants after completion of the Evaluation.

Systems being evaluated in FpVTE 2003 shall not be accessible from outside the room in 

which the evaluation is being conducted. No system entered in the test can have or use any 

wireless networking equipment, modems, or access to the Internet.

7 Test Preparation

Participants were expected to modify their systems as necessary to meet the following test 

requirements:

• Produce properly formatted matrices of similarity scores as output.

• Perform a sequence of subtests with little or no human intervention. Matchers will be 

expected to load datasets and generate similarity matrices as specified in an XML 

document provided during the test.

• Complete within the allotted time.
FpVTE 2003 Sample Datasets and Software was supplied to Participants to assist in 

preparing for FpVTE 2003. The FpVTE 2003 Sample Datasets are representative of the 

FpVTE 2003 Evaluation Datasets in format. Image quality, collection device and other 

characteristics may vary between the Sample and Evaluation Datasets.

8 Final Report

After completion of the evaluations, the Government combined all results into a Final 

Report. The FpVTE 2003 Final Report is expected to be released in early 2004. The FpVTE 

2003 Final Report will contain, at a minimum, descriptive information concerning FpVTE 

2003, descriptions of each experiment, evaluation results, and each Participant’s five-page 

system description document.

The Final Report can be found on the FpVTE 2003 website 

(https://www2.nist.gov/information-technology-laboratory/iad/image-group/fpvte-2003).