ELFT‐EFS Results

NIST Evaluation of Latent Fingerprint Technologies: Extended Feature SeEvaluation #2

Michael Indovina, NISTVladimir Dvornychenko, NISTAustin Hicklin, Noblis

George Kiebuzinski, Noblis

1 
Overview

• ELFT‐EFS is an open evaluation of the accuracy of automated latent f

identification using features marked by experienced latent fingerprint 

• ELFT‐EFS is being conducted by NIST on behalf of the FBI and DHS

• Participants: SAGEM MorphoTrak, NEC, 3M Cogent, Warwick Warp, S

SPEX Forensics. (Results delayed for SPEX; not included here)

• 1066 latent images, with markup by certified latent examiners, were 

against a gallery of both rolled and plain exemplar fingerprints from a

100,000 subjects.

• The test evaluated the accuracy of searches when using: a) latent ima

images combined with selected subsets of the Extended Feature Set (

in ANSI/NIST‐ITL 2011); and c) selected features alone.

2 
NIST Latent Testing – ELFT‐EFS

• ELFT‐EFS is part of NIST’s Evaluation of Latent Fingerprint Technology 

program.

• ELFT‐EFS evaluates the net accuracy of latent matching when using fe

marked by experienced human latent fingerprint examiners.

• ELFT‐EFS is not per se a test of automatic EFS extraction (e.g. conform• When image data are included in a search, automatic feature extracti

used to any degree participants choose (e.g. in addition to, or in place 

specified features)

• A key result of ELFT‐EFS is to measure the performance of differing ”le

manually encoded feature‐based searches, and because such markup 

determine when image‐only searching is sufficient

3 
• ELFT‐EFS Evaluation #1 (2009‐2011)

ELFT‐EFS Testing	‐‐ Background• ELFT‐EFS Public Challenge (2009)

– Anonymous open‐book self‐reported practice test on small datase– Reported in Appendix to ELFT‐EFS Evaluation #1 final report– Technology evaluation using participant provided software, run o

hardware using sequestered datasets from multiple sources

– Goal was to determine near‐term benefits (“as currently impleme

long‐term feasibility or accuracy

– Final report (NISTIR 7775) published March 2011

• ELFT‐EFS Evaluation #2 (2010‐2012)

– Same test plan and data as Evaluation #1 (with minor exceptions)

– Evaluation #1 participants (and new participants) given chance to 

algorithms

– Final report (NISTIR 7859) published May 2012

4 
Caveats

• The performance impact of any specific feature as measured in the th

be limited for several reasons:

– participants may not have yet developed approaches to utilize pro

features

– limited opportunity due to limited presence of such features in th

– human markup of such features may be ineffective for automated 

• The results may not be applicable to other datasets and operational s

different processing constraints.

• The relative performance of image‐based and feature‐based matchin

affected by differences in system resources (cost); image‐only proces

requires much greater resources.

5 
Data

6 
Sources of latent images

Source

Name

# Latent

Images

# Distinct

Fingers

# Subjects Description

Casework 1 368 368 272 Operational casework images

Casework 2 165 165 163 Operational casework images

WVU 440 440 383 Laboratory collected images

FLDS 93 93 15 Laboratory collected images

MLDS 38 38 4 Laboratory collected images (small 

releasable images for examples in r

Total 1066 1066 826

7 
Latents

• All images were 8‐bit grayscale, 1000ppi, uncompressed

• All latent images and their associated features were contained in ANS

records (no system specific transaction files used)

• Finger position was never specified

• Note: NIST is always receptive to more good latent data

8 
Examiner feature markup

• The latent IAFIS/EFS features were marked by 21 IAI certified latent p

examiners (CLPE)

• Features were defined in ANSI/NIST‐ITL 1‐2011, in the Extended Featu

and markup guidelines developed in “EFS Markup Instructions”

• EFS markup included: ridge quality maps, creases, dots, incipient ridg

protrusions, and pores, minutiae with ridge counts, cores, deltas, and 

• A subset of latents had skeletons marked (including associated ridge • No vendor‐specific rules for feature encoding were used ; all encoding 

compliance with the EFS specification

• Features were marked in latent images without reference to exempla

the GroundTruth (GT) dataset

• Which features to use was left entirely to the participants (suppliers o

likely that all available features were not used by participants

9 
Latent Feature Subsets

Subset Description Imag

LA Image only With Image

LB Region of Interest (ROI) With Image

LC ROI, Pattern Class, Quality Map With ImageLD Minutiae with ridge counts With ImageLE Extended features (no Skeleton) With ImageLF Extended features with Skeleton With Image

LG Minutiae with ridge counts (equivalent to

IAFIS LFFS)

Without Ima

10 
Examples of image markup

11 
ELFT‐EFS Latent Datasets

• Baseline

– 1,066 latents and associated feature markup

– Used only for subsets LA (image only), LE (image + EFS) and LG (m

• Baseline‐QA

– 418 latent subset of Baseline which underwent additional QA revi

– Used for all subsets

– All latents have EFS skeleton (subset LF) available

• GT (Ground Truth)

– Variation of the markup of the latents from Baseline‐QA

– Used only for subsets LA, LE, and LG

– GT has “ideal” minutiae markup performed with reference to the 

12 
Exemplars

Exemplar subset # subjects Description

E1 100,000 10 rolled & 10 plain impressions each

E2 10,000 10 rolled impressions each

E3 10,000 10 plain impressions each

13 
Exemplars

• All exemplars are 8‐bit grayscale, 500ppi, compressed with WSQ

• All exemplars are in ANSI/NIST files

• Background (non‐mated exemplars) images from operational data

– Sourced from ink and optical livescan; rolled and slap impressi

– Plains segmented (by auto‐segmentation) from slaps

• Foreground (mated exemplars)

– Plains segmented using manual review and correction from sla– Association with latent not determined by AFIS

• No features included with the exemplars at time of enrollment

14 
Test Procedures

15 
Latent Matching Software

• Each participant submitted a set of SDKs (Software Development Kits

provided the interfaces defined by the ELFT‐EFS‐1 API

• Each participant submitted

– one SDK for exemplar feature extraction and exemplar enrollmen

– one SDK for latent feature extraction

– one SDK for latent 1‐to‐N search

• SDKs were permitted to be Linux or Windows, sequential or multithre

utilize either 32 or 64‐bit execution mode.

• All tests were run on NIST hardware (array of blade servers)

• All searches returned a candidate list with a fixed length of one hundr

candidates.

16 
Timing

Exemplar featureextraction

100 seconds per 10‐finger exemplar set (rolled 

pre‐segmented slap)

Latent enroll 120 seconds per latent

Search For feature searches (subsets LC‐LG):

25ms/exemplar set, per latent, per core

(400 single finger matches/sec; an exemplar se

consists of either 10 rolled or 10 plain prints)

For image searches (subsets LA‐LB):50ms/exemplar set, per latent, per core

(200 single finger matches/sec)

17 
Rank‐based Results

18 
ELFT‐EFS Evaluation #2 Participants

• A: SAGEM MorphoTrak

• B: NEC• C: 3M Cogent

• D: Sonda Technologies• E: Warwick Warp• F: SPEX Forensics
Image only

20 
Image + Extended feature set (no skeleton

21 
IAFIS minutiae + ridge counts (no image)

22 
Summary of rank‐1 results

(searching E1: 100k roll+slap subjects; latent dataset Base

1st place 2nd place 3rd place 

Latent Subset

LA LE 

Image only Image + EFS Minu

A 67.2 70.2 B 63.0 69.9 C 65.0 71.4 D 38.9 n/a 

E 49.2 52.3 

23 
Summary of rank‐1 results

(searching E1: 100k roll+slap subjects; latent dataset Base

24 
Summary of rank‐1 results

(searching E1: 100k roll+slap subjects; latent dataset Baseli

1st place 2nd place 3rd place 

Latent Subset

LA LB LC LD LE LF 

Image only Image +

ROI

Image +

ROI +

Pattern

Class +

Qual map

Image +

Minutiae +

Ridge

Counts

Image +

EFS

Image +

EFS +

Skeleton

A 63.4 64.1 64.1 65.6 65.6 64

B 57.7 60.1 60.1 67.0 67.0 68

C 59.6 60.1 58.6 66.3 67.2 n/

D 31.8 23.9 n/a n/a n/a n/

E 44.0 46.9 47.1 46.9 47.1 48

25 
Summary of rank‐1 results

(searching E1: 100k roll+slap subjects; latent dataset Baseli

26 
Summary of Rank 1 Results

• Minutiae‐only searching performance:

– Minutiae‐only < Image‐only < Image+Features — for all matcher

• Image‐only searching:

– Cogent and Sonda improved about 13 – 16% over Evaluation #1

• Image + Features searching:

– Best performance overall was 71.4% using image + EFS

– The addition of features to the image (Minutiae or EFS) provided 

gains of 3‐7% for the Baseline (1,066 latent) dataset

– Apparently, Sagem & NEC did not utilize the additional EFS featur

performance)

– Skeletons (LF) were only beneficial to NEC and Warwick (and mad

performance worse for Sagem)

27 
Score‐based results

28 
LA: Image only

29 
LE: Image + EFS

30 
LG: IAFIS Minutiae + ridge counts (no imag

31 
Image + Extended feature set + skeleton

32 
Summary of Score‐based Results

• Score‐based results

– are more scalable than rank‐based results, and show how accu

affected by an increase in gallery size

– show feasibility of automated candidate list reduction– show feasibility of improved candidate list management for re

searches (i.e. unsolved latent file)

• At 1% FPIR only 1 in 100 candidate lists will contain a false candida

typically the candidate list is either empty, or contains only one ca

• At 1% FPIR, identification rates for Baseline drop a average 9.0 pct

drop was for Image Only, greatest for Minutiae Only.

33 
Effect of “Ground Truth” markup by S

34 
Matcher A: Effect of ground truth markup

35 
Matcher B: Effect of ground truth markup

36 
Matcher C: Effect of ground truth markup

37 
Summary of Effect of “Ground Truth” mark

• These results

• show the difference between an operationally infeasible ideal mark

markup by latent examiners

• provide a measure of the impact of latent examiner variation in ma

on search performance

• The “GT” results were beneficial for SDKs A/B/C using latent subset LE, 

dramatically beneficial for participants A/B/C/E using latent subset LG

• For latent subset LE the difference in hit rate between Baseline‐QA and 

to about 4‐6%

• For latent subtest LG the difference in hit rate was about 12‐15%: the d

between the markups had a direct impact on accuracy, since the match

recourse to the image

38 
Effect of Using Rolled vs. Flat Exemp

Types

39 
Image only

40 
Image only

41 
Image only

42 
Image + EFS

43 
Image + EFS

44 
Image + EFS

45 
Minutiae Only

46 
Minutiae Only

47 
Minutiae Only

48 
Summary of Rolled vs. Plain Performanc

(E1, E2 & E3; Baseline latent dataset; rank 1; SDK C

Latent Subset

LA LE 

Image only Image + EFS M

Rolls+Plains (E1) 65.0 71.4 Rolls Only (E2) 58.4 63.0 

Plains Only (E3) 52.3 55.2 

49 
Effect of “Value” Determination

50 
Effect of “Value” Determination

All No value Limited

value

Value

Count 1066 25 113 917

LA(Imageonly)

A 67.2% 20.0% 34.5% 72.6%

B 63.0% 8.0% 28.3% 68.4%C 65.0% 8.0% 30.1% 70.8%D 38.9% 4.0% 4.4% 44.1%

E 49.2% 0.0% 10.6% 55.0%

LE (Image +EFS)

A 70.2% 20.0% 35.4% 75.9%

B 69.9% 12.0% 31.0% 76.2%

C 71.4% 20.0% 35.4% 77.1%

E 52.3% 0.0% 17.7% 57.9%

LG(Minutiaeonly)

A 45.1% 4.0% 6.2% 51.2%B 49.8% 0.0% 4.4% 56.8%C 49.3% 0.0% 7.1% 55.8%E 0.0% 0.0% 0.0% 0.0%

51 
Additional Findings

 17.8% of latents in the test were missed by all matchers (at rank 1) — 

these could be individualized by a certified latent examiner

 17.6% of latents in the test could not be individualized by a certified la

examiner— nearly half of these could be matched at rank 1 by one or 

matchers

 Best matcher was 71% accurate vs. 82% “collective accuracy” – sugges

improvement

 90‐94% of identifications recorded at rank 1 for the top 3 matchers

52 
ELFT‐EFS Conclusions

 EFS features effective as an interoperable feature set

 Identical markup was used successfully by all vendors EFS features usually result in a net gain

 Marginal performance boost from non‐minutiae features

 Accuracy of Image+features > Image‐only > Features‐only

 Accuracy of Image‐only ~= image+features for higher quality images Unexpected ability to identify low feature content latents in certain ca

 Elimination of weak candidates via score thresholding could reduce w

significantly with minimal loss of accuracy

 Examiner assessed latent quality and minutiae count reasonably pred

performance

53 
Future Work

• Future ELFT evaluations (in planning)

 Latent palmprint matching

 Latent fingerprint matching at higher throughput (to be more rep

of operations)

 Latent to latent matching

 Reverse latent search (unsolved latent file)

• Latent Interoperability Transmission Specification (LITS)

– proposed standardized latent search transactions based on EFS

• Best practices

– Search and encoding strategies for latent examiners

– When to use image only searches, other search profiles

– What are best strategies for encoding palmprint latents

• Latent quality metrics

54 
For More Information

Web  fingerprint.nist.gov/latent

Email	  latent‐efs@nist.gov

55