Testing Trajectories

J.L. Wayman

JLWayman@aol.com

International Biometric Performance 

Conference 2012 1
Your Favorite Topic Here

TIME ⇒

Level of Knowledge

2 International Biometric Performance 

Conference 2012

“State of the Art”
Your Favorite Topic Here

TIME ⇒

Level of Knowledge

3 International Biometric Performance 

Conference 2012

“State of the Art”

Requirement

“Gap” /

“Challenge”
Your Favorite Topic Here

TIME ⇒

Level of Knowledge

4 International Biometric Performance 

Conference 2012

“State of the Art”

“ROADMAP”
Your Favorite Topic Here

Level of Knowledge

TIME ⇒

5 International Biometric Performance 

Conference 2012

“State of the Art”

TRAJECTORIES
Is Scientific Knowledge Cumulative?

• Contentious issue in the late 20th Century

• IMHO: Yes, but not on a local, short-term level.

• In general, we know more now than before, but....

• In specific areas, over short periods, we know less.

• We learn, then forget or discard

• Why?

– Our agenda is determined in a social context:

• Fads

• Economics

• Perceived needs

• Chance of fame and fortune ...

– We change and our values change

6 International Biometric Performance 

Conference 2012
A Partial List on References for this Talk

• Ernest Nagel, The Structure of Science, 1961

• Thomas Kuhn, The Structure of Scientific 

Revolutions, 1962

• Ian Hacking, Representing and Intervening, 

1982

• Ernan McMillan (ed.), The Social Dimensions 

of Scientific Knowledge, 1992

• Elizabeth Anderson, “Feminist Epistemology: 

An interpretation and a defense”, 1995

International Biometric Performance 

Conference 2012 7
Things We Used to Know, But Have 

Forgotten

• Retinal scanning

• Facial thermography

• Finger circumference 

• Comparative modality testing

• Credit Card applications

8 International Biometric Performance 

Conference 2012
International Biometric Performance 

Conference 2012

9
Two Upward Testing Trajectories 

• Vulnerability assessment

• Estimating large-scale system performance

10 International Biometric Performance 

Conference 2012
Vulnerability Assessment –

“Pre-History”

See Geller B, Almog J, Margot P, Springer E. (1999). 

A Chronological Review of Fingerprint Forgery, 

Journal of Forensic Sciences; 44, 5, 963-968

– Ehmer G., Ein Gaunertrick gegen die 

Daktyloskopie, Arch Kriminal—Anthropol 

Kriminalistik 1909;200.

– Carlson M., Fingerprint can be forged. Virginia Law 

Register 1920;5.

– Lee HC, Easy to detect finger forgeries. Fingerprint 

Magazine 1928;10(Pt3):12–3.

11 International Biometric Performance 

Conference 2012
Media Interest – 1970s

• “Diamonds are Forever” 1971

12 International Biometric Performance 

Conference 2012
Vulnerability Assessment – 1970s

• R.C. Lummis and A. Rosenberg “Test of an Automatic 

Speaker Verification method with intensively trained 

mimics”, JASA (51), p.131(A), 1972

• Kibbler, G.O.T.H, “Evaluation of the Identimat 2000 

Hand Geometry Identifier”, Mitre Corp., Oct., 1972

• D.E. Raphael and J.R. Young, “Automated Personal 

Identification”, SRI, International (1974)

• “Guidelines for Evaluation of Techniques for 

Automated Personal Identification”, Federal 

Information Processing Standards Publication 48, 

NBS, April 1977

13 International Biometric Performance 

Conference 2012
RFC Systems – mid-1970s

• Work on “live/drunk/drugged finger” detection

International Biometric Performance 

Conference 2012 14
1980s -Vulnerability Assessment Goes 

Classified

• The Secret Art

Yes, art -- “The absence of evidence is not evidence of 

absence”

– Sandia: George Ingram and Blackhat Analysis

– Mitre

– SRI

• Identix commerical use of “live finger” detection 

based on red -> white color shift with finger pressure

– Touchblock – ??? (nobody remembers)

15 International Biometric Performance 

Conference 2012
Vulnerability Assessment -1990s

• J. Daugman, T-PAMI, 1993

• Schiphol Travel Pass

• vander Putte and Keuning

• German Federal Office of 

Information Security (BSI)

• 3M Federal Systems

– Osten, Carim, Arneson, and Blan, 

“Biometric, personal 

authentication system”, Feb 17, 

1998 U.S. Patent 5719950

• Missing: US National Biometric 

Test Center

16 International Biometric Performance 

Conference 2012
NSA Goes Public with Vulnerability 

Assessment

17 International Biometric Performance 

Conference 2012

CBS “60 Minutes II”, Jan. 24, 2001
International Biometric Performance 

Conference 2012 18
Vulnerability Assessment -- 2000s

• ANSI X9.84 - 2003, “Biometrics Management and Security 

For The Financial Services Industry”

• Fraunhofer

• CESG

• Common Criteria Recognition Arrangement, ISO/IEC 

15408: 1998

– Biometric Protection Profiles

• CESG

• BSI

• BMO

• ISO/IEC 19792:2009 “Security Evaluation of Biometrics”

19 International Biometric Performance 

Conference 2012
Right Here, Right Now

• Multiple representational frameworks

– BSI

– Fraunhofer

– Common Criteria

– ISO/IEC 30107

– Biometrics Institute

• Satellite Workshop (Friday): Artefact, Liveness, 

and Suspicious Presentation Detection

International Biometric Performance 

Conference 2012 20
Trajectory 2: Estimating performance 

of Large-scale Systems

• Reductionist approach for Philippines SSS, 

1997

– The model

“Under the simplifying, but approximate, assumption 

of statistical independence of all errors, (the) 

independent variables are bin error rate, penetration 

rate, sample-template (‘genuine’) and ‘impostor’ 

distance distributions, number of active templates or 

user models in the database, N, and the number of 

samples submitted for each transaction, M”

– When N = 1, equations must degenerate to 

“verification” system.

21 International Biometric Performance 

Conference 2012
Bernoulli Assumptions, Binomial

Results

22 International Biometric Performance 

Conference 2012
Estimating the Parameters (forgotten)

23 International Biometric Performance 

Conference 2012
Estimating the Parameters (remembered)

24 International Biometric Performance 

Conference 2012
Finger Variability

25 International Biometric Performance 

Conference 2012
Penetration Rate Correlations

26 International Biometric Performance 

Conference 2012
Further Reduction

• M. E. Schuckers, “Using the beta-binomial distribution to assess performance of a 

biometric identification device,” International Journal of Image and Graphics, 2003.

• Because ....each individual will have their own probability of success, then p, the 

usual binomial parameter for probability of success, is not the same for each user. 

Thus, the binomial is not appropriate for assessing the performance... when 

combining outcomes from multiple users. Consequently, we need a model that 

allows for variability in the probability of success among individuals and that allows 

for the possibility that trials by a given individual are not independent. One such 

model is the Beta-binomial model or, more formally, the product Beta binomial.

• Where there are n individuals tested m times and α,β are parameters of the Beta 

distribution of p among the individuals 27 International Biometric Performance 

Conference 2012
An Empirical Approach by NIST 

• Wilson, et al, “Fingerprint Vendor Technology 

Evaluation 2003”, NISTIR 7123, June 2004

• “Some biometric models assume that the false 

accept rate grows linearly with gallery size when true 

accept rate is kept constant. This assumption was 

tested by comparing the results of verification and 

open-set identification ROCs.” (Not quite correct)

• “......Figure 19 and Figure 20 is consistent with the 

observation that the false accept rate grows linearly 

with gallery size, and the true accept rate remains 

constant.”

International Biometric Performance 

Conference 2012 28
A Listing of Alternative 

Representations

• Jarosz, Fondeur, Dupré, “Large-scale Identification 

System Design” (2005)

1. Extrapolation from experience

2. Identification as succession of N verifications

3. Extrapolation from extreme value

4. Extrapolation when distance can be modeled

The influence of classification on reductionist models:

FMR = f (binning)

29 International Biometric Performance 

Conference 2012
Additional Work

• Schuckers, ME “A parametric correlation 

framework for the statistical evaluation and 

estimation of biometric-based classification 

performance in a single environment,” IEEE 

Transactions on Information Forensics and 

Security 4 (2009), 231-241

• Walter Scheirer, Anderson Rocha, Ross Micheals, 

and Terrance Boult, “Meta-Recognition: The 

Theory and Practice of Recognition Score 

Analysis”, IEEE T-PAMI, Nov. 2010

30 International Biometric Performance 

Conference 2012
Right Here, Right Now

Which representation?

• Michael Schuckers, “Scaling of Biometric False Match 

Rates Using Extreme Value Theory”

• Brian DeCann and Arun Ross, “Modeling an 

Anonymous Identification System”

• Patrick Grother, “Evaluation of 1:N Recognition 

Algorithms: IREX III and MBE 2010 Methods + Analysis”

• Srikanth Nadhamuni, “Very Large Scale Multimodal 

Testing Methods + Results for India's UID System”

International Biometric Performance 

Conference 2012 31
A Forward and Upward Trajectory

• Scientific knowledge is not strictly cumulative, but depends 

on our (us, right here, right now) continuity of interest, 

clarity of vision and openness to what is possible, necessary, 

fruitful.

• We pursue dead ends and hot leads with equal gusto

– Which is which will be left to our future historians

• We don’t always remember what we already know

– History is a great teacher!

• Sometimes it all comes together for lasting progress, but 

often along competing tracks

• Two areas ready for advancement this week: vulnerability 

assessment, understanding large-scale performance 

International Biometric Performance 

Conference 2012 32
International Biometric Performance 

Conference 2012 33