NIST INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE – Tes'ng for Emerging Modali'es  Interna'onal Biometric Group  Page 2 

• Longitudinal perspec1ves on performance tes1ng for 

emerging biometrics 

• Differences in tes1ng emerging vs. established modali1es 

• Lessons learned  
NIST INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE – Tes'ng for Emerging Modali'es  Interna'onal Biometric Group  Page 3 

• A novel modality, or a modality with limited deployment / test 

history (e.g. keystroke dynamics) 

• A technology that takes a substan1ally new approach to an 

established modality (e.g. ultrasonic fingerprint)  

• Emerging is a maDer of perspec1ve 

– In 2000, iris recogni1on was commercialized, but s1ll emerging  

– Palm vein and finger vein were considered emerging in North America 

in 2006, notwithstanding extensive deployment in Japan  

– Can be prototype, pre‐commercial, or early‐stage commercial 

• Ambiguity on how to conduct performance tes1ng  

– How should subjects use devices (do best prac1ces exist)? 

– How is sample data obtained from the sensor / system? 

– How is the matcher implemented? 

– How are results analyzed? 
• Net Nanny BioPassword (2000) 

Transac'onal  – Scenario evalua1on, real‐1me matching 

FNMR FMR – 200 subjects, ~2000 comparisons  

19.91%  1.30% 

• AuthenWare AuthenTest (2009) 

– Scenario evalua1on, real‐1me matching  Transac'onal 

– Addi1onal offline post‐processing  FNMR FMR

– 500 subjects, ~7000 comparisons  3.20%  3.26% 

• Observa1ons

– Needed regular access to a trained, controlled popula1on 

– Ideally tested in situ due to emulate impact of keyboards, input devices 

– Tradi1onal biometric terminology – samples, templates, comparison 

scores – not directly applicable to this technology  

– How to obtain quan1ty of signatures sufficient to build robust models? 

– Are trained typists more prone to false matching? 

– Test results may substan/ally understate actual performance 

NIST INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE – Tes'ng for Emerging Modali'es  Interna'onal Biometric Group  Page 4 
• Fujitsu PalmSecure (2006) 

– Hybrid scenario/technology evalua1on 

– Online capture, offline matching 

– ~650 subjects, 2 posi1ons,  

~22k samples,  ~50m comparisons 

Transac'onal  FNMR  FMR 

Same‐Visit  0.57%  0.056% 

Cross‐Visit  0.69%  0.063% 

• Observa1ons 

– Vendor needed tuning samples from subjects of specific ethnici1es  

– Female error rates were an order of magnitude higher than male 

– Housing / cradle was a prototype designed for standalone tes1ng 

– Most difficul1es related to capture instruc1ons 

– Offline matching took weeks – matcher never implemented for volume 

– Lack of granular thresholds reduced visibility into performance  

– While considered emerging, technology was in many ways mature

NIST INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE – Tes'ng for Emerging Modali'es  Interna'onal Biometric Group  Page 5 
• Hitachi UBReader TS‐E3F1 (2006) 

– Hybrid scenario/technology evalua1on 

– Online capture, offline matching 

– ~650 subjects, 2 posi1ons,  

~22k samples,  ~25m comparisons 

Transac'onal  FNMR @ 0.10% FMR 

Same‐Visit 0.34% 

Cross‐Visit 1.94% 

FNMR @ 0.01% FMR 

0.68% 

2.77% 

• Observa1ons 

– Vendor had to port matching capability from on‐card to server‐based 

– Vendor modified na1ve recogni1on sample capture to acquire a 14‐15 

second image stream for each capture aDempt  

– Capture logic relies on mul1‐pass quality assessments, such that typical 

capture behavior for genuine and impostor captures different  

– Tester implemented “BeDer‐instance” matching logic  

– Crea/ve problem solving o<en required to test emerging modali/es  

NIST INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE – Tes'ng for Emerging Modali'es  Interna'onal Biometric Group  Page 6 
Transac'onal  FNMR @ 0.10% FMR  FNMR @ 0.01% FMR 

Same‐Visit 0.067%  0.067% 

Cross‐Visit 0.372%  0.661% 

• TST Biometrics BiRD 3 (2009) 

– Hybrid scenario/technology evalua1on 

– Online capture, offline matching 

– ~500 subjects, 6 posi1ons,  

~37k samples,  ~32m comparisons 

• Observa1ons 

– Example of an emerging approach to an established modality 

– Vendor used results to support engineering decisions (e.g. default 

thresholds, interoperability with contact sensors) 

– Presenta1on dura1on considerably longer than for contact devices  

– Interoperability with contact systems essen1al 

– When appropriate, emulate governing principles from mainstream 

modali/es 

NIST INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE – Tes'ng for Emerging Modali'es  Interna'onal Biometric Group  Page 7 
NIST INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE – Tes'ng for Emerging Modali'es  Interna'onal Biometric Group  Page 8 

• GenKey (2007) 

– Technology evalua1on 

– Offline matching (and  

a separate online test) 

– ~1200 subjects, ~20k samples,  

 ~4.5m comparisons 

– Numerous exploratory tests conducted

• Observa1ons 

– Tes1ng intended to assess the viability of (paper) token issuance from 

a workflow perspec1ve; matching secondary to the concept 

– Experimenta1on with different thresholds necessary to get into the 

vicinity of sensible match scores  

– Vendor primarily interested from a marke1ng perspec1ve 

– Tes/ng was meant to validate what the vendor already knew 
NIST INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE – Tes'ng for Emerging Modali'es  Interna'onal Biometric Group  Page 9 

• Aprilis Holosensor (2005‐6) 

– Hybrid scenario/technology evalua1on

– Online capture, offline matching 

– ~650 subjects, 4 posi1ons,  

~9k samples, ~32m comparisons 

– EER ~20%

  

• Observa1ons 

– Company disintegrated, more or less, over the course of the evalua1on 

– Thus, no matching technology (used NBIS Bozorth) and no support 

– Technology on the extreme side of prototype‐to‐commercial spectrum 

– Had to develop best prac1ces for presenta1on on the fly 

– In retrospect, we needed a way to evaluate pure imaging capabili1es 

(e.g. fidelity) without reliance on matching 

– Some/mes modali/es are emerging for a reason 
NIST INTERNATIONAL BIOMETRICPERFORMANCE CONFERENCE – Tes'ng for Emerging Modali'es  Interna'onal Biometric Group  Page 10 

• Tes1ng typically assumes a consulta1ve aspect 

• Vendors expect feedback on how to improve their technology  

– Tester may be iden1fying and solving problems that the vendor has not 

seen or an1cipated 

• Bugs in development somware, libraries 

• Results may be generated solely for the vendor 

• While test approaches should reflect relevant best prac1ces, 

flexibility and crea1vity may be required to accommodate 

novel technology aspects 

• Allow for trial and error, more dry run tes1ng, further  

explora1on of parameters that may impact quality