Beyond current testing standards: 

A framework for evaluating human-sensor interaction

Interna(onal Biometric Performance Tes(ng Conference 2010    |   March 2‐4, 2010 

Eric P. Kukula, Ph.D. 

Visiting Assistant Professor & Senior Biometric Researcher

kukula@purdue.edu 

Stephen J. Elliott, Ph.D.

BSPA Lab Director & Associate Professor 

elliott@purdue.edu 

Biometric Standards, Performance, and Assurance Laboratory | www.bspalabs.org

Purdue University, Department of Industrial Technology 
Agenda 

• Current Testing Standards and Norms 

•The Missing Link? 

• What performance evaluations should also explain 

• Usability & Biometrics: Our systems should be usable? 

• The Human-Biometric Sensor Interaction (HBSI) 

• HBSI Framework 

•Applications & Uses 

•Questions 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
Scope 

From Mansfield & Grother’s The Wide World of 

Biometric Testing 

Have tests been driven by 

VS 

What should be done 

what can be done 

• Observe and count • Measure FRR after data 

collection mispresentation effects 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 

A General Biometric System Model (ISO/IEC 19795‐1) 
What do our tesEng standards say? 

• Distinctions between technology and scenario

evaluations according to ISO/IEC 19795-2: 

Type of Test Technology Scenario 

Objective of 

Test 

Typical 

metrics 

Measure performance of 

algorithm(s) on a standardized 

corpus 

•Most error rates [FMR, FNMR, 

FTE, FTA] 

•Not end-to-end throughput 

•Good for large-scale identification 

system performance where difficult 

to assemble large test crew 

Measure performance of endto-end system in simulated 

application 

•Predicted end-to-end 

throughput 

•FMR, FNMR, FTE, FTA, 

GFAR, GFRR 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
What do our tesEng standards say? 

Table 6 - Registry of Biometric Performance Testing Methodology Standards 

Domain of Applicability Recommended Standard 

Physical and logical access control tests (1) ISO/IEC 19795-1:2005 

(2) ISO/IEC 19795-2:2006 

Testing of performance and interoperability of (1) ISO/IEC 19795-1:2005 

cross-supplier implementations generating and (2) ISO/IEC 19795-4:2008 

matching instances of standardized biometric 

data interchange data 

•What about the: 

• Environment 

• ISO/IEC 1st WD 29197, Evaluation methodology for environmental 

influence in biometric systems 

• Human-Sensor Interaction 

• ... 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
What Performance Evalua(ons Should Also Explain

•Is the algorithm the cause of matching errors? 

•Is the application or environment the problem? 

•Is the design of the sensor the problem? 

•Are the users/agents causing the issue? 

• Can users/agents do what the system/sensor is asking for? 

• Do users/agents understand how to use the system/sensor? 

• Can users/agents produce repeatable images? 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
Usability & Biometrics “Usability” 

•Usability 

•The extent to which a product can be used by specified users to 

achieve specified goals with effectiveness, efficiency and 

satisfaction in a specified context of use (ISO 9241-11:1998, ISO/ 

IEC 25062:2006) 

•Failure to Acquire (FTA) 

• Traditional measure of “usability” in biometrics 

• Proportion of verification or identification attempts for which the 

system fails to capture or locate an image or signal of sufficient 

quality (ISO/IEC 19795-1) 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
The Human‐Biometric Sensor InteracEon (HBSI) 

• Derived from multiple research fields to better

understand and evaluate overall functionality

and performance of a biometric system 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
HBSI Framework for Biometric InteracEons 

• Objective 

• Classify every human-sensor interaction “event” with the 

resulting biometric system “reaction” 

• Event + Reaction = HBSI episode 

• Purpose 

• Understand and classify all interactions / movements / 

behaviors that occur with a biometric device to improve 

performance, quality, and usability 

• Examines a biometric system from 2 perspectives: 

• User 

• Biometric System 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
HBSI Episodes 

Event Reaction 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 

HBSI Framework for Biometric InteracEons 
Technology EvaluaEons & Offline Analysis 

• How do we analyze: 

• Previously collected data 

• Black-box systems 

• Can we calculate: 

• FTE 

• FTA 

• ... 

• Modifications 

• Failure to Extract (FTX) 

• Capable of handling black-box systems 

• Non-Biometrically Captured Data (NBC) 

• Analyzing data not originally collected for 

biometric systems. 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
ApplicaEon & Use of the HBSI Framework 

•Government & Integrators 

• System development 

• Field Readiness 

• Training 

•Vendors 

• Alpha & Beta Testing 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
Future Work 

•Evaluate more modalities with the framework 

• physical-interactive 

• image-based 

• behavioral 

• Refinement of the metrics 

•T&E Standard Methodology? 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
HBSI PublicaEons 

Available on: http://www.bspalabs.org/publications 

S. Elliott and E. Kukula, “A definitional framework for the human biometric 

sensor interaction model,” in 

Proc. SPIE Symposium on Defense, Security, Sensing: Biometric

Technology Human Identification VII Conf., Orlando, FL, April 5-9, 2010. 

E. Kukula, M. Sutton, and S. Elliott, “ 

The Human-Biometric Sensor Interaction Evaluation Method: Biometric 

Performance and Usability Measurements,” IEEE Transactions on 

Instrumentation and Measurement, vol. 59 , no. 4, Apr 2010. doi: 10.1109/ 

TIM.2009.2037878. p. 784-791 

E. Kukula, “Design and evaluation of the human-biometric sensor interaction 

method,” Ph.D. dissertation, Purdue Univ., West Lafayette, IN, 2008. 

             Biometric Standards, Performance & Assurance Laboratory  |  www.bspalabs.org 
 

 

Thank you for your a,en.on.  

Ques.ons? 

Contact Information: 

BSPA Laboratory | www.bspalabs.org Eric P. Kukula, Ph.D. Purdue University, Knoy Hall of Technology Visiting Assistant Professor & 401 North Grant Street Senior Biometric Researcher West Lafayette, IN 47907-2021 kukula@purdue.edu Phone: (765) 494-2311 

Stephen J. Elliott, Ph.D. Fax: (765) 496-2700

Associate Professor & 

BSPA Lab Director 

elliott@purdue.edu