Testing Laboratory 

Interactions for Evaluating 

Biometrics 

Raul Sanchez-Reillo, 

J. Uriarte-Antonio, I. Tomeo-Reyes, B. Fernandez-Saavedra 

Carlos III University of Madrid – Scientific Park 

c/ Gregorio Peces Barba, 1; 28918 - Leganes (Madrid) - SPAIN 

e-mail: {rsreillo, juriarte, itomeo, mbfernan}@ing.uc3m.es 
●

●

●

IBPC 2010 Outline NIST

● Introduction 

•Accredited Evaluations 

•Quality Assurance in Evaluations 

Evaluations in Biometrics 

•Is there any difference from other product 

evaluations? 

Proposals for Interoperable Evaluations 

•Conformance 

•Performance 

Conclusions and Future working lines 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 2 
Introduction IBPC 2010 

NIST 

● Nowadays most of products can be certified to comply 

with certain rules 

Introduction ● In order to accepted, those certifications should either:

• Follow a defined and recognized scheme 

Evaluations in • Being issued by accredited laboratories that evaluate those 

Biometrics products following ISO 17025, and all relevant international 

standards 

Interoperable ● Main advantage in using an accredited laboratory is Evaluations 

that they provide evaluations that are: 

• Interoperable Conclusions 

• Traceable 

• Repeatable 

● i.e.: a certificate issued by an accredited laboratory 

shall be accepted by any other certification scheme 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 3 
Introduction IBPC 2010 

NIST 

● An accredited laboratory, in order to maintain 

its accreditation shall demonstrate periodically 

Introduction its technical capability and evaluation quality 

● For keeping evaluation quality, a lab shall: Evaluations in 

Biometrics • Maintain all tools calibrated 

•Perform auto-analysis, i.e. repeat an evaluation 

Interoperable 

Evaluations and reach the same results 

•Perform intercomparisons with other laboratories, 

Conclusions i.e. the same test is given to another laboratory, as 

to see if they reach the same result. 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 4 
● There are currently few Labs that carry on Evaluations 

of Biometric products 

• As soon as Society, Industry and/or Administrations start 

requesting certified products, there will be the need of more 

Evaluations in Biometrics IBPC 2010 

NIST 

Introduction 

Evaluations 

in 

Biometrics 

Interoperable 

Evaluations 

Conclusions 

labs to start business 

● Several types of evaluations can be carried out in

Biometrics, among them we can find: 

• Conformance Testing 

• BioAPI Conformance (ISO/IEC 24709-x) 

• Data format Conformance (ISO/IEC 29109-x and rev.19794-x) 

• Performance Testing (ISO/IEC 19795-x) 

• Technology Evaluations 

• Scenario Evaluations 

• Operational Evaluations 

• Security 

• Following Common Criteria 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 5 
IBPC 2010 Evaluations in Biometrics NIST 

● So far, Biometrics do not seem to present much of a difference 

from other technologies 

• We currently have published standard that can be used as the basis 

Introduction for the certification 

● Unfortunately there are major differences that make it difficult to 

Evaluations have Accredited Labs for Biometrics 

in 

Biometrics • Testing databases are not interchangeable 

• Previous experiences have shown that the same algorithm can perform 

differently depending on the database used 

Interoperable • Data protection laws difficult the exchange of biometric data in many Evaluations countries 

• Variability in human properties make it difficult to create “standard 

Conclusions databases” 

• Scenario and operational evaluations require end-users to act 

• Data sets will be different for auto-analysis and intercomparisons. 

• As it is a rather new technology, there are not certified tools to be 

acquired 

• Some of such tools could be Test-DBs 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 6 
IBPC 2010 Evaluations in Biometrics NIST 

Introduction 

Evaluations 

in 

Biometrics 

Interoperable 

Evaluations 

Conclusions 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 7 
IBPC 2010 

NIST 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 8 

Interoperable Evaluations 

● Conformance Evaluation: 

•Problems arise with intercomparisons in Data 

Format Conformance 

•Two approaches: 

• Send the data records to be analyzed by the other 

laboratory 

o Serious agreements between the labs as to comply with 

both, data protection laws and audit requirements 

• Use a certified conformance testing program 

o Is there any available? 

Scenario and Operational Evaluations: 

•A really difficult point: 

• Not possible to exchange the “data set” 

• Not even possible to “save” the “data set” used 

●

Introduction 

Evaluations in 

Biometrics 

Inter -

operable 

Evaluations 

Conclusions 
IBPC 2010 

NIST 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 9 

Interoperable Evaluations 

● Technology Evaluation: 

• A clear and detailed methodology is defined 

• But Test Databases are the difficult point 

• ISO/IEC 19795-1 says 

• “For technology testing, a generic application and population 

may be envisioned, ensuring that the tests are neither too hard 

nor too easy for the systems being evaluated” 

o Due to variability of human properties, can we find an interoperable 

definition for generic? 

o How can we measure the difficulty of a database? 

• Work is being carried out within ISO/IEC 29198 in SC37 for 

fingerprints 

• But, in case we can build a Standard Test Database ... 

• How can we deal with Data Protection Laws? 

• Nowadays there are several databases around the globe 

• Some of them public, some private 

• So, why not a change in the approach? 

Introduction 

Evaluations in 

Biometrics 

Inter -

operable 

Evaluations 

Conclusions 
IBPC 2010 

NIST 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 10 

Interoperable Evaluations 

ARES 

BM 

BSP1

BSP2

BSPN

RM 

BioAPI 

19795 

Introduction 

Evaluations in 

Biometrics 

Inter -

operable 

Evaluations 

Conclusions 

● Why not keeping DBs secured and in a single place, 

and executing the evaluation remotely? 

● ARES is an Automatic Remote Evaluation System that allows 

developers/laboratories to test their BSPs/TOEs with non-directlyaccessible DBs. 
IBPC 2010 

NIST 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 11 

ARES: General Architecture 

Introduction 

Evaluations in 

Biometrics 

Inter -

operable 

Evaluations 

Conclusions 
IBPC 2010 

NIST 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 12 

AES: General Architecture 

BTPEM

DBPI1 DBPI2 DBPIN

BioAPI 

Framework BioAPI 

Layer 

S 

P 

BSP Load I BSPDB

Application 

Evaluation

Application 

RMIEM

Report 

Generator 

BSPi

RMIEI

BTPEI

EI 

EM 

Introduction 

Evaluations in 

Biometrics 

Inter -

operable 

Evaluations 

Conclusions 
IBPC 2010 

NIST 

Raul Sanchez-Reillo – IDTestingLab (UC3M) 13 

●

●

●

●

●

Introduction 

Evaluations in 

Biometrics 

Interoperable 

Evaluations 

Conclusions 

Certification of Biometric products are about to be 

requested 

Laboratories should be ready to perform those 

evaluations 

But tools shall be available 

• Certified tools for conformance testing 

• ARES can be a possibility for Performance Testing (dealing 

with Technology Evaluations) 

International legal procedures should be developed as 

to be able to exchange personal data 

Quality for Scenario and Operational Evaluations can 

only be audited by the procedure followed, not by the 

results obtained 

Conclusions 
Thank you!!

Questions?

Raul Sanchez-Reillo, 

J. Uriarte-Antonio, I. Tomeo-Reyes, B. Fernandez-Saavedra 

Carlos III University of Madrid – Scientific Park 

c/ Gregorio Peces Barba, 1; 28918 - Leganes (Madrid) - SPAIN 

e-mail: {rsreillo, juriarte, itomeo, mbfernan}@ing.uc3m.es