Performance Without Conformance? Value 

of Level 1, 2 and 3 Conformance

Fernando L. Podio

Computer Security Division (CSD)

NIST/ITL

IBPC 2012
• The existence of biometric standards alone is not 

enough to demonstrate that products meet the technical 

requirements specified in the standards. 

• Conformance testing:

• Captures the technical description of a specification and 

measures whether an implementation faithfully implements 

the specification.

• Supports increasing levels of confidence in product quality 

and increases the probability of successful interoperability 

and data interchange.

Conformance to Biometric Standards
• Can the lack of conformance to biometric standards 

(data interchange formats in this discussion) also 

seriously influence operational access to the 

biometric data and/or deter the system from 

achieving the expected performance? 

Conformance to Biometric Standards

• Conformance testing plays the 

role of the canary in the coal 

mine warning on possible issues 

related with the expected level of 

performance.
First Generation (G1)

19794-1:2006 – Part 1: Framework

-2:

2005

Finger 

minutiae /

AMD 1 

:2010 

(minutia 

descript.) 

-3:

2006 

Finger 

pattern 

spectral 

-4:

2005

Finger 

image

-5:

2005

Face 

image /

AMD 

2:2009 

(3D)

-6:

2005

Iris image

-7:

2007

Signature 

sign 

-8:

2006

Finger 

pattern 

skeletal

-9:

2007

Vascular 

image

-10:

2007

Hand 

geometry 

silhoutte

-2:

2010

-4:

2010

-6:

2011

-7:

2011

-8:

2011

-9:

2011

G1

Parts in binary encoding

ISO/IEC JTC 1 SC 37 Biometric Data interchange Formats 

and Related Conformance Testing Methodology Standards 

29109-1:2009 – Part 1: Generalized conformance testing methodology (CTM)

-5:

2011/1st

rev DIS/

2nd rev

CD

-10:

2010

From Dr. Busch’s BCC 2010 Presentation: “Status and Trends for Biometric Data Interchange Formats Standardization” and SC 37/WG 3 Roadmap

AMD: Amendment

WD: Working Draft

CD: Committee Draft 

DIS: Draft International Standard 

CTMs: separate documents

-2:

201x

4th

WD

Level 3 (semantic) CTM
Christoph Busch 

-13:

201x

WD

-2:

201x

3rd

PDAM1

-4: 

201x

2nd

DAM1

-5:

201x

PDAM1

-6:

201x

3rd

PDAM1

-8:

201x

PDAM1

-9:

201x

DAM 1

-11:

201x

3rd

PDAM1

-14:

201x

2nd

PDAM 1 

G2

19794-1rev Amd.1 Conformance testing methodology (bin. encoding) - DAM 1 

WD: Working Draft / CD: Committee Draft / DIS: Draft International Standard / FDIS: Final Draft International Standard

PDAM: Proposed Draft Amendment / DAM: Draft Amendment 

Second Generation (G2)

-2:

2011

Finger 

minutiae

-4:

2011

Finger 

image

-5:

2011

Face 

image

-6:

2011

Iris 

image

-7:

201x

3rd CD

Signature

Sign time 

series

-8:

2011

Finger 

pattern 

skeletal

-9:

2011

Vascular 

image

-11:

201x

DIS

Signature 

sign proc. 

dynamic

Updated from Dr. Busch’s BCC 2010 Presentation: “Status and Trends for Biometric Data Interchange Formats Standardization” and SC 37/WG 3 

Roadmap

-13:

201x

2nd WD 

Voice

-14:

201x

DIS

DNA

19794-1rev:2011, Part 1 Framework 19794-1:2011 AMD 2 Framework XML - PDAM

-2:

2011

1st WD

AMD 2

-4:

2011

1st WD

AMD 2

-5:

2011

1st WD

AMD 2

-6:

2011

1st WD

AMD 2

-7:

2011

1st WD

AMD 1 

-9:

2011

1st WD

AMD 2

Binary encoding

XML encoding

ISO/IEC JTC 1 SC 37 Biometric Data interchange Formats 

and Related Conformance Testing Methodology Standards 
Hierarchy of Conformance Tests for 

Biometric Data Interchange Formats

• Conformance Testing Methodology standards include:

• Summary of the conformance test requirements in the 

data format standard that indicates what level of 

conformance testing is applicable to each requirement. 

• Three levels of conformance testing are defined: 

• Level 1 conformance testing (L1)

• Level 2 conformance testing (L2)

• Level 3 conformance testing (L3)

• Conformance test assertions to be tested for the 

corresponding requirements.
Conformance Test Suites (CTSs) Designed to Test 

Implementation of Biometric Data Interchange Format Standards

7 

ISO/IEC 19794-x – 1G

-2:

2005

Finger 

minutiae

-4:

2005

Finger 

image

-5:

2005

Face 

image



ISO/IEC 19794-x rev – 2G

-4rev:

2011

Finger 

image

-5rev:

2011

Face 

image

-6rev:

2011

Iris 

image

-13:

20xx

WD

Voice

Base Standard

CTS Planned, Under 

Dev., or Developed   

-2rev:

2011

Finger 

minutiae

-14:

20xx

5th CD

DNA

Under 

Dev.

Under 

Dev.

. 

 Planned Planned

CTS Installer version 

released

CTS source Code 

version released

Under 

Dev.



CTS for standard 

Profile (PIV) – 

Installer version 

Off-card 

Beta



On-card 

Beta



 

  

Standards developed 

by JTC1/SC37

--

--
Level 1 (L1) Conformance Testing 

• L1 testing checks field by field and byte by byte 

conformance with the standard, both in terms of 

ranges, values and, as required, cardinality. 

• Most values are used to determine which L2/L3 

tests to run.

• If no L1 processing is performed, L2 tests (and in 

some cases) L3 tests may not be performed.
Representation

• The standard defines a field for 

Finger Position (e.g., left index 

finger, right index and middle ring). 

• Allowed values are: 0 - 10, 13 - 15, 

and 40 - 50. 

• If the value in the field would be 51, 

a decision statement would have no 

L2 tests to perform for 51.

• Any use of the record / further tests 

based on finger position may be 

jeopardized by this error. 

General Header

• Format Identifier

• Version Number

• Length of Record

• Number of Finger Representations

• Device Certification Block Flag

Representation

• Representation Length

• Capture Date and Time

• Capture Device Technology Identifier, 

Capture Device Vendor Identifier, 

Capture Device Type Identifier

• Quality and Certification Blocks

• Finger Position (51)

• Representation Number

• Image Spatial Sample Rate (horiz and 

vert) 

• Impression Type

• Size of Scanned Image in X-Dir and in YDir

• Minutiae Field Length

• Ridge Ending Type

• Number of Minutiae

• Minutia Type

• Minutia X, Minutia Y and Angle Ø

• Minutia Quality

• Extended Data Block Length

• Extended Data Area Type Code

• Extended Data Area Length

• Extended data

L1 Testing – 2G Finger Minutiae Data Format (19794-

2:2011) - Error Example & Possible Impact 
Level 2 (L2) Conformance Testing

• L2 conformance testing checks the internal consistency 

of a data format implementation under test (IUT), 

relating values from one part of the IUT to values from 

other parts of the IUT. 

• L2 tests involve interactions between multiple values 

from different parts of the IUT and sometimes are 

derived from implicit assumptions that are not explicitly 

stated in a standard. 

• L2 tests require more complex validation than L1 tests.

• Example: Length (e.g., Record, Representation, 

Extended Data Area). Length errors can create havoc 

in L1, L2 and L3 testing and record parsing/processing.
Representation General Header

• Format Identifier

• Version Number

• Record Length

• Number of Representations

• Certification Flag

• Number of Eyes Represented = 0 

(laterality is unknown)

• Representation Length

• Capture Date and Time

• Capture Device Technology Identifier

• Quality Block

• Representation Number

• Eye Label = 1 

(SUBJECT_EYE_LABEL_RIGHT = 1)

• Image Type

• Image Format

• Iris Image Property Bit

• Image Width

• Image Height

• Bit Depth

• Range

• Roll Angle of Eye

• Roll Angle Uncertainty

• Iris Center Smallest X

• Iris Center Largest X

• Iris Center Smallest Y

• Iris Center Largest Y

• Iris Diameter Smallest

• Iris Diameter Largest

• Image Data Length

• Iris Image Data (Right eye)

L2 Testing – 2G Iris Data Format (19794-6:2011)

Error Example & Possible Impact 

• Inconsistency between the Number 

of Eyes Represented and the Eye 

Label field in the representation.

• The General Header specifies that 

the Number of Eyes Represented is 

0; meaning the record should ONLY 

contains representations where the 

Eye Label = 0 (Undefined). 

• A DB query to select a specific 

subset of records may disregard this 

record. Do you need it?
Representation

• Example assumes that the system is designed 

to handle records that are not expected to 

have any representation with extended data so 

it assumes the length of the image to be: 

Representation Length – Length of 

(Representation Header) 

• The “Length of the image” could be incorrect:

• How robust the record/image processing is?

• Is the image detected using “magic bytes”?

• Is the image processed correctly?

• Was L2 testing performed to detect possible 

errors/inconsistencies?

General Header

• Format Identifier

• Version Number

• Record Length

• Number of Finger/Palm Representations

• Certification Flag

• Number of Distinct Finger/Palm Positions

Extended Data

• Representation Length = 2000

• Capture Date and Time

• Capture Device Technology Identifier

• Capture Device Vendor Identifier

• Capture Device Type Identifier

• Quality Block

• Certification Block

• Finger/Palm position

• Representation Number

• Scale Units

• Scan Spatial Sampling Rate (horiz) 

• Scan Spatial Sampling Rate (vert) 

• Image Spatial Sampling Rate (horiz) 

• Image Spatial Sampling Rate (vert) 

• Bit Depth

• Image Compression Algorithm

• Impression Type

• Horizontal Line Length

• Vertical Line Length

• Image Data Length = 1500

• Image (compressed)

L2 Testing - 2G Finger Image Data Format (19794-4:2011) 

– Possible Impact Due to Unexpected Extended Data
Representation

• The Image Format field was erroneously 

filled out as 0x0AHex, specifying that the 

image is a JPEG2000 format; however 

the image inserted is a PNG image 

(Image Format of 0x0EHex).

• If the system uses the Image Format 

field as directions on how to parse the 

Image Data, once attempting to parse 

the PNG as a JPEG2000, it finds it 

“incompatible” and may not use the 

record. 

• Was L2 testing performed to detect this 

inconsistency?

General Header

• Format Identifier

• Version Number

• Record Length

• Number of Representations

• Certification Flag

• Number of Eyes Represented

• Representation Length

• Capture Date and Time

• Capture Device Technology Identifier

• Quality Block

• Representation Number

• Eye Label

• Image Type

• Image Format = 0x0AHex (JPEG2000)

• Iris Image Property Bit

• Image Width

• Image Height

• Bit Depth

• Range

• Roll Angle of Eye

• Roll Angle Uncertainty

• Iris Center Smallest X

• Iris Center Largest X

• Iris Center Smallest Y

• Iris Center Largest Y

• Iris Diameter Smallest

• Iris Diameter Largest

• Image Data Length

• Iris Image Data = PNG Image

L2 Testing – 2G Iris Data Format (19794-6:2011)

Image Format Specified vs. Iris Image Data Format
Level 3 (L3) Conformance Testing (Semantic ) 

• L3 conformance testing checks if the biometric data 

implementation is a faithful representation of the input 

biometric data record or “parent biometric data” and 

ensures requirements are satisfied that are not merely 

“L1 & L2” requirements.

• The input biometric data record may be (e.g.,):

• An image

• Raw sensor output such as a time series of data points from 

a digitization tablet.

• For some requirements in a standard, L3 testing might 

be significantly difficult or even impossible to test. 
“L3” Standard Requirements/Error – Examples

• 19794-4 :2011 (finger image data format):

• “Each record shall pertain to a single subject.”

• Incorrectly labeled fingerprint (labeling a Right thumb as a 

Right index finger).

• 19794-5:2005 (face image data format):

• Pose Angle: “The (3 multi-byte) Pose Angle field (BY, BP, BR) 

shall represent the estimate or measure pose of the subject in 

the image.” 

• 19794-2 (finger minutiae format)

• See detailed discussion in: ”The Benefit of Ground Truth Data to 

Semantic Conformance Testing of Fingerprint Minutia Encoding”, Dr. 

Christoph BuschFraunhofer IGD / Hochschule Darmstadt, Germany

http://biometrics.nist.gov/cs_links/ibpc2010/pdfs/Busch_Christoph_IBPC2010-gtm-100224.pdf
Representation

General Header

• Format Identifier

• Version Number

• Record Length

• Number of Representations

• Certification Flag

• Number of Eyes Represented = 1 

(left or right eye is known to be present)

• Representation Length

• Capture Date and Time

• Capture Device Technology Identifier

• Quality Block

• Representation Number

• Eye Label = 1 

(SUBJECT_EYE_LABEL_RIGHT = 1)

• Image Type

• Image Format (Raw, JPEG2000,PNG)

• Iris Image Property Bit

• Image Width

• Image Height

• Bit Depth

• Range

• Roll Angle of Eye

• Roll Angle Uncertainty

• Iris Center Smallest X

• Iris Center Largest X

• Iris Center Smallest Y

• Iris Center Largest Y

• Iris Diameter Smallest

• Iris Diameter Largest

• Image Data Length

• Iris Image Data (Left eye)

L3 Testing – 2G Iris Data Format (19794-6:2011)

Error Example & Possible Impact 

• Discrepancy between Eye 

Label indicating that the 

representation is on a right 

eye and the iris image data 

(which is of a left eye). 

• Is there an impact on 

recognition performance?
• Conformance testing plays the role of the canary in the 

coal mine: lack of conformance to the required standard(s) 

can, in many cases, jeopardize the expected recognition 

performance or prevent access to the data (as well as 

impact the overall operational performance).

• Implementers may handle non-conformant records in 

different ways during processing.

• All levels of conformance testing are important and 

necessary. Whenever possible, in addition to L1 and L2 

tests, L3 tests should be performed.

• More research on L3 testing procedures is necessary.

Conclusions on Conformance Testing
Contact 

Fernando L. Podio

NIST/ITL CSD

Phone number: 301-975-2947

email: fernando.podio@nist.gov 

Background slides:

• Role of ISO/IEC SC 37 on biometric data formats

• Role of NIST/ITL in support of these standards

• ANSI/NIST-ITL 1-2011 Implementation Conformance Error 

Example and NIST/ITL Conformance Test Tools Roadmap

• Additional Conclusions 
• Scope:

• “Standardization of generic biometric technologies pertaining to 

human beings to support interoperability and data interchange 

among applications and systems. Generic human biometric 

standards include: common file frameworks; biometric application 

programming interfaces; biometric data interchange formats; 

related biometric profiles; application of evaluation criteria to 

biometric technologies; methodologies for performance testing 

and reporting and cross jurisdictional and societal aspects.

• SC 37, in particular, is responsible for the development of 

biometric data interchange format standards and the 

associated conformance testing methodologies (CTMs) 

for these biometric data formats.

ISO/IEC JTC 1/SC 37 (SC 37)
• NIST/ITL participates in the development of 

biometric (CTM) standards and other conformity 

assessment efforts. 

• It also develops Conformance Test Architectures and 

Test Suites designed to test implementations of:

• International (SC 37) biometric data interchange formats

• ANSI/NIST-ITL Standards (developed with NIST/ITL’s 

leadership), focus on Law Enforcement, Military, 

Intelligence, Border Management and Homeland Security 

applications.

NIST/ITL’s Support for Conformance Testing
Type 1

Type 10 Face, 

other body 

part, or scar, 

mark tattoo 

(SMT) image

10.001 

10.002 

... 

... 

... 

... 

... 

... 

... 

... 

10.999 

10.999 

Error Example - Multiple Tagged Image Fields in an 

ANSI/NIST-ITL 1-2011 Implementation 

• There are two Fields 10.999 

in the Type 10 Record. 

• There should not be more 

than one field 10.999.

• To process the image:

– Would you access the last 

field of the Record or

– Would you scan the 

Records until you find a 999 

Field and stop (since there 

is supposed to be only 

one)?

• Is the biometric data contains 

in the 10.999 the same?

• Which Field is supposed to 

be used in matching? 

• The advantage of performing 

L2 / L3 tests before matching 

is apparent.

10.999 contains the biometric data 

associated with the Record.

 
Christoph Busch 

Conformance Test Suites Designed to Test Implementations of 

ANSI/NIST-ITL (AN-20XX) Standards

22

ANSI/NIST-ITL 1-2007

Target: CTA/CTS for selected Record 

Types: 1 (Transaction information), 4 

(High-resolution grayscale fingerprint 

image), 10 (Facial and SMT image), 13 

(Variable-resolution latent image), 14 

(Variable-resolution fingerprint image), 

and 17 (Iris image)

ANSI/NIST-ITL 1-2011

Target: CTA/CTS for Data Conventions, 

Information Associated with several 

Record Types, and selected Record 

Types: 1, 4, 10, 13, 14, 15 (Variableresolution palm print image record), 17, 

and Traditional Encoding

Base Standard

CTS Planned, 

Under Dev., or 

Developed

CTS Installer 

version released

CTS source 

Code version 

released

CTS for Profiles 

of the AN-20xx 

Standards 

Required Conformance Test 

Architecture Changes From Previous 

Version Made 



Evaluating users’ needs

Code under development

Under consideration

Evaluating users’ needs

Other 

Record 

Types?

Evaluating 

need

Evaluating 

need

V. 0.5

V. 0.5 
• For many applications, suppliers declaration of 

conformance may be acceptable since other approaches 

to conformance testing (e.g. testing by third-party 

accredited labs) tend to be more expensive.

• However, the public availability of conformance test tools 

may help end-users interested in requiring conformance 

to biometric standards gain a higher level of confidence in 

products. 

• These test tools, can also support product developers 

interested in using the same testing tools available to endusers and can also benefit testing labs interested in 

offering this type of service to developers and end-users. 

Additional Conclusions