From Ground Truth 

to Semantic Conformance Testing 

Demonstrated by the Example of Face Image Data 

René Salamon (BSI) 

Peter Ebinger (IGD) 

Federal Office for Information Security (BSI) 

Fraunhofer Institute for Computer Graphics Research IGD 

IBPC2010 / 03-04-2009, Gaithersburg (USA) 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
 

Agenda 

□ Conformance Testing 

□ Ground Truth 

□ Practical Approach 

□ Conclusion 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Conformance Testing

- Principles -

TOE 

features 

use case 

functionalities 

input data output data

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
TOE 

features 

use case 

functionalities 

input data 

usage 

well defined 

well defined 

well defined well defined 

output data

well defined 

behaviour 

Conformance Testing

- Requirements -

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Conformance Testing

- Specification Requirements -

Input Data 

Property 1 

range of reference values point (valid & (“zero point“) invalid) 

measure / 

scale . . . 

Property n 

range of reference values point (valid & (“zero point“) invalid)

. . . 

measure / 

scale . . . 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Conformance Testing

- Test Data Requirements -

Test Data 

Property 1 

position 

relative to validity reference (Yes / No) point 

(“value“) 

measuring 

method . . . 

Property n 

position 

relative to validity reference (Yes / No) point 

(“value“)

measuring 

method . . . 

. . . 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Conformance Testing

- Test Data Requirements -

Test Data 

Property 1 

measuring 

method . . . 

validity

 (Yes / No) 

position 

relative to 

reference 

point 

(“value“) . . . 

Property n 

measuring 

method . . . 

validity

 (Yes / No) 

position 

relative to 

reference 

point 

(“value“) 

? 

? 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
 

 

Ground Truth 

- Definition -

Definition of Ground Truth in Biometrics 

“Ground truth“ is reliable biometric data captured 

within a defined setup with known parameters 

and combined with additional metadata that 

describes the properties of the biometric data 

determined by defined and documented 

mechanisms and/or scientific experts. 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Ground Truth 

- Measurability -

Characteristics of Standardized Properties 

“Soft“ Properties 

□ no well defined 

reference/zero point 

□ undefined range 

“Strict“ Properties 

□ well known & accepted 

reference/zero point 

□ defined range 

measureable not measureable 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Ground Truth 

- Measurability Requirements -

□ measure has to be close to reality 

□ measure has to be internationally reproducible 

□ measure has to be applicable 

□ reference points have to be clearly seperated from each 

other 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
 

Practical Approach

- Project Idea -

ISO Standards, ICAO, BSI TR 

(ISO/IEC 19794-5, ISO/IEC 29109-5, ICAIO TAG MRTD/NTWG, BSI TR-03104) 

Experience with facial 

image data processing

software 

Requirements of 

developers and users 

of biometric systems

FIReBIRD 

Quality assurance and improvement of 

facial image data processing systems 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
 

Practical Approach

- Realization -

Acquisition of valid and invalid images 

(according to ISO/IEC 19794-5) 

E. g. in respect to 

□ head gear, sun glasses, eye patches 

□ pose angles (pitch, yaw and roll), expressions 

□ lighting, shadows, under and over exposure, focus 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Practical Approach

- Color Measurement -

Eye Colour 

□ 5 colour classes 

(brown, grey-brown, 

multiple, gray-blue, blue) 

□ 3 colour depths 

(dark, medium, bright) 

Hair Colour 

□ 5 colour classes 

(black, brown, 

blond,grey, red) 

□ 3 colour depths 

(dark, medium, bright) 

Skin Colour 

in evaluation 

synthetic eyes synthetic hair spectrophotometer 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Practical Approach

- Eye Color Table -

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Practical Approach

- Hair Color Ring -

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Practical Approach

- Miscellaneous Data -

Shape 

□ head shape 

□ eye shape 

□ line of mouth 

□ ... 

Personal Data 

□ class of age 

□ gender 

Miscellaneous 

□ glasses 

□ hairdo 

□ makeup 

□ ... 

shemes checklist checklist 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Practical Approach

- Handling “Soft“ Properties I -

Pose Variation – Looking for the Zero Point 

□ extreme pose variation = killer for facial image processing 

□ absolute zero point not defined 

□ aid: Frankfurt Plane 

□ problem: usually covered feature points 

□ standard has yet no answer 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
 

Practical Approach

- Handling “Soft“ Properties II -

15 x SLR cameras 

□ 1 central camera for full frontal view 

□ 8 cameras for vertical variation (pitch):± 4, ± 10, ± 20 and ±45 

degrees 

□ 6 cameras für horizontal variation (yaw): ± 4, ± 10 and ± 20 

degrees 

3 x 3D scanner 

□ complete scan of the whole face 

□ no holes due to shadowing effects or occlusions 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Conclusion 

- Quality Of Testing -

Quality of testing depends on ... 

□ quality / accuracy of specifications (standards, guidelines) 

□ quality of test methods 

□ reliability of test data. 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Conclusion 

- Precision In Specification -

Lacking precision in specification causes ... 

□ less exact measurements 

□ vague conformance estimations 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Conclusion 

- Lack Of Measure -

Lack of measure hinders ... 

□ development of conformant systems 

□ production of conformant data 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Conclusion 

- “Zero Point“ -

A scientifically definied “zero point“ is needed to ... 

□ become a property measurable 

□ get a defined scale for measurement 

□ get comparable measurements 

□ determine whether a property is in line with a standard 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Conclusion 

- Standardization Demands -

Standardization has (at least) to define ... 

□ a scale 

□ a reference point (or “zero point“) 

□ value ranges 

for every property / feature it deals with. 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Thank You 

For Your 

Attention ! 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Contact 

René Salamon 

Bundesamt für Sicherheit in der 

Informationstechnik 

Godesberger Allee 185-189 

D-53175 Bonn 

Tel.: +49 228 – 9582 5142 

eMail: rene.salamon@bsi.bund.de 

Peter Ebinger 

Fraunhofer Institut Graphische 

Datenverarbeitung 

Fraunhoferstrasse 5 

D-64283 Darmstadt 

Tel.: +49 6151 – 155 531 

eMail: peter.ebinger@igd.fraunhofer.de 

R. Salamon, P. Ebinger IBPC2010 04-03-2010 
Partners 

U.S. National Institute of 

Standards and Technology 

R. Salamon, P. Ebinger IBPC2010 04-03-2010