A novel framework for 

evaluation of ID photo quality

Hiroyuki Suzuki† Shizuo Sakamoto†† Takashi Miyai††† 

Kazuya Nakano† Masafumi Takeda†

† Tokyo Institute of Technology †† NEC Corporation

††† New Media Development Association

International Biometric Performance Testing Conference 2012

March 8, 2012, Gaithersburg
IBPC2012 Mar.8 2012

Outline

• Motivation

– Existing ways to evaluate ID photo quality and problems

– A concept of developing a novel framework for ID photo 

quality evaluation

• Method

– How to design an evaluation function for ID photo quality

• Experiments

– A paired comparison method is applied to the proposed 

framework

– Classification experiments are conducted

• Conclusions

2/21
IBPC2012 Mar.8 2012

Ways to make an ID photo

Handmade using digital camera and 

printer for home use

Specialized shop 

for ID photo ID photo booth

In the case of handmade, the ID photo quality varies considerably.

Good quality blurred Improper layout Weak printing

3/21
IBPC2012 Mar.8 2012

Factors and Standard of ID photo 

quality

• Factors of ID photo quality

– Photographing conditions

• Layout, yaw angle, hairstyle, shadow, accessory, etc.

– Printing and display

• Size, Position, Brightness, Color, Contrast, etc.

– Digital data format

• Number of pixels, Bits per pixel, Compression method, File format, etc. 

• Standardization of ID photo quality

– ISO/IEC 19794-5

• defines what a good quality ID photo is.

– Some of the evaluation values have to be determined by subjective factors of 

human inspector.

– ISO/IEC TR 29794-5

• Provides supporting information on ID photo quality

– It provides some specific examples for ID photo quality evaluation, but it is not 

enough to evaluate ID photo appropriately.

4/21
IBPC2012 Mar.8 2012

Existing Ways to evaluate ID photo 

quality

• Subjective evaluation by screening experts

– A mainstream way of ID photo quality evaluation

– It requires manpower

• Automatic evaluation using evaluation software

– The reliability of evaluation results is not so high

Photographing

condition

Out of Focus

Expected varied 

value

Actual

varied value

Sharpness

Eye Open

Eye Gaze Frontal

Eye Tinted

Gray Scale Density

Smile Mouth Closed

Mouth Closed

Eye Tinted

Gray Scale Density

Hot Spot

Table. An example of evaluation values by evaluation software

5/21
IBPC2012 Mar.8 2012

Purpose

• To develop a framework for evaluation of ID 

photo quality, which can output appropriate 

evaluation values that are equivalent to those 

provided by experts.

Evaluation 

Function 

Good!!

Design based on 

leaning NG×

algorithm

ID photo screening 

expert

Good!!

NG×

Same 

result

6/21
IBPC2012 Mar.8 2012

Subject in this study

• To examine the possibility of applying the 

proposed framework to a paired comparison 

method.

Right is better!!

Left is better!!

A pair of ID photo

Evaluation 

Function Design based on 

leaning 

algorithm

ID photo screening 

expert

Right is better!!

Left is better!!

7/21
IBPC2012 Mar.8 2012

Diagram to design evaluation 

function

ID photo

Extract feature 

values from 

facial image

Feature

values

design

evaluation

function

Input

data

Output

data

ID photo ID photo ID photo

Feature

values

Feature

values

Feature

value

Compare and 

determine

which is better

ID photo Screening experts

ID

photo

ID

photo

ID

photo

ID

photo

ID

photo

ID

photo

Series of ID photo pairs

Evaluation

function

Two ID photos of same 

person’s face, 

photographed under 

different conditions

Subjective 

evaluation 

results

Commercial

software

8/21
IBPC2012 Mar.8 2012

Photographing ID photos for experiments

• 11 ID photos per a person, including one best 

practice and ten non-standard facial images

– Best practice

• In accordance with ISO/IEC 19794-5

– Non-standard

• Smile (two types), eyeglasses (two types), blurred (two types), high 

exposure, cast shadow, background shadow, hair in front of face

• Acquired from nine men and seven women

Best 

practice

Smile Glasses High exposure Cast shadow Background 

shadow

9/21
IBPC2012 Mar.8 2012

Photographing set up

10/21
IBPC2012 Mar.8 2012

Subjective experiments by experts 

and quantification

• A pair of same person’s ID photos that are photographed 

under different conditions are printed on a piece of 

photo paper.

• Four experts, who are engaged in ID photo screening 

operation, determine which ID photo is better.

Table. Quantification of subjective evaluation resu

Score Subjective evaluation result

-2 Four experts say “right ID photo is better than left one”

-1 Three experts say “right ID photo is better than left one”

One expert says “left ID photo is better than right one”

0 Two experts say “right ID photo is better than left one”

Two experts say “left ID photo is better than right one”

1 One expert says: “right ID photo is better than left one”

Three experts say “left ID photo is better than right one”

2 Four experts say “left ID photo is better than right one”

• Scores are assinged 

according to the right 

table.

12/21
IBPC2012 Mar.8 2012

Extract feature values from facial 

image

• In this experiments, output date of commercial 

software are applied as feature values of a facial 

image

– We uses three commercial software products

• Preface, Aware, Inc.

• FaceIT, L-1 Identity Solutions, Inc.

• FaceVACS, Cognitec Systems GmbH

– Several evaluation items that are closely-linked to the ID 

photo quality such as below are used for designing the 

evaluation function

• Position, yaw, eye opening level, noise, lighting uniformity, 

background uniformity, contrast

11/21
IBPC2012 Mar.8 2012

Designing evaluation function

• Evaluation function is designed based on two 

types of learning algorithm

– Neural Network (NN)

– Support vector machine

 (SVM)

13/21
IBPC2012 Mar.8 2012

Experiments

• Classification experiments using the designed evaluation 

functions are conducted

• Number of ID photo pairs

– For learning: 760 pairs (maximum)

– For classification: 780 pairs, non-overlapping with ID photos for learning

• Evaluation software products and number of feature values

– Preface (8 feature values), FaceVACS (17 feature values), FaceIT (21 

feature values)

• Learning algorithms

– NN

• Feed forward NN based on three layer perceptron

– SVM

• Three types of kernel functions, linear, polynomial, and Gaussian

14/21
IBPC2012 Mar.8 2012

Multiple classification and binary 

classification

15/21

ID photo 

Screening 

experts

ID

photo

I

ID

photo

II

Subjective 

evaluation

1

0

-1

-2

2

Score

Evaluation

function

2 1 0 -1 -2

Score

positive negative

Evaluation

function

Score

ID

photo

I

ID

photo

II

Multiple 

classification

Binary

classification

Classification by evaluation function

Classification by experts
IBPC2012 Mar.8 2012

Experimental Results

(Multi classification by SVM)

16/21

Soft A

Soft B

Soft C

Bottom right axis: output score

Bottom left axis: score of input data

Vertical axis: Classification accuracy
IBPC2012 Mar.8 2012

Experimental results

(binary classification by SVM)

17/21

0

10

20

30

40

50

60

70

80

90

100

two men

four men

six men

eight men

two women

four women

six women

one woman

and one man

two women

and two men

three women

and three men

four women

and four men

Software A

Software B

Software C

Classification accuracy [%]

Training data set for designing evaluation functions
IBPC2012 Mar.8 2012 18/21

0

10

20

30

40

50

60

70

80

90

100

two men

four men

six men

eight men

two women

four women

six women

one woman and

one man

two women and

two men

three women

and three men

four women

and four men

Software A

Software B

Software C

Experimental results

(binary classification by NN) Classification accuracy [%]

Training data set for designing evaluation functions
IBPC2012 Mar.8 2012

Comparison of SVM vs. NN

• SVN and NN is compared in maximum, average, standard 

deviation of the classification accuracy with the variety of 

the training set for designing evaluation functions. 

0

10

20

30

40

50

60

70

80

90

100

FaceIt FaceVACS PreFace

NN

SVM

0

10

20

30

40

50

60

70

80

90

100

FaceIt FaceVACS PreFace

NN

SVM

0

2

4

6

8

10

12

14

16

18

FaceIt FaceVACS PreFace

NN

SVM

Max Average

Std. dev.

:NN

:SVM

19/21

Classification accuracy

[%]

Classification accuracy

[%]

Standard deviation

[%]

Soft A Soft B Soft C Soft A Soft B Soft C

Soft A Soft B Soft C
IBPC2012 Mar.8 2012

Summary of experiments

• About 80% accuracy is obtained in binary 

classification, while the classification accuracy 

in multiple classification is not so high.

• SVM is superior to NN in terms of generalization 

capability for unknown data.

– The classification accuracy of SVM is almost constant 

regardless of the training data set, while the one of NN 

sometimes drops significantly.

• There is little difference in the classification 

accuracy between three commercial software 

products.

20/21
IBPC2012 Mar.8 2012

Conclusions

• We have proposed a framework for designing a 

evaluation function for ID photo quality

– It can output an evaluation value equivalent to those are 

provided by experts .

• The proposed framework has been applied to a 

paired comparison method

– The effectiveness has been shown by conducting 

numerical experiments.

• We plan to develop an evaluation function which 

can actually evaluate ID photo quality.

21/21
IBPC2012 Mar.8 2012

Acknowledgement

• This research has been supported by Ministry of 

Economy, Trade and Industry of Japan (FY2008-FY2009), 

and New Energy and Industrial Technology Development 

Organization (FY2010).

Thank you for you king attention!!