Face Recognition Vendor Test (FRVT)

Performance of Automated Age Estimation Algorithms

NIST Interagency Report 7995

M. Ngan and P. Grother

Information Access Division

National Institute of Standards and Technology

March 20, 2014
FRVT 1

Executive Summary

Introduction

Facial age estimation is an area of study new to the Face Recognition Vendor Test (FRVT) with Still Facial Images Track.

While peripheral to automated face recognition, it has become a growing area of research, given its potential use in various

applications. The motivation for age estimation systems has grown in the last few decades, given the rise of the digital

age and the increase in human-computer interaction. Age-based access control and verification (e.g., age verification for

alcohol/tobacco purchases), age estimation in crime and mass disaster investigation (e.g., age determination of unknown

human bodies at a crime scene to help with victim identification), age-adaptive targeted marketing (e.g., displaying agespecific advertisements from digital signage), age-invariant person identification (e.g., identifying missing children), and

age-based indexing of face images are potential applications of automated facial age estimation.

NIST performed a large scale empirical evaluation of facial age estimation algorithms, with participation from five commercial providers and one university, using three large operational datasets comprised of facial images from visas and law

enforcement mughots, leveraging a combined corpus of over 7 million images. NIST employed a lights-out, black-box testing methodology designed to model operational reality where software is shipped and used ”as-is” without algorithmic

training. Core age estimation accuracy was baselined over a large homogeneous population, then assessed demographically by age group, gender, and ethnicity. The impact of input-driven variations, namely image quality and number of

image samples per subject was captured, and assessments of age-verification accuracy and estimation accuracy in children

were documented.

Key Results

Core Accuracy and Speed: Age estimation accuracy depends strongly on the provider of the core technology. Broadly,

there is a twofold difference between the most accurate and the least accurate algorithm in terms of the percentage of

images correctly classified to within five years and mean absolute error (MAE)1. Using the most accurate age estimation

algorithm, (i.e., B31D from Cognitec), the chance of accurately estimating the age of a person within five years of their

actual age over an ethnically-homogeneous database of 6 million images is 67%, with an MAE of 4.3 years. All algorithms

can perform age estimation on a single image in less than 0.15 seconds with one server-class processor. The most accurate

algorithm, on average, performs estimation in 0.125 seconds.

The main dataset used for overall accuracy assessment is comprised of 6 million ethnically-homogeneous images. Although image collection was subject to the guidelines published by the Department of State (DoS), the images are compressed JPEG files which exhibit artifacts of JPEG compression causing reduction in image detail. With more detail available in less compressed images, age estimation performance may improve, but errors will still likely exist due to ageing

variation driven by intrinsic and extrinsic factors.

Impact of Demographic Data on Accuracy: For a heterogeneous dataset of 240 thousand images, it is empirically observed that age is more accurately estimated in males than females, with the tendency for adult females to be underestimated in age. A majority of the algorithms demonstrated lower accuracy and higher MAE on an ethnically-heterogeneous

population than a homogeneous population, which suggests that ethnicity has an impact on age estimation. South Americans tend to be overestimated in age, and Asians tend to be understimated. A majority of the algorithms estimate age

more accurately for the most operationally relevant age group, i.e., adults age 18-55. The adult age group is also where

estimation accuracy is closest among the algorithms. The majority of algorithms exhibit the highest MAE in the senior age

group, i.e., age 56-99.

These results state empirical observations for the particular dataset, but they do not determine cause. The impact of extrinsic factors potentially driving the observed results between gender and ethnicity, such as cosmetics and plastic surgery,

are not studied in this report. Further research would be required to objectively verify these conjectures.

Age Verification Accuracy: For a system with an objective to verify that a person is at least 21 years old, a 17 year-old

1For more details on cumulative score and mean absolute error, see sections 2.4.2 and 2.4.1.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
FRVT 2

has a 29% chance of passing for 21, as achieved by the most accurate algorithm (i.e., B30D from Cognitec). This falseverification percentage increases as a person gets closer to age 21. To ensure that 98% of underage individuals are detected

would falsely provide an underage result for 39% of people who are actually above age 21. The same Detection Error

Tradeoff (DET) analysis can be done for other verification ages of interest to support specific applications.

Impact of Image Quality on Accuracy2: Comparing poor quality webcam photos and better quality mugshot images, the

majority of the algorithms demonstrate lower accuracy with the webcam images, with varying rates in accuracy degradation between the algorithms. Observable biasing in the error distribution to the right is seen in poor quality webcam

images, which suggests the occurrence of overestimation. There is no clear explanation for the overestimation seen, although it may anecdotally be related to the demographics of the subjects captured in the webcam images.

Impact of Number of Image Samples on Accuracy: The FRVT Application Programming Interface (API) [11] supports

multiple still image input to the algorithm software for age estimation, which enables the analysis of age estimation performance versus the number of image samples of the same person. For contemporaneous mugshot images of the same

subject collected within a one year period, the results show MAE monotonically decreasing as the number of image samples provided increased for all algorithms. There is an improvement in MAE of about one year between one and four

input images. Age estimation times increase linearly with respect to the number of image samples, which is the expected

behavior.

Comparison against Academic Methods: A performance evaluation was done with the commonly benchmarked FGNET Aging Database [1] in an attempt to compare FRVT age estimation participants with published methods from the

academic literature. A fair performance comparison could not be made due to observed fundamental differences in testing protocol employed by academia versus NIST. The published methods from academia were tested through a protocol

which allowed the implementation to train on a subset of the images through every iteration of testing. NIST employed

a lights-out, black-box testing methodology that did not involve any type of algorithm training during evaluation, as designed to model operational reality.

2Guidelines for quality aspects of facial images are documented in ANSI/NIST-ITL 1-2011 [4], Annex E. For the study documented in this report, image

quality is broadly defined by image size, resolution, and contrast and subject illumination and pose.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
FRVT 3

Acknowledgements

The authors would like to thank the sponsors of this activity. These are the Criminal Justice Information

Systems (CJIS) division and the Biometric Center of Excellence (BCOE) of the Federal Bureau of Investigation,

the Office of Biometric Identity Management (OBIM) office in the Department of Homeland Security (DHS),

and the Science and Technology (S&T) Directorate, also in DHS.

Disclaimer

Specific hardware and software products identified in this report were used in order to perform the evaluations

described in this document. In no case does identification of any commercial product, trade name, or vendor,

imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it

imply that the products and equipment identified are necessarily the best available for the purpose.

Release Notes

. Appendices: This report is accompanied by a number of appendices which present exhaustive results on a peralgorithm basis. These are machine-generated and are included because the authors believe that visualization of

such data is broadly informative and vital to understanding the context of the report.

. Typesetting: Virtually all of the tabulated content in this report was produced automatically. This involved

the use of scripting tools to generate directly type-settable LATEX content. This improves timeliness, flexibility,

maintainability, and reduces transcription errors.

. Graphics: Many of the figures in this report were produced using Hadley Wickham’s ggplot2 [22] package running under , the capabilities of which extend beyond those evident in this document.

. Contact: Correspondence regarding this report should be directed to FRVT2012 at NIST dot GOV.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
CONTENTS FRVT 4

Contents

EXECUTIVE SUMMARY 1

ACKNOWLEDGEMENTS 3

DISCLAIMER 3

RELEASE NOTES 3

1 INTRODUCTION 6

1.1 PURPOSE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.2 APPLICATION SCENARIOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 METHODOLOGY 7

2.1 TEST ENVIRONMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 ALGORITHMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.3 IMAGE DATASET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.4 PERFORMANCE METRICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.4.1 MEAN AND MEDIAN ABSOLUTE ERROR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.4.2 CUMULATIVE SCORE (CS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.4.3 AGE VERIFICATION ERROR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 RESULTS 10

3.1 LARGE HOMOGENEOUS POPULATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1.1 ACCURACY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1.2 SPEED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.1.3 FAILURE TO COMPUTE RATE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2 AGE GROUPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.3 ETHNICITY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.4 GENDER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.5 IMAGE QUALITY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.6 MULTIPLE IMAGE SAMPLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.7 FG-NET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.8 SPECIFIC APPLICATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.8.1 AGE VERIFICATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.8.2 CHILDREN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.8.3 TIME INTERVAL ESTIMATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

APPENDIX A ADDITIONAL FIGURES AND TABLES 27

A.1 AGE ESTIMATION PERFORMANCE BY AGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

A.2 AGE VERIFICATION ACCURACY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

A.2.1 AGE 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

A.2.2 AGE 18 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

A.2.3 AGE 20 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

List of Figures

1 ACCURACY OVER HOMOGENEOUS POPULATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2 AGE ESTIMATION TIMING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3 ACCURACY BY AGE GROUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4 AGE ERROR BY ETHNICITY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5 ACCURACY BY GENDER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

6 AGE UNDERESTIMATION RATE BY GENDER AND AGE GROUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

7 WEBCAM AND MUGSHOT IMAGES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

8 ACCURACY OF WEBCAM VS MUGSHOT IMAGES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

9 MAE BY NUMBER OF IMAGE SAMPLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

10 TIMING BY NUMBER OF IMAGE SAMPLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
LIST OF TABLES FRVT 5

11 AGE VERIFICATION FOR AGE 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

12 DET FOR VERIFICATION AGE OF 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

13 AGE ESTIMATION ERROR CORRELATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

14 AGE VERIFICATION ACCURACY FOR AGE 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

15 AGE VERIFICATION ACCURACY FOR AGE 18 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

16 AGE VERIFICATION ACCURACY FOR AGE 20 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

List of Tables

1 PARTICIPANTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2 IMAGE DATASET DESCRIPTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 IMAGE DATASET SIZES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4 FAILURE TO COMPUTE RATES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5 ACCURACY WITHIN 5 YEARS BY AGE GROUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

6 MAE BY AGE GROUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

7 ACCURACY BY ETHNIC PROXY GROUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

8 MAE BY ETHNIC PROXY GROUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

9 ACCURACY ON FG-NET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

10 ACCURACY WITHIN 1 YEAR FOR CHILDREN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

11 MAE FOR CHILDREN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

12 TIME INTERVAL ESTIMATION ERROR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

13 CS(5) AND MAE FOR AGE 0-30 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

14 CS(5) AND MAE FOR AGE 31-66 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

15 CS(5) AND MAE FOR AGE 67-99 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
FRVT 6

1 Introduction

1.1 Purpose

Facial age estimation is an area of study new to the Face Recognition Vendor Test (FRVT) with Still Images Track. Automated facial age estimation is the calculation of an individual’s age by computer software based on features derived from

a person’s face image. Age estimates are a soft biometric [16] and its characterization has become a growing area of study

given its potential use in various applications. The progression of age brings changes to the appearance of the human face.

A number of factors including the variation between ethnicities and gender, as well as dependence on external factors such

as health conditions and lifestyle [9], have introduced challenges to age estimation.

The main goals of this evaluation are to:

- Provide an objective assessment of current automated age estimation technology.

- Leverage massive operational corpora. The availability of images from large populations (in the millions) supports

statistical significance of the studies. The use of operational images brings greater operational relevance to the test

results.

- Investigate age estimation accuracy across various factors, including age group, ethnicity, and gender.

1.2 Application Scenarios

The motivation for age estimation systems has grown in the last few decades given the rise of the digital age and the

increase in human-computer interaction. The process of age determination has potential application in at least the areas

described below:

Age-based access control and verification has long been a familiar concept where a person’s age is verified (e.g., ID

check) prior to physical access to a place or product being sold or virtual access to a website is granted. Examples include

preventing minors from accessing adult websites and age verification for tobacco and alcohol purchases. In Japan, ageverification cameras are installed in a number of cigarette vending machines to estimate the patron’s age prior to allowing

their purchase.

Age estimation in crime and mass disaster investigation, for example, age determination of unknown human bodies is

important in the setting of a crime investigation or a mass disaster, because the age can guide investigators to the correct

identity among a large number of possible matches. Automated facial age estimation may offer a less invasive alternative

for age determination as compared to some traditional techniques used in forensic sciences [5].

Age-adaptive human-computer interaction is on the rise given the popularity of digital signage and the opportunity for

targeted digital marketing. Advertisements targeted for a certain age group can be displayed based on the age of the

audience walking past a digital sign.

Age-based indexing of face images, that is, the use of age as criterion for indexing into large-scale biometric databases

for faster retrieval has been discussed [21] and can also apply to automatic sorting and image retrieval from digital photo

albums and the internet.

Age-invariant person identification is required in some face recognition applications where age compensation is required

(e.g., identifying missing children over time), given a significant age difference may exist between the probe and gallery

images.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
FRVT 7

2 Methodology

2.1 Test Environment

The evaluation was conducted offline at a NIST facility. Offline evaluations are attractive because they allow uniform,

fair, repeatable, and large-scale statistically robust testing. However, they do not capture all aspects of an operational

system. While this evaluation is designed to mimic operational reality as much as possible, it does not include a live

image acquisition component or any interaction with real users. Testing was performed on high-end server-class blades

running the Linux operating system. Most of the blades were 6-core machines with dual processors running at 3.47 GHz

with 192 GB of main memory. The test harness used concurrent processing to distribute workload across dozens of blades.

2.2 Algorithms

The FRVT program was open to participation worldwide. The participation window opened on July 25, 2012, and submission to the final phase for age estimation algorithms closed on October 4, 2013. There was no charge to participate.

The process and format of algorithm submissions to NIST was described in the FRVT Still Face Image and Video Concept,

Evaluation Plan and Application Programming Interface (API) document [11]. Participants provided their submissions

in the form of libraries compiled on a specified Linux kernel, which were linked against NIST’s test harness to produce

executables. NIST provided a validation package to participants to ensure that NIST’s execution of submitted libraries

produced the expected output on NIST’s test machines.

FRVT had three submission phases where participants could submit algorithms to NIST. This report documents the results

of all algorithms submitted in the final phase or the most recent submission for participants who only submitted in prior

phases.

Table 1 lists the FRVT participants who submitted algorithms for age estimation, and the alphanumeric code associated

with each of their submissions. For each participant, the algorithms are labeled numerically by chronological order of

submission. The letter codes assigned to the participants are also located at the bottom of each page for reference.

Participant Letter Code Submissions

Aug. 2012 Mar. 2013 Oct. 2013

Cognitec B B10D B20D B30D,B31D

NEC E E10D E30D,E31D,E32D

Tsinghua University F F10D F30D

MITRE K K10D

Zhuhai-Yisheng P P30D

JunYu Tech. Q Q10D

Table 1: FRVT Age Estimation Participants

2.3 Image Dataset

This report documents the use of the following datasets3:

- LEO: This dataset consists of facial images collected by various law enforcement (LEO) agencies and transmitted

to the FBI as part of various criminal record checks. The majority of images are traditional mugshot photos with a

3Operational datasets used in this study were shared with NIST only for use in biometric technology evaluations under agreements in which biometric

samples were anonymously coded by the provider; code translations were never shared with NIST; and no personally identifiable information (PII)

beyond the biometric sample was shared with NIST.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
2.3 IMAGE DATASET FRVT 8

small subset of images captured with webcams.

- DoS/P: This dataset consists of facial images for visa applicants.

- DoS/Natural: This dataset consists of facial images for non-immigrant visa applicants.

- FG-NET: This is a public dataset composed of personal photographs that is widely used for benchmarking age

estimation performance in academia.

Facial images from visa applications will henceforth be referred to as ”visa images” in this report.

The dataset properties are summarized in Table 2.

Property LEO DoS/P DoS/Natural FG-NET

Collection

Environment

Law enforcement

booking

Visa application

process

Visa application

process

Personal photos

Collection Era  ̃1960s-2008  ̃2006-2010  ̃1996-2002 Unknown

Digital, Paper Scan Digital, few paper Mostly digital Mostly digital Unknown

Documentation See NIST Special

Database 32 Vol. 1

Image size Various, 480x600,

240x240, 768x960

Most 252x300 Most 252x300 Most  ̃400x500

Compression JPEG  ̃20:1 JPEG,

mean size: 16.2kB

JPEG,

mean size: 9.2kB

JPEG,

mean size: 44.2kB

Eye to eye distance Mean = 108 pixels,

SD = 40 pixels

Median = 71 pixels Median = 71 pixels

Frontal pose Moderate control.

Known profile

images excluded.

Well controlled Well controlled Uncontrolled

Full frontal geometry Mostly not. Varying

amounts of the torso

are visible.

Yes, in most cases.

Faces are more

cropped (i.e., smaller

background) than

ISO4 Full Frontal

requires.

Yes, in most cases.

Faces are more

cropped (i.e., smaller

background) than

ISO Full Frontal

requires.

Source Operational data Operational data Operational data Public dataset

Notable Population

Characteristics

Predominantly

mugshots with a

small subset of

webcam images

Various Predominantly

Mexican

Over 50% between

age 0-13

Table 2: Image dataset descriptions.

The datasets are characterized by population sizes well in excess of all published age estimation tests (See Table 9a). The

number of images are given in Table 3.

4The International Organization of Standardization, (ISO), is an international standard-setting body composed of representatives from various national

standards organizations.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
2.4 PERFORMANCE METRICS FRVT 9

Quantity LEO DoS/P DoS/Natural FG-NET

Number of age labeled face images2378635 243023 6249313 1002

Number of subjects 1802874 222862 5738141 82

Age 18-109 0-100 0-99 0-69

Table 3: Image dataset sizes.

2.4 Performance Metrics

The following performance measures will be reported in the assessment of age estimation:

2.4.1 Mean and Median Absolute Error

Mean Absolute Error (MAE) is defined as the average of the absolute errors between the estimated ages and the actual

ages. i.e.,

MAE =

PN

k=1 |aˆk − ak|

N

, (1)

where aˆk is the estimated age for the k-th test image, ak is the corresponding ground-truthed age, and N is the number of

test images.

Median Absolute Error is defined as the median of the absolute error values, i.e., in an ordered set of increasing values,

Median Absolute Error = (N + 1

2

)

th value,

(

N

2

)

th value + ( N

2 + 1)th value

2

for Nodd , Neven respectively (2)

2.4.2 Cumulative Score (CS)

CS is defined as the percentage of test images such that the absolute error is not higher than a threshold, t, (in years). i.e.,

Given

H(x) = (

0, if x < 0

1, if x ≥ 0

,

CS(t) = (1 −

PN

k=1 H(|aˆk − ak| − t)

N

) × 100,

(3)

where aˆk is the estimated age for the k-th test image, ak is the corresponding ground-truthed age, and N is the number of

test images. ”Accuracy” is defined by CS(t), and both terms are used interchangeably in this report.

2.4.3 Age Verification Error

For age verification, the fundamental error rates for a particular verification age A is defined as:

False negative rateA(T) = Number of people equal or above age A with an age estimate < threshold, T

Number of people equal or above age A

(4)

False positive rateA(T) = Number of people below age A with an age estimate ≥ threshold, T

Number of people below age A

(5)

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
FRVT 10

A false negative would occur when a person who is older than a certain verification age A is estimated as being younger

than A. A false positive occurs when a person who is younger than A is estimated as being older than A. These error rates

are plotted as a Detection Error Tradeoff (DET) characteristic, where, for example, in an age-based access control scenario,

the rate of people younger than the minimum age requirement being granted access (i.e., the false positive rate) is traded

off against the rate of people who meet the minimum age requirement being denied access (i.e., the false negative rate).

3 Results

3.1 Age Estimation in Large Homogeneous Population

3.1.1 Accuracy

The DoS/Natural dataset contains a subset of 6,172,395 images over an ethnically-homogeneous population spanning ages

0-99, both male and female. As such, statistically significant baseline age estimation performance results can be generated,

as presented in Figure 1.

Algorithm

B30D

B31D

E30D

E31D

E32D

F30D

K10D

P30D

Q10D

Accuracy w/in 5 years

66%

67%

50%

50%

57%

34%

31%

39%

40%

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 25

Absolute Age Estimation Error (years)

Cumulative Score (%)

B30D B31D E30D E31D E32D F30D K10D P30D Q10D

(a) Cumulative score vs. Absolute age estimation error

Algorithm

B30D

B31D

E30D

E31D

E32D

F30D

K10D

P30D

Q10D

Mean

4.5

4.3

6.5

6.6

5.3

10.2

12.2

8.3

8.5

Median

3.2

3.2

5

5

4.2

7.9

9.4

6.6

6.8

(b) Mean and median of absolute age

estimation error, in years

Figure 1: Line plot showing the accuracy of algorithms at absolute error levels and table of summary error statistics generated with

6,172,395 images over an ethnically-homogeneous population.

Results and notable observations:

• For the most accurate algorithm (i.e., B31D), 67% of estimates were accurate to within five years with an MAE of 4.3

years.

• It is clear from the results that participant B’s algorithms outperform the rest. The other algorithms have substantially

lower performance numbers, with the next most accurate, which is participant E, being 10% lower in accuracy within

five years (i.e., E32D).

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.1 LARGE HOMOGENEOUS POPULATION FRVT 11

• For all of the algorithms, the mean absolute error is higher than the median absolute error, which is indicative of

skewness of the error distribution and the existence of large error values that drive the mean upward. Depending

on the algorithm, this could be a result of higher error driven by certain age groups, which is discussed in Section

3.2.

• The dataset used is comprised of visa images. Although image collection was subject to the guidelines published

by the DoS, the images are highly compressed, which reduces the amount of detail in the images. With more detail

available in less compressed images, age estimation accuracy may improve.

3.1.2 Speed

Speed could be an important performance factor in some age estimation applications where there exists a limited window

of time for a decision based on the outcome, such as when a person walks past a digital sign. The use of age as criterion

for indexing into large-scale biometric databases would levy rapid speed requirements on age estimation algorithms to

make it operationally viable.

Figure 2 presents the distribution of age estimation times for each algorithm. Age estimation time is the amount of time

elapsed computing the age estimation from pixel data of a face image. It does not include any pre-processing steps

performed by the test software such as loading the image from disk or extracting image data from a compressed JPEG file.

The timing machine was a server-class blade with a CPU running at 3.47 GHz. For more details on the testing environment,

see Section 2.1.

CS(5) = 66%

CS(5) = 67%

CS(5) = 50%

CS(5) = 50%

CS(5) = 57%

CS(5) = 34%

CS(5) = 31%

CS(5) = 39%

CS(5) = 40%

B30D

B31D

E30D

E31D

E32D

F30D

K10D

P30D

Q10D

0.00 0.05 0.10 0.15 0.20 0.25

Age Estimation Time (sec)

Algorithm

Figure 2: Boxplots of the distribution of age estimation times. Plots were generated over 5,000 age estimates. For reference, CS(5) against

a population of 6,172,395 is reported on the right.

Results and notable observations:

• Age estimation time varies considerably from one participant to another. K10D can perform estimation in less than

0.025 seconds while B31D takes about five times longer than that, on average.

• While B31D has the highest estimation times, it has the highest accuracy, while K10D, with the fastest estimation

speeds, exhibits the lowest accuracy. No clear speed-accuracy tradeoff exists between the other algorithms. B30D is

among the most accurate, but not among the slowest.

• Participant E appears to have fixed age estimation times, while exhibiting significant variation in accuracy between

its algorithms.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.2 AGE GROUPS FRVT 12

3.1.3 Failure to Compute Rate

The accuracy results presented above were computed for cases where age estimation did not fail. The error metrics do

not include a penalty for cases where an algorithm failed to generate an age estimate. Per the FRVT API [11], a failure

to compute occurs when an algorithm’s code returns a non-zero return value from a call to its age estimation function,

and hence fails to generate an age estimate. This can be a result of software issues (e.g., memory corruption), algorithmic

limitations (e.g., failure to find eyes in small images), elective refusal to process the input (e.g., image is assessed to have

insufficient quality), or specific vendor-defined failures. Table 4 presents the fraction of images for which algorithms failed

to generate an age estimate over 6,172,395 images.

Num Images

6172395

B30D

0.0003

B31D

0.0003

E30D

0.0003

E31D

0.0003

E32D

0.0003

F30D

0.0002

K10D

0.4787

P30D

0.0006

Q10D

0.0003

Table 4: Table summarizing failure to compute ratio over 6,172,395 images.

Results and notable observations:

• Eight out of nine of the algorithms have insignificant failure to compute ratios over the massive number of images

processed. While the dataset used is comprised of visa images collected under published DoS guidelines, the existence of a small number of bad images is inevitable given the operational nature of the data. Issues with images

included occlusion, closed eyes, and pathological quality.

• Participant K exhibits a significantly high failure rate, failing on approximately half of the images, with the reason

being involuntary failure to extract features from the image (as indicated in the FRVT API).

3.2 Age Groups

The facial ageing process drives different types of changes among different age groups. While facial ageing is mostly

represented by craniofacial growth in younger age groups, it is mostly represented by relatively large texture changes in

older age groups [9]. This introduces a challenge to age estimation algorithms given the ageing variation between age

groups. Accuracy across three major age groups, which are youth (0-17), adult (18-55), senior (56-99), over an ethnicallyhomogeneous population is assessed and summarized in Figure 3 and Tables 5 and 6. Given that age group definition and

composition are often application-specific, the CS(5) and MAE at each age is provided in Appendix A for reference.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.2 AGE GROUPS FRVT 13

0−17 18−55 56−99

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 0 5 10 15 20 0 5 10 15 20

Absolute Age Estimation Error (years)

Cumulative Score (%)

B30D B31D E30D E31D E32D F30D K10D P30D Q10D

Figure 3: Line plots showing the accuracy of algorithms at absolute error levels by age group. Plots were generated over an ethnicallyhomogeneous population.

Age Group

0−17

18−55

56−99

Overall

Num Images

1605807

3781607

784981

6172395

B30D

86

60

48

66

B31D

82

64

51

67

E30D

55

55

15

50

E31D

54

55

15

50

E32D

56

63

32

57

F30D

6

52

7

34

K10D

5

45

11

31

P30D

49

41

8

39

Q10D

38

42

28

40

Table 5: CS(5), in percentage, by age group and overall CS(5) from Figure 1a for reference.

Age Group

0−17

18−55

56−99

Overall

Num Images

1605807

3781607

784981

6172395

B30D

2.6

4.9

6.2

4.5

B31D

3

4.5

5.7

4.3

E30D

5.3

5.5

13.8

6.5

E31D

5.4

5.5

13.9

6.6

E32D

5.3

4.6

8.9

5.3

F30D

18.6

5.6

14.7

10.2

K10D

21

6.6

14

12.2

P30D

6.1

7.6

16.7

8.3

Q10D

10.9

7

10.9

8.5

Table 6: MAE, in years, by age group and overall MAE from Table 1b for reference.

Results and notable observations:

• All of the algorithms estimate age more accurately for a particular age group better than others, and no algorithm

has consistent MAE across all age groups.

• For the most operationally relevant age group, i.e., adults (age 18-55), algorithms are closer in performance, with

B31D and E32D being the top performers.

• Participant B’s performance is superior in the youth and senior age groups, leading the next most accurate algorithm

in 5-year accuracy by 30% and 16% respectively.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.3 ETHNICITY FRVT 14

• Algorithms exhibit lower accuracy and higher MAE in the senior age group, which could be driven by large errors

seen in the higher ages in this group.

3.3 Ethnicity

The way a person ages can depend on a number of intrinsic factors, ethnicity being an important one of them, driving

genetically inherited ageing patterns as well as extrinsic characteristics such as habitation climate and cultural behavior

choices. Methods to address age estimation across ethnicity have been published in academic studies [12, 17]. The DoS/P

dataset contains a respectable number of visa images across multiple ethnic proxies. The term ethnic proxy is used, because

an individual could be a citizen of a country but not necessarily be of that country’s ethnic descent. Ethnic proxy groups

with a minimum of at least 2,800 images were extracted and used in the analysis captured in Tables 7 and 8 and Figure 4.

B30D

B31D

E30D

E31D

E32D

F30D

K10D

P30D

Q10D

ARG

70

70

51

52

63

36

29

44

34

BRZL

67

66

47

49

60

35

29

39

34

CHIN

45

41

42

44

52

24

29

22

44

COL

63

61

47

48

57

28

25

33

41

DF

68

70

50

50

65

35

32

40

40

DOMR

58

55

41

40

51

28

26

25

37

ECUA

65

65

43

44

56

25

24

33

39

GUAT

68

67

45

44

57

25

23

37

37

IND

61

63

50

50

61

41

38

35

51

ISRL

72

72

52

53

64

34

30

41

38

KOR

46

42

50

50

58

27

24

30

52

PERU

69

71

50

48

65

37

26

47

33

PHIL

55

52

47

47

54

31

34

22

49

POL

63

63

46

46

58

31

31

36

35

RUS

57

57

42

43

48

26

31

31

36

TWAN

37

32

50

51

54

33

30

18

50

VENZ

67

67

48

47

58

27

27

32

44

Mean

61

60

47

47

58

31

29

33

41

Table 7: CS(5), in percentage, by ethnic proxy group5.

B30D

B31D

E30D

E31D

E32D

F30D

K10D

P30D

Q10D

ARG

4.1

4.1

6.8

6.7

4.9

10

10.8

8

10

BRZL

4.5

4.6

7.2

7

5.3

10.1

10.9

9.1

9.4

CHIN

6.8

7.3

8

7.7

6

12.9

10.1

12.7

7.6

COL

4.8

5

7.6

7.7

5.8

12

12.6

10.2

9.2

DF

4.1

3.9

7.1

7.1

4.7

9.8

10.8

7.9

10.3

DOMR

5.3

5.5

8.5

8.7

6.3

11.1

11.6

11.8

9.5

ECUA

4.6

4.8

8.1

7.9

6

12.9

12.9

9.9

9.9

GUAT

4.3

4.3

7.9

7.9

5.6

12.3

13.4

9.7

10.3

IND

5

4.8

7.1

7.1

5.3

8.9

8.9

9.9

8.2

ISRL

3.8

3.9

6.2

6

4.7

10.4

11.2

8.2

9.3

KOR

7.5

8.1

7.2

7

5.7

12.1

12

10.9

6.5

PERU

4.2

4

7

6.9

4.9

9.6

11.2

8.1

9

PHIL

5.7

6.1

7.1

7.1

5.7

10.4

9.7

12.2

6.9

POL

4.7

4.6

7.3

7.3

5.4

10.9

10.2

9.6

10.1

RUS

5.3

5.4

7.9

7.8

6.5

11.7

10.2

10.1

10

TWAN

8.8

9.6

7.2

6.8

6.1

11.1

10.5

14

6.8

VENZ

4.3

4.5

7.3

7.3

5.4

12.4

12

9.9

9.2

Mean

5.2

5.3

7.4

7.3

5.5

11.1

11.1

10.1

9

Table 8: MAE, in years, by ethnic proxy group5.

5DF is Mexico City, and DOMR is the Dominican Republic.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.3 ETHNICITY FRVT 15

B30D B31D E30D E31D E32D F30D K10D P30D Q10D

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

0.00

0.03

0.06

0.09

ARG BRZL CHIN COL DF DOMR ECUA GUAT IND ISRL KOR PERU PHIL POL RUS TWAN VENZ

−20−10

0

1020

−20−10

0

1020

−20−10

0

1020

−20−10

0

1020

−20−10

0

1020

−20−10

0

1020

−20−10

0

1020

−20−10

0

1020

−20−10

0

1020

Age Estimation Error (years)

Density

Figure 4: Density plots showing age estimation error across various ethnic proxies5. Plots were generated for ethnic proxy groups with

at least 2,800 images.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.4 GENDER FRVT 16

Results and notable observations:

• The average of the CS(5) and MAE across all of the ethnic proxy groups is shown in the last column of Tables 7

and 8, respectively. Compared to the performance results over an ethnically homogeneous population (Figure 1),

a majority of the algorithms demonstrate a lower mean accuracy and higher average MAE on this heterogeneous

dataset, which suggests that ethnicity has an impact on age estimation. It is interesting to note that E32D and Q10D

appear to have nearly consistent performance results between the two datasets.

• Observable bias and skew in error distribution can be seen across certain countries. Bias/skewing to the right,

which is indicative of overestimation in age, is seen in South American countries like Argentina, Brazil, and Peru.

Conversely, bias/skewing to the left indicates underestimation in age, which is observed in Asian countries such as

China, Korea, Taiwan, and the Philippines.

• There could be extrinsic factors driving the deviations seen in the error distributions, such as increased sun exposure

in the South American countries and the popularity of plastic surgery in the east Asian countries [2], although any

explanation would be anecdotal at this point in time and would require further research to solidify.

3.4 Gender

Gender is one of a number of demographic traits that drives a person’s ageing pattern, both inherently and extrinsically.

Gender has been known to impact age estimation, and methods to address age estimation across gender have been published in academic studies [12, 13]. The DoS/P dataset contains a large number of gender-labeled visa images with a

balanced number of males and females. Accuracy between males and females is assessed and summarized in Figure 5.

B30D B31D E30D

E31D E32D F30D

K10D P30D Q10D

0

10

20

30

40

50

60

70

80

90

100

0

10

20

30

40

50

60

70

80

90

100

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 0 5 10 15 20 0 5 10 15 20

Absolute Age Estimation Error (years)

Cumulative Score (%)

F M

(a) Cumulative score vs. Absolute age estimation error

Gender

Num Images

B30D

B31D

E30D

E31D

E32D

F30D

K10D

P30D

Q10D

Female

118108

5.7

5.9

8.5

8.6

6.2

11.2

11.9

11.6

9.5

Male

124894

4.7

4.7

6.1

5.8

4.8

10.2

 9.9

 8.5

8.2

(b) MAE, in years

Figure 5: Line plot showing the accuracy of algorithms at absolute error levels and table summarizing MAE by gender. Plot and table

were generated with 243,023 visa images.

Results and notable observations:

• All algorithms estimate age more accurately on males than females and exhibit higher MAE in females than males.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.5 IMAGE QUALITY FRVT 17

• Figure 6 shows the fraction of age estimates that are less than the actual age, broken out by age group and gender. A

notable observation is that much of the difference in error between male and female occurs in the adult (18-55) age

group, with females having higher understimation ratios. This could be attributed to the common use of cosmetics in

young and middle-aged adult females, although any explanation would be anecdotal and requires further research

to objectively solidify.

B30D B31D E30D E31D E32D F30D K10D P30D Q10D

0.00

0.25

0.50

0.75

1.00

Youth

Adult

Senior

Youth

Adult

Senior

Youth

Adult

Senior

Youth

Adult

Senior

Youth

Adult

Senior

Youth

Adult

Senior

Youth

Adult

Senior

Youth

Adult

Senior

Youth

Adult

Senior

Age Group

Underestimation ratio

Gender

F

M

Figure 6: Bar plots showing the fraction of age estimates that are less than the actual age by gender between youths (age 0-17), adults

(age 18-55), and seniors (age 56-99). Plots were generated with 243,023 images over a heterogeneous population.

3.5 Image Quality

The LEO dataset is a large database composed of images collected during law enforcement booking. A large portion of the

images are traditional mugshot photos captured under controlled lighting, background, and pose conditions. However,

there is a subset of low resolution webcam images captured under moderately to less controlled environment, with large

(a) Webcam (b) Mugshot

Figure 7: Examples of webcam and mugshot images from the MEDS

dataset.

variations in illumination, contrast, and frontal pose.

Webcams are commonly used in remote internetbased applications from a laptop or desktop computer, and webcam images may have unique value

in virtual age verification for certain websites and

potentially webcam-enabled banking ATMs. Guidelines for quality aspects of facial images are documented in ANSI/NIST-ITL 1-2011 [4], Annex E. For

this particular study, image quality is broadly defined by image size, resolution, and contrast and

subject illumination and pose. Leveraging the LEO

database allows for the extraction of a fair number

of images used in the analysis of age estimation accuracy versus image quality. 190,852 poor quality

webcam images of size 240x240 and 157,906 better

quality mugshot images of size 768x960 were used

in this analysis. Table 7 contains sample photos extracted from the publicly available Multiple Encounter Deceased Subject (MEDS) Database [3], which is representative of image qualities from the LEO dataset. Figure 8 shows the accuracy

and error distribution of the algorithms on the two different types of images.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.6 MULTIPLE IMAGE SAMPLES FRVT 18

B30D B31D E30D

E31D E32D F30D

K10D P30D Q10D

0

10

20

30

40

50

60

70

80

90

100

0

10

20

30

40

50

60

70

80

90

100

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 0 5 10 15 20 0 5 10 15 20

Absolute Age Estimation Error (years)

Cumulative Score (%)

webcam mugshot

(a) Cumulative score vs. Absolute age estimation error

B30D B31D E30D

E31D E32D F30D

K10D P30D Q10D

0.00

0.02

0.04

0.06

0.08

0.00

0.02

0.04

0.06

0.08

0.00

0.02

0.04

0.06

0.08

−20 −10 0 10 20 −20 −10 0 10 20 −20 −10 0 10 20

Age Estimation Error (years)

Density

webcam mugshot

(b) Density of age estimation error distribution

Figure 8: Line plot of accuracy and density plot of age estimation error between webcam and mugshot images.

Results and notable observations:

• Six out of nine algorithms demonstrate lower accuracy with poor quality webcam images.

• The impact of image quality on accuracy varies significantly between algorithms. B30D exhibits an 18% difference

in 5-year accuracy while E30D and E31D show no discernible differences between webcam and mugshot images.

P30D exhibits slightly higher accuracy in webcam over mugshot images.

• There is an observable bias in the error distribution to the right exhibited by algorithms on webcam images, which

suggests that overestimation is occurring. There is no clear explanation for this biasing, although it may anecdotally

be related to the demographics of the subjects captured in the webcam images.

3.6 Multiple Image Samples

In certain applications, there are opportunities for multi-sampling of images, such as imagery being captured from video

of people walking past a digital sign. For such scenarios, the question arises of whether accuracy improves if the age

estimation implementation is provided multiple contemporaneous images of the same subject. This could drive whether

a system, for example, used for targeted digital marketing, could set a minimum threshold on the number images of

a person to process, based on some time-accuracy tradeoff, prior to making a decision on the type of advertisement to

display.

The FRVT API [11] supports multiple still image input to the algorithm software for age estimation, which enables the

analysis of age estimation performance versus the number of image samples of the same person. The LEO dataset includes K > 1 contemporaneous images for some subjects, with contemporaneous, here, being defined as images of the

same subject collected within a twelve month span. This allows for the modelling of a scenario where age estimation

implementations can exploit multiple images. 11,920 subjects with at least four contemporaneous mugshot images were

extracted. The subjects’ age ranged from 18-74, with a large distribution falling between age 18-42. Figure 9 shows the

effects of the number of image samples on MAE. Note that some algorithms did not support processing K > 1 images

and are not included in Figure 9.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.6 MULTIPLE IMAGE SAMPLES FRVT 19

B30D B31D E30D E31D E32D F30D P30D

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

3.5

4.0

4.5

5.0

5.5

6.0

6.5

7.0

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

Number of Images

Mean Absolute Error (years)

Figure 9: Line plots showing MAE vs. the number of image samples per subject and the bootstrap 95% confidence interval around the

MAE. Plots were generated with 11,920 subjects.

B30D B31D E30D E31D E32D F30D P30D

0.0

0.2

0.4

0.6

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

Number of Images

Age Estimation Time (sec)

Figure 10: Boxplots summarizing age estimation time vs. the number of image samples per subject. Plots were generated with 11,920

subjects.

Results and notable observations:

• Figure 9 shows the MAE monotonically decreasing as the number of image samples increases for all algorithms.

There is a decrease in MAE of 10-20% ( ̃1 year) between one and four images, depending on the algorithm. The separation between confidence intervals (not overlapping) between one, two, and three image samples demonstrates statistical significance and stability in the observed decrease in MAE. While the results documented are generated from

images collected within a twelve month span, analysis with truly contemporaneous data (e.g., sequential frames

from video) may further solidify the trends observed.

• It is also interesting to observe that the improvement in MAE starts to decrease as the number of image samples

increases, and overlapping of the confidence intervals starts to occur between three and four images, which could

be attributed to the law of dimishing returns.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.7 FG-NET FRVT 20

• Figure 10 shows age estimation times increasing linearly with respect to the number of image samples, as expected.

3.7 FG-NET

The FG-NET Aging Database [1] is a publicly available dataset that is widely used in academia for age estimation performance benchmarking. It contains 1,002 images for 82 subjects ranging from age 0-69, although over 50% of the images are

between age 0-13. Given published results from academia are publicly available for this dataset, it would be of interest

to conduct a performance comparison against the FRVT participants, many of which are commercial algorithms. Table

9 tabulates the performance of the FRVT participants against the published methods for automatic age estimation from

academic literature.

Publication MAE (years) CS(5)

Luu et al. [18] 4.1 73%

Chao et al. [7] 4.4 NA

Chang et al. [6] 4.5 74.7%

Han et al. [15] 4.6 74.8%

Choi et al. [8] 4.7 73%

Guo et al. [14] 4.8 47%

Wu et al. [23] 5.9 62%

Suo et al. [19] 6.0 55%

Thukral et

al. [20]

6.2 NA

Geng et al. [9] 6.8 65%

(a) Published methods [15],

using LOPO testing protocol

Algorithm MAE (years) CS(5)

B30D 7.7 59%

B31D 8.6 51%

E30D 6.9 48%

E31D 6.5 53%

E32D 5.8 56%

F30D 15.9 16%

K10D 23.8 8%

P30D 7.3 44%

Q10D 15.1 23%

(b) FRVT participants,

using lights-out, black-box testing protocol

Table 9: Tables summarizing MAE and CS(5) on FG-NET.

Results and notable observations:

• Many of the academic methods performed better than the FRVT participants in terms of MAE and CS(5).

• An important point to be made is that all of the published methods in Table 9a were tested through a Leave-OnePerson-Out (LOPO) protocol when running their algorithms on FG-NET, which allowed the implementation to train

on a subset of the images through every iteration of testing. For the participant results documented in this report,

NIST employed a lights-out, black-box testing methodology that did not involve any type of algorithm training

during evaluation. This is designed to model operational reality where software is shipped and used ”as-is” without

algorithmic training. Given the fundamental differences in testing approaches, a fair benchmark comparison against

academic methods cannot be performed for the FG-NET dataset in this report.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.8 SPECIFIC APPLICATIONS FRVT 21

3.8 Specific Applications

3.8.1 Age Verification

Many age estimation systems strive to classify an individual’s age or age range to verify that the person meets specific age

requirements. Common examples in the United States include the purchasing of alcohol or admission into a casino, which

●

●●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

0

10

20

30

40

50

60

70

80

90

100

101112131415161718192021

Actual age (years)

% of age estimations equal or above age 21

● B30D ● B31D ● E30D ● E31D ● E32D ● F30D ● K10D ● P30D ● Q10D

Figure 11: Line plot showing the probability of algorithms estimating that

a person at a particular age is equal or above age 21. Plot was generated

over a heterogeneous population of 243,023 images.

has a minimum age requirement of 21. Figure 11

shows the probability of algorithms classifying a

person at a particular age as equal or above age 21.

It can be observed that a person at age 17 has anywhere from a 29% to 85% chance of passing for age

21 or above, depending on the algorithm. The most

accurate algorithm (B30D) achieves the lowest false

verification percentage of 29% at age 17. This percentage increases as a person gets closer to 21. System thresholds can be set based on the probabilities

observed and the costs associated with errors in age

verification.

Consider an age estimation system that screens for

underage individuals attempting to purchase alcohol. One might set the cost of a false negative, in

this case, misclassifying someone over the age of

21 as under, to the inconvenience incurred by having to show proof of their age. The cost of a false

positive, i.e., allowing a person under the age of

0.01

0.02

0.05

0.10

0.20

0.50

1.00

0.00010.00020.00050.00100.00200.00500.01000.02000.05000.10000.20000.50001.0000

False Positive Rate

False Negative Rate

B30D B31D E30D E31D E32D F30D K10D P30D Q10D

Algorithm

B30D

B31D

E30D

E31D

E32D

F30D

K10D

P30D

Q10D

FNR @ FPR = .02

0.40

0.39

0.49

0.49

0.44

0.79

0.87

0.63

0.80

Figure 12: DET curve plotting false negative rate against false positive rate

for a verification age of 21. Plot was generated over a heterogeneous population of 243,023 images.

21 to purchase alcohol, could be the incurrence of

a fine or even serving time in jail. Given it would

be reasonable to argue that the costs are asymmetric in this scenario, i.e., the cost of a false positive

is much greater than that of a false negative, tighter

confidence levels could be set to minimize false positives. Figure 12 presents DET accuracy for a verification age of 21. A false positive rate of 0.02 would

impose a false negative rate of 0.39 for the most accurate algorithm (B31D) at that threshold. If a system

were to ensure that 98% of underage individuals are

detected and not allowed to purchase alcohol or enter a casino, it would impose on 39% of people who

are actually above 21 to be asked for proof of age,

unnecessarily.

The same analysis can be done for other verification ages of interest. The legal age for purchasing

alcohol and tobacco products in some places such as

Hong Kong and the United Kingdom is 18. There are

cigarette vending machines in Japan that have ageverification systems in place to verify that a person

is at least 20 based on facial images captured prior to allowing them to make a purchase. Certain airlines have restrictions

and extra fees for unaccompanied minors traveling under the age of 14, so the ability to detect minors under age 14 who

appear to be traveling alone may be of interest to airports. The age verification probability and DET plots for these other

scenarios are provided in Appendix A.2 for reference.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.8 SPECIFIC APPLICATIONS FRVT 22

3.8.2 Children

There are age estimation applications that are specific to children. For example, age-invariant identity verification [10,24],

which involves age regression or progression to predict how the subject looked like in the past or will look like in the future,

certainly contains an age estimation component, and can support law enforcement in finding missing children. Missing

children recovered from human trafficking or abduction may not know their own age, and automated age estimation could

potentially aid in the investigation process. Tables 10 and 11 present the percentage of age estimates accurate to within

one year and MAE for children ages 0-14 from an ethnically-homogeneous population extracted from the DoS/Natural

dataset.

Age

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

Num Images

133813

 94610

 82846

 82872

 84510

 92982

 92143

 90537

 89960

 90182

 98834

 97996

 94830

 94545

 97467

B30D

85

74

63

50

44

39

37

36

28

23

24

24

33

38

24

B31D

83

34

27

33

34

33

34

29

25

21

22

23

18

12

18

E30D

60

29

24

21

19

17

15

12

10

9

7

6

5

4

4

E31D

65

29

25

21

19

17

14

11

10

8

7

6

5

5

4

E32D

68

33

29

27

24

20

16

12

9

7

6

4

3

3

3

F30D

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

K10D

0

0

0

0

0

0

0

0

0

0

0

0

1

1

2

P30D

3

5

6

7

7

8

8

8

9

10

10

10

10

10

11

Q10D

0

62

44

17

13

14

28

25

9

6

2

2

2

1

0

Table 10: CS(1), in percentage, for children by age.

Age

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

Num Images

133813

 94610

 82846

 82872

 84510

 92982

 92143

 90537

 89960

 90182

 98834

 97996

 94830

 94545

 97467

B30D

2.4

1.4

1.5

1.7

1.8

1.9

2.1

2.3

2.6

2.8

2.9

2.9

2.9

3.1

3.6

B31D

2.0

2.0

2.1

2.2

2.2

2.3

2.5

2.8

3.1

3.2

3.3

3.4

3.6

3.8

4.1

E30D

1.8

2.3

3.0

3.5

4.1

4.5

5.1

5.8

6.3

6.7

7.0

7.2

7.2

7.1

6.8

E31D

1.7

2.2

3.0

3.6

4.3

4.7

5.3

6.1

6.6

6.9

7.3

7.4

7.3

7.1

6.7

E32D

1.6

2.3

2.9

3.3

3.8

4.4

5.1

6.1

6.7

7.2

7.6

7.7

7.6

7.1

6.5

F30D

28.3

27.0

25.3

24.0

22.8

21.5

20.4

19.3

18.4

17.5

16.6

15.5

14.3

12.8

11.4

K10D

33.7

31.1

29.4

28.2

27.3

26.5

25.1

23.1

21.4

19.9

18.6

17.1

15.8

14.5

13.2

P30D

10.9

 7.2

 6.7

 6.4

 6.2

 6.0

 5.9

 5.8

 5.6

 5.4

 5.3

 5.3

 5.3

 5.2

 5.0

Q10D

 3.3

 2.7

 4.3

 6.0

 7.4

 9.1

10.4

12.2

13.8

14.9

15.7

15.7

15.4

14.9

14.1

Table 11: MAE, in years, for children by age.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.8 SPECIFIC APPLICATIONS FRVT 23

3.8.3 Time Interval Estimation

The ability to accurately determine the amount of time that has passed between two images of the same subject has

potential application in law enforcement investigations. 473,923 pairs of same-subject images with an age delta between

0-4 years were used in the analysis of using age estimation technology in time interval determination. Figure 13 presents

the age estimation error for the first image plotted against the error associated with the second image, over all pairs of

images, which supports correlation analysis between the age estimation errors.

B30D B31D E30D

E31D E32D F30D

K10D P30D Q10D

−10

−5

0

5

10

−10

−5

0

5

10

−10

−5

0

5

10

−10 −5 0 5 10 −10 −5 0 5 10 −10 −5 0 5 10

Age Estimation Error (Image 1)

Age Estimation Error (Image 2)

2000

4000

6000

Frequency

Figure 13: Hexbin plots showing the correlation between age estimation error for image 1 vs. image 2. Plots were generated with

473,923 pairs of same-subject images. The dashed line annotates where estimation error for image 1 and image 2 are equal.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
3.8 SPECIFIC APPLICATIONS FRVT 24

Given the standard deviation of the age estimation error for image 1, σx, and image 2, σy, the correlation coefficient, ρ, is

defined as the covariance, σxy, divided by the product of σx and σy. i.e.,

ρ =

σxy

σxσy

,

σxy =

PN

k=1(xk − x ̄)(yk − y ̄)

N

,

(6)

where xk is the age estimation error for the k-th first image, yk is the error for the k-th second image, x ̄ is the mean error

across all first images, y ̄ is the mean error across all second images, and N is the number of image pairs.

For the dataset used, the maximum time interval between any given image pair is less than four years, so for simplicity,

σx = σy = σ. The standard deviation of the difference of x and y can be related to their individual standard deviations

and the correlation coefficient between them. i.e.,

σx−y =

p

2σ

2(1 − ρ) (7)

Table 12 presents the standard deviation and correlation coefficient of the age estimation errors along with the standard

deviation associated with time interval estimation error based on two independent age estimates of the same subject at

different ages.

Algorithm σ (years) ρ σx−y (years)

B30D 6.2 0.55 5.9

B31D 5.8 0.65 4.8

E30D 8.5 0.76 5.9

E31D 8.5 0.76 5.9

E32D 6.7 0.68 5.4

F30D 10.9 0.86 5.8

K10D 11.8 0.82 7.1

P30D 9.2 0.81 5.7

Q10D 10.1 0.71 7.7

Table 12: Table summarizing the standard deviation of time interval estimation error derived from the age estimation error and correlation coefficient between independent age estimates from 473,923 pairs of same-subject images.

As observed from Figure 13 and Table 12, there is a positive linear correlation between the error of two independent age

estimates of the same subject at different ages, with varying degrees of correlation between the algorithms. The algorithm

with the lowest time interval estimation error (i.e. B31D), produces a standard deviation in error of 4.8 years. Some of the

algorithms that exhibit large age estimation error also demonstrate high correlation in error, which indicates consistency

in overestimation or underestimation, resulting in relatively lower time interval estimation error.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
REFERENCES FRVT 25

References

[1] FG-NET Aging Database. http://www-prima.inrialpes.fr/FGnet/html/benchmarks.html.

[2] International Society of Aesthetic Plastic Surgery (ISAPS) International Survey on Aesthetic/Cosmetic

Procedures Performed in 2011. http://www.isaps.org/Media/Default/global-statistics/

ISAPS-Results-Procedures-2011.pdf.

[3] NIST Special Database 32 - Multiple Encounter Dataset 2 (MEDS-II), NISTIR 7807. http://www.nist.gov/itl/

iad/ig/sd32.cfm.

[4] NIST Special Publication 500-290, ANSI/NIST-ITL 1-2011, Data Format for the Interchange of Fingerprint, Facial and

Other Biometric Information. http://www.nist.gov/itl/iad/ig/ansi_standard.cfm.

[5] K. Alkass, B. A. Buchholz, S. Ohtani, T. Yamamoto, H. Druid, and K. L. Spalding. Age estimation in forensic sciences:

application of combined aspartic acid racemization and radiocarbon analysis. Mol Cell Proteomics, 9(5):1022–1030,

2010.

[6] K. Y. Chang, C. S. Chen, and Y. P. Hung. Ordinal hyperplans ranker with cost sensitivities for age estimation. In Proc.

IEEE CVPR, pages 585–592, 2011.

[7] W. L. Chao, J. Z. Liu, and J. J. Ding. Facial age estimation based on label-sensitive learning and age-oriented regression. Pattern Recognition, 46(3):628–641, 2013.

[8] S. E. Choi, Y. J. Lee, S. J. Lee, K. R. Park, and J. Kim. Age estimation using a hierarchical classifer based on global and

local facial features. Pattern Recognition, 44(6):1262–1281, 2011.

[9] X. Geng, Z. H. Zhou, and K. Smith-Miles. Automatic age estimation based on facial aging patterns. IEEE Trans.

Pattern Anal. Machine Intell., 29(12):2234–2240, 2007.

[10] D. Gong, Z. Li, D. Lin, J. Liu, and X. Tang. Hidden factor analysis for age invariant face recognition. In Proc. ICCV,

pages 1–8, 2013.

[11] P. Grother, G. W. Quinn, and M. Ngan. FRVT Still Face Image and Video Concept, Evaluation Plan and API Version

1.4, 2013. http://www.nist.gov/itl/iad/ig/frvt-2012.cfm.

[12] G. Guo and G. Mu. Human age estimation: What is the influence across race and gender? In IEEE International

Workshop on Analysis and Modeling of Faces and Gestures (AMFG’10), 2010.

[13] G. Guo, G. Mu, Y. Fu, C. Dyer, and T. S. Huang. A study on automatic age estimation using a large database. In Proc.

ICCV, pages 1986–1991, 2009.

[14] G. Guo, G. Mu, Y. Fu, and T. S. Huang. Human age estimation using bio-inspired features. In In Proc. IEEE CVPR,

pages 112–119, 2009.

[15] H. Han, C. Otto, and A. K. Jain. Age estimation from face images: Human vs. machine performance. In Proc. of IAPR

ICB, pages 1–8, 2013.

[16] A. K. Jain, S. C. Dass, and K. Nandakumar. Soft biometric traits for personal recognition systems. In Proc. Intl Conf.

Biometric Authentication (ICBA 04) LNCS 3072, pages 731–738, 2004.

[17] K. Ricanek Jr., C. Chen Y. Wang, and S. J. Simmons. Generalized multi-ethnic face age-estimation. In Third IEEE

International Conference on Biometrics, pages 127–132, 2009.

[18] K. Luu, K. Seshadri, M. Savvides, T. Bui, and C. Suen. Contourlet appearance model for facial age estimation. In Proc.

IJCB, pages 1–8, 2011.

[19] J. Suo, S. C. Zhu, and X. Chen. A compositional and dynamic model for face aging. IEEE Trans. PAMI, 32(3):385–401,

2010.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
REFERENCES FRVT 26

[20] P. Thukral, K. Mitra, and R. Chellappa. A hierarchical approach for human age estimation. In Proc. IEEE ICASSP,

pages 1529–1532, 2012.

[21] J. L. Wayman. Large-scale civilian biometric systems - issues and feasibility. In Card Tech / Secur Tech ID, 1997.

[22] Hadley Wickham. ggplot2: Elegant Graphics for Data Analysis. Springer, New York, 2009. ISBN 978-0-387-98141-3.

[23] T. Wu, P. Turaga, and R. Chellappa. Age estimation and face verification across aging using landmarks. IEEE Trans.

IFS, 7(6):1780–1788, 2012.

[24] F. J. Xu, K. Luu, M. Savvides, T. D. Bui, and C. Y. Suen. Investigating age invariant face recognition based on periocular

biometrics. In Proc. IJCB, pages 1–7, 2011.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
FRVT 27

Appendix A Additional Figures and Tables

Appendix A contains supplementary figures and tables for all age estimation algorithms.

A.1 Age Estimation Performance by Age

Age

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

Num Images

133690

 94406

 82556

 82481

 84111

 92524

 91634

 90004

 89419

 89652

 98327

 97440

 94317

 94070

 96940

 97522

 96440

 99469

 98734

105770

118645

124101

128841

133521

138148

139873

141798

138453

133628

129021

123436

B30D

93 / 2.4

97 / 1.4

97 / 1.5

97 / 1.7

97 / 1.8

96 / 1.9

94 / 2.1

91 / 2.3

90 / 2.6

90 / 2.8

88 / 2.9

84 / 2.9

80 / 2.9

77 / 3.1

73 / 3.6

71 / 3.9

71 / 4

72 / 4

73 / 3.9

69 / 3.8

69 / 3.8

70 / 3.8

71 / 3.9

72 / 4

70 / 4

68 / 4.2

65 / 4.3

63 / 4.5

60 / 4.7

59 / 4.9

57 / 5.1

B31D

95 / 2

97 / 2

96 / 2.1

95 / 2.2

94 / 2.2

91 / 2.3

89 / 2.5

86 / 2.8

83 / 3.1

79 / 3.2

82 / 3.3

82 / 3.4

77 / 3.6

71 / 3.8

66 / 4.1

64 / 4.3

65 / 4.4

65 / 4.3

68 / 4.1

72 / 3.9

71 / 3.8

71 / 3.8

72 / 3.7

72 / 3.7

73 / 3.7

72 / 3.8

69 / 3.9

68 / 4.1

65 / 4.2

62 / 4.4

61 / 4.5

E30D

92 / 1.8

90 / 2.3

84 / 3

79 / 3.5

73 / 4.1

65 / 4.5

60 / 5.1

53 / 5.8

47 / 6.3

43 / 6.7

38 / 7

35 / 7.2

33 / 7.2

32 / 7.1

33 / 6.8

37 / 6.5

42 / 6.1

48 / 5.7

53 / 5.3

58 / 4.9

63 / 4.6

66 / 4.3

69 / 4.1

70 / 4

71 / 4

70 / 4

69 / 4.1

67 / 4.3

64 / 4.4

61 / 4.6

59 / 4.8

E31D

93 / 1.7

90 / 2.2

84 / 3

78 / 3.6

72 / 4.3

64 / 4.7

58 / 5.3

50 / 6.1

45 / 6.6

41 / 6.9

36 / 7.3

33 / 7.4

32 / 7.3

32 / 7.1

34 / 6.7

37 / 6.3

43 / 5.9

49 / 5.5

55 / 5.1

60 / 4.7

64 / 4.4

67 / 4.2

69 / 4.1

70 / 4

70 / 4

69 / 4.1

68 / 4.2

66 / 4.3

64 / 4.5

61 / 4.6

59 / 4.8

E32D

93 / 1.6

89 / 2.3

85 / 2.9

81 / 3.3

76 / 3.8

69 / 4.4

61 / 5.1

51 / 6.1

44 / 6.7

38 / 7.2

32 / 7.6

29 / 7.7

27 / 7.6

30 / 7.1

36 / 6.5

44 / 5.8

51 / 5.3

58 / 4.8

63 / 4.5

68 / 4.1

72 / 3.8

75 / 3.5

77 / 3.4

78 / 3.4

78 / 3.4

76 / 3.5

75 / 3.7

72 / 3.8

68 / 4.1

65 / 4.3

62 / 4.5

F30D

0 / 28.3

0 / 27

0 / 25.3

0 / 24

0 / 22.8

0 / 21.5

0 / 20.4

0 / 19.3

0 / 18.4

0 / 17.5

0 / 16.6

2 / 15.5

4 / 14.3

8 / 12.8

13 / 11.4

19 / 10.2

24 / 9.3

29 / 8.6

32 / 8.1

35 / 7.6

38 / 7.2

40 / 6.9

42 / 6.5

45 / 6.2

48 / 5.9

49 / 5.7

51 / 5.5

53 / 5.2

55 / 5

57 / 4.9

59 / 4.7

K10D

0 / 33.7

0 / 31.1

0 / 29.4

0 / 28.2

0 / 27.3

0 / 26.4

0 / 25.1

0 / 23.1

1 / 21.4

1 / 19.8

2 / 18.6

3 / 17.1

5 / 15.8

8 / 14.5

11 / 13.2

15 / 12.1

19 / 11.2

22 / 10.4

23 / 10.1

25 / 9.6

29 / 9

32 / 8.4

35 / 7.9

38 / 7.4

41 / 7

45 / 6.6

48 / 6.2

50 / 5.9

52 / 5.6

54 / 5.5

56 / 5.3

P30D

20 / 10.9

35 / 7.2

38 / 6.7

41 / 6.4

43 / 6.2

46 / 6

47 / 5.9

47 / 5.8

49 / 5.6

52 / 5.4

53 / 5.3

53 / 5.3

53 / 5.3

54 / 5.2

56 / 5.1

60 / 4.7

64 / 4.4

69 / 4

74 / 3.6

77 / 3.4

77 / 3.4

77 / 3.4

74 / 3.6

70 / 3.9

65 / 4.3

59 / 4.7

54 / 5.1

48 / 5.6

43 / 6

39 / 6.5

36 / 7

Q10D

92 / 3.3

90 / 2.7

85 / 4.3

78 / 6

71 / 7.3

62 / 9.1

52 / 10.4

40 / 12.2

31 / 13.8

22 / 14.9

15 / 15.7

10 / 15.7

7 / 15.4

6 / 14.9

5 / 14.1

6 / 13.3

7 / 12.5

9 / 11.7

11 / 10.7

13 / 10

16 / 9.5

19 / 8.9

27 / 8.4

35 / 8

40 / 7.5

43 / 7.1

46 / 6.7

47 / 6.4

49 / 6.2

45 / 6.1

48 / 5.9

Table 13: CS(5), in percentage / MAE, in years, for age 0-30. Table was generated over an ethnically-homogeneous population.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
A.1 AGE ESTIMATION PERFORMANCE BY AGE FRVT 28

Age

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

Num Images

118117

113140

109142

105909

103135

 99884

 97533

 93916

 91463

 94042

 90172

 86397

 83247

 80605

 77477

 74183

 70689

 67881

 65641

 64889

 63424

 60969

 59545

 56893

 55271

 52583

 49828

 48544

 46032

 46064

 43832

 41420

 39435

 37404

 34833

 32009

B30D

56 / 5.3

54 / 5.4

53 / 5.5

53 / 5.6

54 / 5.6

54 / 5.5

53 / 5.5

53 / 5.5

54 / 5.5

55 / 5.5

57 / 5.5

58 / 5.5

57 / 5.5

55 / 5.4

55 / 5.4

56 / 5.3

57 / 5.4

57 / 5.4

56 / 5.4

56 / 5.4

56 / 5.4

54 / 5.4

56 / 5.4

57 / 5.4

56 / 5.4

56 / 5.4

55 / 5.4

55 / 5.5

55 / 5.7

53 / 5.8

50 / 5.9

48 / 6.1

45 / 6.2

44 / 6.3

44 / 6.5

43 / 6.6

B31D

61 / 4.7

60 / 4.8

59 / 4.9

58 / 5

57 / 5

58 / 5

57 / 5

57 / 5.1

58 / 5

58 / 5

59 / 5

59 / 5.1

60 / 5

59 / 5

59 / 5

59 / 4.9

59 / 4.8

59 / 4.8

61 / 4.8

61 / 4.8

62 / 4.8

61 / 4.8

60 / 4.8

62 / 4.7

62 / 4.8

62 / 4.8

61 / 4.9

60 / 4.9

57 / 5

57 / 5.2

54 / 5.3

53 / 5.5

51 / 5.7

48 / 5.8

46 / 5.9

46 / 6.1

E30D

56 / 5

54 / 5.1

53 / 5.3

51 / 5.4

51 / 5.5

50 / 5.6

50 / 5.7

50 / 5.8

50 / 5.9

51 / 5.9

51 / 6

50 / 6.2

49 / 6.3

48 / 6.5

48 / 6.7

46 / 6.9

45 / 7.2

43 / 7.4

41 / 7.7

39 / 8

36 / 8.3

34 / 8.7

32 / 9.1

30 / 9.4

28 / 9.7

26 / 10.1

24 / 10.5

23 / 10.9

22 / 11.2

20 / 11.7

19 / 12.1

18 / 12.5

16 / 13

15 / 13.4

14 / 13.8

13 / 14.2

E31D

57 / 4.9

56 / 5.1

54 / 5.2

53 / 5.3

52 / 5.4

52 / 5.5

51 / 5.6

51 / 5.7

51 / 5.8

52 / 5.8

51 / 6

50 / 6.2

49 / 6.3

48 / 6.6

47 / 6.8

45 / 7.1

42 / 7.3

40 / 7.6

38 / 7.9

36 / 8.3

34 / 8.6

31 / 9

30 / 9.3

28 / 9.6

27 / 10

25 / 10.4

24 / 10.7

23 / 11.1

22 / 11.4

21 / 11.8

20 / 12.2

19 / 12.6

17 / 13.1

16 / 13.5

15 / 13.9

14 / 14.2

E32D

60 / 4.7

57 / 4.8

56 / 5

54 / 5.1

54 / 5.2

54 / 5.2

53 / 5.3

54 / 5.3

54 / 5.3

55 / 5.3

55 / 5.3

56 / 5.3

56 / 5.3

57 / 5.3

57 / 5.2

58 / 5.2

58 / 5.2

59 / 5.2

58 / 5.2

59 / 5.3

58 / 5.3

57 / 5.4

57 / 5.5

56 / 5.6

54 / 5.7

53 / 5.9

51 / 6.1

48 / 6.4

46 / 6.6

44 / 6.8

41 / 7.2

38 / 7.5

35 / 7.9

32 / 8.3

29 / 8.7

26 / 9.1

F30D

62 / 4.5

64 / 4.4

66 / 4.2

68 / 4.2

69 / 4.1

69 / 4

69 / 4.1

68 / 4.1

66 / 4.2

65 / 4.3

63 / 4.5

61 / 4.7

58 / 4.9

54 / 5.2

51 / 5.4

48 / 5.7

46 / 6

43 / 6.3

41 / 6.6

39 / 6.9

36 / 7.3

34 / 7.7

31 / 8.1

28 / 8.5

26 / 9

23 / 9.4

21 / 9.9

17 / 10.5

15 / 11

12 / 11.6

8 / 12.2

6 / 12.8

4 / 13.5

2 / 14.2

1 / 14.8

1 / 15.5

K10D

57 / 5.1

57 / 5.1

58 / 5

58 / 4.9

58 / 4.9

58 / 4.9

57 / 5

57 / 5.1

56 / 5.2

54 / 5.4

52 / 5.6

50 / 5.8

48 / 6

46 / 6.3

44 / 6.5

41 / 6.9

39 / 7.1

37 / 7.5

35 / 7.8

32 / 8.2

31 / 8.5

28 / 8.9

26 / 9.3

24 / 9.7

22 / 10.1

20 / 10.6

19 / 11

17 / 11.4

16 / 11.7

15 / 12.2

13 / 12.5

13 / 12.9

12 / 13.5

11 / 13.9

10 / 14.2

9 / 14.6

P30D

33 / 7.4

31 / 7.8

30 / 8.2

28 / 8.5

27 / 8.9

27 / 9.2

26 / 9.4

25 / 9.8

25 / 10

26 / 10

25 / 10.3

25 / 10.5

25 / 10.7

24 / 10.9

24 / 11.1

24 / 11.3

23 / 11.4

23 / 11.6

23 / 11.8

23 / 11.9

22 / 12.1

22 / 12.2

22 / 12.5

21 / 12.7

20 / 12.9

19 / 13.1

18 / 13.3

17 / 13.6

15 / 13.9

14 / 14.2

12 / 14.5

10 / 15

8 / 15.5

7 / 15.9

5 / 16.4

4 / 16.8

Q10D

50 / 5.9

49 / 5.8

47 / 5.8

46 / 5.7

50 / 5.6

50 / 5.5

51 / 5.5

54 / 5.5

61 / 5.4

62 / 5.3

57 / 5.4

57 / 5.5

56 / 5.7

55 / 5.9

52 / 6.2

50 / 6.5

48 / 6.8

44 / 7

37 / 7.3

34 / 7.6

37 / 7.9

33 / 8.3

32 / 8.6

32 / 8.9

27 / 9.2

26 / 9.6

26 / 9.9

25 / 10

25 / 10.3

24 / 10.6

24 / 10.7

26 / 10.9

26 / 11.1

24 / 11.1

25 / 11.1

27 / 11.1

Table 14: CS(5), in percentage / MAE, in years, for age 31-66. Table was generated over an ethnically-homogeneous population.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
A.1 AGE ESTIMATION PERFORMANCE BY AGE FRVT 29

Age

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

Num Images

28903

26617

24863

23395

20728

18417

16212

14849

13184

11889

10836

 9351

 7551

 6248

 5231

 3953

 3273

 2617

 2201

 1834

 1399

 1251

 909

 668

 514

 322

 235

 156

 117

 62

 43

 35

 15

B30D

43 / 6.6

41 / 6.7

40 / 6.8

41 / 6.7

41 / 6.8

41 / 6.8

44 / 6.8

59 / 6.6

57 / 6.5

55 / 6.4

53 / 6.5

54 / 6.3

54 / 6.4

54 / 6.8

53 / 7.4

51 / 8

46 / 8.7

0 / 9.4

0 / 9.9

0 / 10.8

0 / 11.7

0 / 12.3

0 / 13

0 / 14.5

0 / 14.7

0 / 15.6

0 / 17.2

0 / 17.7

0 / 20.5

0 / 20.4

0 / 20.1

0 / 23.2

0 / 23.2

B31D

45 / 6.2

46 / 6.3

44 / 6.4

44 / 6.4

44 / 6.5

45 / 6.4

45 / 6.5

56 / 6.4

55 / 6.4

54 / 6.4

52 / 6.5

50 / 6.4

51 / 6.5

51 / 6.9

49 / 7.4

45 / 8.1

42 / 8.6

0 / 9.4

0 / 9.9

0 / 10.7

0 / 11.5

0 / 12

0 / 12.7

0 / 14.3

0 / 14.6

0 / 15.5

0 / 16.8

0 / 17.4

0 / 20

0 / 19.7

0 / 20.3

0 / 22.7

0 / 23.2

E30D

12 / 14.7

11 / 15.2

10 / 15.7

9 / 16.2

8 / 16.7

7 / 17.4

6 / 18

5 / 18.6

4 / 19.1

4 / 19.5

3 / 20.5

2 / 20.9

2 / 21.7

1 / 22.6

1 / 23.3

1 / 23.8

0 / 24.7

0 / 25.3

0 / 26.5

0 / 27.2

0 / 27.8

0 / 28.5

0 / 29

0 / 30.5

0 / 31.9

0 / 32.4

0 / 33

0 / 34.2

0 / 36.3

0 / 34.7

0 / 35.6

0 / 38.2

0 / 37.8

E31D

13 / 14.8

11 / 15.3

10 / 15.7

9 / 16.2

8 / 16.7

7 / 17.4

6 / 18

5 / 18.6

4 / 19.1

4 / 19.5

3 / 20.4

2 / 20.9

2 / 21.8

1 / 22.6

1 / 23.3

1 / 23.7

0 / 24.8

0 / 25.5

0 / 26.6

0 / 27.2

0 / 27.9

0 / 28.5

0 / 29.2

0 / 31

0 / 31.9

0 / 32.8

0 / 33.7

0 / 34.6

0 / 36.9

0 / 35

0 / 37.7

0 / 39.8

0 / 38

E32D

23 / 9.5

21 / 10

18 / 10.5

16 / 11

15 / 11.6

13 / 12.1

12 / 12.6

10 / 13.1

9 / 13.8

8 / 14.3

7 / 15.1

6 / 15.7

5 / 16.3

4 / 17.1

4 / 17.7

3 / 18.3

2 / 19

2 / 19.9

1 / 20.7

0 / 21.5

0 / 22.1

0 / 23

0 / 23.6

0 / 24.9

0 / 25.6

0 / 26.5

0 / 27.6

0 / 28.4

0 / 30.7

0 / 29.9

0 / 31.2

0 / 32.8

0 / 34.4

F30D

0 / 16.2

0 / 16.9

0 / 17.7

0 / 18.4

0 / 19.1

0 / 20

0 / 20.8

0 / 21.6

0 / 22.3

0 / 23

0 / 23.9

0 / 24.7

0 / 25.5

0 / 26.5

0 / 27.2

0 / 28.1

0 / 29.1

0 / 29.9

0 / 30.8

0 / 31.5

0 / 32.5

0 / 33.5

0 / 34.1

0 / 35.5

0 / 36.2

0 / 36.8

0 / 38

0 / 38.9

0 / 40.6

0 / 41.6

0 / 42

0 / 42.4

0 / 47

K10D

8 / 15.1

7 / 15.7

6 / 16.1

5 / 16.4

4 / 17.1

4 / 17.5

4 / 18

3 / 18.7

3 / 19.1

2 / 19.4

2 / 20.3

1 / 20.7

1 / 21.4

1 / 22.1

1 / 22.5

0 / 22.8

1 / 23.4

0 / 24.2

0 / 24.9

0 / 25.6

0 / 25.9

0 / 26.5

0 / 26.5

0 / 28.8

0 / 29.6

0 / 29.8

0 / 30.3

0 / 31.5

0 / 32.4

0 / 31.3

0 / 30.3

0 / 34.5

0 / 44.1

P30D

3 / 17.3

2 / 17.9

1 / 18.5

1 / 19.2

0 / 19.7

0 / 20.5

0 / 21.3

0 / 21.8

0 / 22.5

0 / 23.1

0 / 24

0 / 24.7

0 / 25.4

0 / 26.5

0 / 27.3

0 / 27.8

0 / 28.7

0 / 29.6

0 / 30.6

0 / 31.3

0 / 32.4

0 / 33.5

0 / 34.1

0 / 35.6

0 / 36.5

0 / 36.9

0 / 38.6

0 / 39.1

0 / 41.1

0 / 41.6

0 / 42.3

0 / 41.7

0 / 47.8

Q10D

29 / 11

32 / 11

36 / 10.9

44 / 10.8

45 / 10.9

45 / 10.8

45 / 10.9

43 / 11.1

42 / 11.4

40 / 11.8

36 / 12.5

31 / 13

21 / 13.9

0 / 14.4

0 / 15.4

0 / 15.5

0 / 17

0 / 17.7

0 / 18.7

0 / 19.7

0 / 20

0 / 21.8

0 / 22.2

0 / 23.5

0 / 25.4

0 / 25.4

0 / 26.3

0 / 28.8

0 / 30.3

0 / 29.2

0 / 28.3

0 / 30.6

0 / 38.8

Table 15: CS(5), in percentage / MAE, in years, for age 67-99. Table was generated over an ethnically-homogeneous population.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
A.2 AGE VERIFICATION ACCURACY FRVT 30

A.2 Age Verification Accuracy

A.2.1 Age 14

●

●

●●

●

●●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●●

●

●

●

●

●●

●

●

●

●

●

0

10

20

30

40

50

60

70

80

90

100

789

1011121314

Actual age (years)

% of age estimations equal or above age 14

● B30D ● B31D ● E30D ● E31D ● E32D ● F30D ● K10D ● P30D ● Q10D

(a) Probability that age estimate is equal or above age 14

0.0001

0.0002

0.0005

0.0010

0.0020

0.0050

0.0100

0.0200

0.0500

0.1000

0.2000

0.5000

1.0000

0.00010.00020.00050.00100.00200.00500.01000.02000.05000.10000.20000.50001.0000

False Positive Rate

False Negative Rate

B30D B31D E30D E31D E32D F30D K10D P30D Q10D

Algorithm

B30D

B31D

E30D

E31D

E32D

F30D

K10D

P30D

Q10D

FNR @ FPR = .02

0.15

0.14

0.29

0.30

0.34

0.80

0.90

0.52

0.85

(b) DET for verification age of 14

Figure 14: Line plot showing the probability of algorithms estimating that a person at a particular age is equal or above age 14 and DET

curve plotting false negative rate against false positive rate for verification age of 14. The dotted line in (a) highlights the probability

that a person at age 12 is estimated as equal or above age 14. Plots were generated over a heterogeneous population of 243,023 images.

A.2.2 Age 18

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

0

10

20

30

40

50

60

70

80

90

100

101112131415161718

Actual age (years)

% of age estimations equal or above age 18

● B30D ● B31D ● E30D ● E31D ● E32D ● F30D ● K10D ● P30D ● Q10D

(a) Probability that age estimate is equal or above age 18

0.0001

0.0002

0.0005

0.0010

0.0020

0.0050

0.0100

0.0200

0.0500

0.1000

0.2000

0.5000

1.0000

0.00010.00020.00050.00100.00200.00500.01000.02000.05000.10000.20000.50001.0000

False Positive Rate

False Negative Rate

B30D B31D E30D E31D E32D F30D K10D P30D Q10D

Algorithm

B30D

B31D

E30D

E31D

E32D

F30D

K10D

P30D

Q10D

FNR @ FPR = .02

0.33

0.30

0.43

0.43

0.40

0.80

0.88

0.59

0.83

(b) DET for verification age of 18

Figure 15: Line plot showing the probability of algorithms estimating that a person at a particular age is equal or above age 18 and DET

curve plotting false negative rate against false positive rate for verification age of 18. The dotted line in (a) highlights the probability

that a person at age 16 is estimated as equal or above age 18. Plots were generated over a heterogeneous population of 243,023 images.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.
A.2 AGE VERIFICATION ACCURACY FRVT 31

A.2.3 Age 20

●

●●

●

●

●

●

●

●●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

0

10

20

30

40

50

60

70

80

90

100

1011121314151617181920

Actual age (years)

% of age estimations equal or above age 20

● B30D ● B31D ● E30D ● E31D ● E32D ● F30D ● K10D ● P30D ● Q10D

(a) Probability that age estimate is equal or above age 20

0.0001

0.0002

0.0005

0.0010

0.0020

0.0050

0.0100

0.0200

0.0500

0.1000

0.2000

0.5000

1.0000

0.00010.00020.00050.00100.00200.00500.01000.02000.05000.10000.20000.50001.0000

False Positive Rate

False Negative Rate

B30D B31D E30D E31D E32D F30D K10D P30D Q10D

Algorithm

B30D

B31D

E30D

E31D

E32D

F30D

K10D

P30D

Q10D

FNR @ FPR = .02

0.39

0.36

0.46

0.46

0.42

0.79

0.87

0.64

0.81

(b) DET for verification age of 20

Figure 16: Line plot showing the probability of algorithms estimating that a person at a particular age is equal or above age 20 and DET

curve plotting false negative rate against false positive rate for verification age of 20. The dotted line in (a) highlights the probability

that a person at age 18 is estimated as equal or above age 20. Plots were generated over a heterogeneous population of 243,023 images.

B = Cognitec E = NEC F = Tsinghua University K = MITRE P = Zhuhai-Yisheng Q = JunYu Tech.