1 

 

Demographic Effects on Estimates of 

Automatic Face Recognition Performance

O’ Toole, A. J.

Phillips, P. J.

An, X.

Dunlop J.

NISTIR 7757
 2 

NISTIR 7757

Demographic Effects on Estimates of Automatic 

Face Recognition Performance

O’Toole, A. J. 

Dunlop, J. 

An, X. 

U. of Texas at Dallas 

Phillips, P. J. 

NIST 

Information Access Division 

March 2011

U.S. Department of Commerce 

Gary Locke, Secretary 

National Institute of Standards and Technology 

Patrick D. Gallagher, Director
Demographic Effects on Estimates of Automatic Face Recognition

Performance

Alice J.O’Toole

School of Behavioral and Brain Sciences

The University of Texas at Dallas

Richardson, TX 75080, USA

otoole@utdallas.edu

P. Jonathon Phillips

National Institute of Standards and Technology

Gaithersburg, MD 45435, USA

jonathon@nist.gov

Xiaobo An

Department of Electrical Engineering

The University of Texas at Dallas

Richardson, TX 75080, USA

xiaobo.an@student.utdallas.edu

Joseph Dunlop

School of Behavioral and Brain Sciences

The University of Texas at Dallas

Richardson, TX 75080, USA

joseph.dunlop@utdallas.edu

Abstract— The intended applications of automatic face recognition systems include venues that vary widely in demographic

diversity. Formal evaluations of algorithms do not commonly

consider the effects of population diversity on performance.

We document the effects of racial and gender demographics

on the accuracy of algorithms that match identity in pairs

of face images. In particular, we focus on the effects of the

“background” population distribution of non-matched identities

against which identity matches are compared. The algorithm we

tested was created by fusing three of the top performers from

a recent US Government competition. First, we demonstrate

the variability of algorithm performance estimates when the

non-matched identities were demographically “yoked” by race

and/or gender (i.e., “yoking” constrains non-matched pairs to

be of the same race or gender). We also found a shift in the

match threshold required to maintain a stable false positive

rate when demographic control scenarios varied. These results

were verified with two independent data sets that differed

in demographic characteristics. In a second experiment, we

explored the effects of progressive increases in population

diversity on algorithm performance. We found systematic, but

non-general, effects when the balance between majority and

minority populations of non-matched identities shifted. Finally,

we show that identity match accuracy differs substantially

when the non-match identity population varied by race. The

results indicate the importance of the demographic composition

and modeling of the background population in predicting the

accuracy of face recognition algorithms.

I. INTRODUCTION

The appearance of a face is determined by its gender,

race/ethnicity, age, and identity. The goal of most face

recognition algorithms is to identify someone as a unique

individual. Often this test requires the algorithm to match

the identity of faces between images that may vary in the

quality or nature of the viewing conditions. The diversity of

faces in the real world means that face recognition algorithms

must operate over a backdrop of appearance variability that

is not related to an individual’s unique identity. Thus, face

recognition algorithms intended for real-world applications

must perform predictably over changes in the demographic

composition of the intended application populations. The

most likely application sites for algorithms include airports,

border crossings, and crowded city sites such as train and

metro stations. These locations are characterized by ethnically diverse populations that may vary by the time of year

(e.g., tourist season) or even by the time of day (e.g., flights

from the Far East arrive in the morning and European flights

in the afternoon).

The performance of state-of-the-art automatic face recognition algorithms has been tested extensively over the last

two decades in a series of U.S. Government-sponsored tests

(e.g., [1]). Measures of algorithm performance in these tests

provide the best publicly available information for making

decisions about the suitability and readiness of algorithms

for applications of importance to security and surveillance.

Traditionally, these tests have emphasized measuring performance over photometric conditions such as illumination and

image resolution [2], [3], [4], [5]. They have also concentrated primarily on the quality of the “match” (i.e., similarity)

between images of the same individuals, considering this

to be the critical factor determining algorithm performance.

When the degree of similarity between matched identities is

high, as is the case when photometric factors are controlled,

the algorithm is expected to perform well.

Much less consideration has been given to the distribution

against which matched identities are compared. To digress

briefly, all measures of the performance of face recognition

algorithms rely both on the distribution of data for the

population of identity matches (i.e., pairs of images of

the same person) and on the distribution of data for nonmatched identities (i.e., pairs of images of different people).

By definition, the identity match population contains image

pairs of the same race and gender. The non-matched identity

population, however, may be structured in several ways. To

date, in most evaluations, this distribution consists of pairs

of faces that have different identities and which may, or may

not, be of different races or genders.

More formally, identity match decisions (same person or

different people?) are generally based on a computed simi-
larity score between two face images. If this score exceeds a

threshold similarity score, the faces are judged as an “identity

match” (sometimes referred to as an “identity verification”).

Otherwise, the images are judged as non-matched identities.

False accept errors occur when the computed similarity score

between a pair of face images exceeds the match threshold.

In most applications, this threshold is set to achieve a low

false alarm rate (commonly on the order of 0.001).

As noted, in many formal evaluations of algorithms,

the distribution of similarity scores for the non-matched

face images includes face pairs that may differ in gender,

race/ethnicity, and age [1]. The inclusion of these categorically mismatched image pairs may lead to an over-estimation

of an algorithm’s ability to discriminate the identity of face

pairs. In other words, when non-match face pairs differ on

categorical variables such as race and gender, some part of

the performance of the algorithms may be due to the easier

problem of discriminating faces based on race or gender, than

to the more challenging problem of recognizing individual

face identities.

In this study, we focus on the problem of how the

demographic composition of non-match populations affects

estimates of algorithm accuracy. Although demographic variables have been shown to affect both human ([6], [7], [8],

[9]) and machine ([10], [11]) accuracy recognizing faces,

the effects of these variables in the non-matched identity

distributions have not been studied previously. The first goal

of this study was to document the effects of yoking nonmatch pairs according to the categorical variables of race

and gender, individually, and together. By “yoking” we mean

controlling the demographic variables within a non-match

pair, such that both faces in the pair are of the same gender,

same race, or same gender and race. We also examine the

implications of demographic control of the non-match pairs

for the choice of a threshold for match/non-match decisions.

The second goal was to examine algorithm accuracy with

systematic variations in the proportion of a “second” ethnic

group in the non-match distribution. Finally, we measured

algorithm accuracy when the identity matches were of a

particular race and identity mismatches were from another

race. This was compared to the case when the match and

non-match distributions were of the same race.

II. ALGORITHM FUSION AND TEST PROTOCOL

In this section, we overview the algorithm and test protocol

common to all experiments. The source of data for these

experiments was the FRVT 2006 [1], a U.S. Government

sponsored test of face recognition algorithms conducted by

the National Institute of Standards and Technology (NIST)

(Details of this test and its results can be found elsewhere,

[1]). We focused on the uncontrolled-uncontrolled identity

match test in the FRVT 2006 (Stimulus Set 1). In that

test, algorithms matched identity in pairs of images taken

under uncontrolled illumination conditions. To demonstrate

the general nature of the problem, we replicated a subset

of our results with a second data set (Stimulus Set 2) that

Fig. 1. Example same-identity image pair for the uncontrolled-uncontrolled

identity matching task.

TABLE I

SUMMARY DEMOGRAPHIC INFORMATION FOR STIMULUS SET 1.

Ethnicity Total Female Male

Caucasian 716 318 398

Asian 264 121 143

Hispanic 26 10 16

Southern Asian 23 2 21

Middle Eastern 6 1 5

African-American 12 1 11

Unknown 41 16 21

Total 1088 469 619

differed substantially in demographical characteristics from

Stimulus Set 1.

1) Stimulus Set 1: The Notre Dame multi-biometric data

set [1] was the primary source of face stimuli for the test.

This dataset consists of 9,307 images of 570 individuals.

The images were photographed with a 6 Mega-pixel Nikon

D70 camera and were taken under uncontrolled illumination

conditions, either outside or in a corridor or hallway. An

example image pair appears in Figure 1 and the relevant

demographic breakdown of faces appears in Table I. (Recall

that multiple images of individuals are represented in these

lists).

2) Stimulus Set 2: The Sandia data set, collected at

the Sandia National Laboratory, consists of high-resolution

frontal face images taken under both controlled and uncontrolled illumination [1]. The images were taken with a 4

Megapixel Canon PowerShot G2. The average face size for

the controlled images was 350 pixels between the centers

of the eyes and 110 pixels for the uncontrolled images. For

comparison with Stimulus Set 1, the demographic breakdown

of faces appears in Table II, with additional detail given to

the age of subjects, which is more variable in this data set.

3) Algorithm Test Procedure: In the FRVT 2006 evaluation, each algorithm computed a similarity score for all possible pairs of target and query images. This yielded a matrix of

similarity scores where element si,j represents the similarity

between the i

th target and the jth query image. The goal

of the algorithms was to distinguish matched-identity image

pairs and non-matched identity image pairs using the similarity scores. The performance of the algorithms was evaluated

at NIST using receiver operator characteristic (ROC) curves

that plot the proportion of false alarms (false accepts) against

the proportion of hits (identity verifications). Because face

recognition algorithms are required to operate at low false

accept rates, a second measure was also computed. This is
TABLE II

SUMMARY DEMOGRAPHIC INFORMATION FOR SEX, RACE, AND AGE IN STIMULUS SET 2. VALUES WITHIN A DEMOGRAPHIC CATEGORY ARE BY

PERCENT (NUMBERS ARE ROUNDED). IF THE NUMBER OF SUBJECTS IN A GIVEN CATEGORY IS LESS THAN 2.5%, THEN THE CELL IS LEFT BLANK.

Sex Race Age

Dataset Female Male Caucasian East Asian Hispanic 18-29 30-39 40-49 50-59 60+

Notre Dame 62 38 76 13 92 7

Sandia 55 45 64 21 15 11 23 35 18

the verification or hit rate at the 0.001 false accept rate. In

this paper, we will use both types of measures.

4) Algorithm Fusion Data: The experiments were conducted using data extracted from a fusion of three of the top

performing algorithms in the FRVT 2006. The fusion of the

similarity matrices operated by first estimating the median

and the median absolute deviation (MAD) from 1 in 1023

similarity scores (mediank and MADk reference the median

and MAD for the kth algorithm). The fused similarity scores

were computed as the sum of each algorithm’s similarity

scores after the median has been subtracted and then divided

by the MAD. Thus, if sk is a similarity score for the kth

algorithm, and

!

sf is a fusion similarity score, then sf =

sk− mediank / MADk.

The fused data were partitioned subsequently into three

performance strata, representing face image pairs that were

matched at high, moderate, and low levels of performance.

This performance stratification has been referred to elsewhere as the “Good, Bad, and Ugly (GBU) Challenge

Problem” [12]. In this study, we used face pairs from the

“bad” and “ugly” performance strata, which we will refer to

henceforth as the moderately difficult and the difficult face

pair conditions. The GBU challenge problem highlights the

broad range of algorithm performance when frontal image

matches are made. Of note, the pairs of identities used for

both the matched and mismatched data were held constant in

these strata. Thus, only the images of these identities differed

across the three strata. As such, the performance variations

among the three levels were due to photometric quality issues

(e.g., illumination) or to other extraneous variations in facial

expression rather than to recognizability differences of the

individual identities.

For any given matched identity pairing (i.e., images of the

same person), there were multiple similarity scores available

in all three performance strata. We used data from the

moderately difficult and difficult face pairs because they had

substantial error rates. In each condition, the fused algorithm

data consisted of a matrix of similarity scores between all

possible pairs of 1088 target face images and 1088 query

images. This included 3, 306 matched identity pairs and

1, 180, 438 non-matched identity pairs. Refer again to Table I

for the ethnic and gender composition of the identities in the

1, 088 target and query images used to create the similarity

matrix. As Table I makes clear, Caucasians and Asians are

the best represented races in the data set. We use only these

two races in the experiments we report in Sections IV and

V of this paper. For Section III, which used both Stimulus

Sets 1 and 2, all available races were considered.

III. DEMOGRAPHIC PAIRING IN NON-MATCH

IDENTITY DISTRIBUTIONS

The goal of these experiments is to document changes

in performance estimates for face recognition algorithms

as a function of the demographic characteristics of the

non-matched identity population. In the first section, we

show that performance estimates vary substantially when the

non-match population consists of face pairs that are yoked

by demographic groups (gender-only, race-only, gender and

race). In the second section, we examined the effects of demographic controls on the appropriate choice of a threshold

cutoff point for assigning an identity match decision to the

face pairs.

A. Partitioning the Performance Estimation Process by Demographic Bins

We computed ROC curves in four ways that vary in the

demographic controls applied to the non-matched identity

distribution. Figure 2 shows the fusion algorithm’s performance for the moderately difficult (left) and difficult (right)

face pairs. In the No Demographic Matching (No DM)

condition, we used all available pairs of mis-matched identity

to compute the ROC curve. Thus, non-match pairs could

be of the same or different gender and/or race. In the

Gender-matched Demographic (DM Gender Only) condition,

we used only same-sex pairs to compute the ROC curve,

although these pairs could be of a different race. In the Racematched Demographic (DM Race Only) condition, we used

only same-race pairs, although these pairs could be of a different gender. In the Gender and Race matched Demographic

(DM Gender-Race) condition, the non-matched pairs were of

the same-race and same-gender.

The graphs show a substantial decrease in performance as

the non-match distribution becomes more demographically

controlled. The difference in performance is further seen in

Table III, which shows the verification rates at the .001 false

alarm rate, as the demographic controls change. These rates

change markedly between the condition where there is no

demographic control and the condition where both gender

and race are controlled. Of note, the verification rate for

the difficult face pairs nearly doubles when the demographic

controls are removed completely. For the moderately difficult

pairs, the verification rate differs by 10 percent between the

no control and full control conditions.

Analogous ROC curves for the Sandia data set appear in

Figure 2. Similar to the results for Stimulus Set 1, the more
TABLE III

VERIFICATION RATE AT THE .001 FALSE ALARM RATE WITH VARIABLE

TYPES OF DEMOGRAPHIC CONTROL

Demographic Control Moderately Difficult Difficult

No DM .7925 .1515

DM Gender Only .7402 .1165

DM Race Only .7408 .1137

DM Gender and Race .6851 .0823

demographically controlled conditions showed decreased

performance. This finding bolsters our conclusions about the

effects of demographic control using an independent data set

with a markedly different demographic structure.

B. Estimating the Decision Threshold Based on Demographic Composition of Non-Matched Identities

In application-based scenarios, face recognition algorithms

use a threshold similarity score to determine if two face

images are the same identity or are different identities.

Threshold similarity scores are usually chosen to optimize

an operational criterion such as a false alarm rate. Typically,

thresholds for these automatic face recognition systems are

generally set to maintain a false alarm rate of 0.001. Here we

look at the implications of variable demographic controls in

the choice of a similarity threshold cutoff for identity match

decisions.

In Figure 3, we plot the false accept rate as a function of

threshold similarity score using the four demographic control

procedures described previously (No DM, DM Gender Only,

DM Race Only, DM Gender-Race). (Note that lower more

negative values indicate higher similarities.) The left graph

shows the function for the moderately difficult pairs and

the right graph shows the function for the difficult pairs.

The graph below shows the shift for the Sandia data set.

In all cases, the threshold that produces a false accept rate

of .001 is shifted for the different demographic control

conditions. Moreover, the full functions of thresholds are

similarly shifted. This indicates that to operate at a particular false alarm rate, threshold values must be set taking

into consideration the nature of demographic controls in

the non-match population. The results indicate that as the

demographic controls tighten, the similarity cutoff threshold

shifts to higher similarity scores.

IV. SIMULATIONS ON MIXED DEMOGRAPHICS

In real world applications, the representativeness of different demographic categories varies arbitrarily in different

population contexts, from nearly 100 percent of a single

majority race to various degrees of inclusion of (an)other

race(s). In this section, we systematically explore the effects

of progressive increases in population diversity on algorithm

performance. Again, we focus on diversity in the nonmatched identity distribution. We begin by measuring algorithm performance when only one race of faces is included

in the non-match identity distribution (Caucasian). Next,

we gradually increase the numbers of faces of a second

race (Asian) in this distribution and reassess performance.

We refer to this case as the Caucasian-to-Asian condition.

In a second simulation, we reversed the process. In this

case, we start with only Asians in the non-matched identity

distribution, and progressively include more Caucasians.

We refer to this case as the Asian-to-Caucsian condition.

In the Caucasian-to-Asian condition, the matched identity

distribution contained only Caucasian face pairs. In the

Asian-to-Causian condition, the matched identity distribution

contained only Asian face pairs. (In Section V we explore

other combinations of the identity match distributions.)

Because of the imbalance in the number of Caucasian

and Asian faces available in the database, these two conditions investigate different scenarios of population shifts. The

Caucasian-to-Asian condition simulates a population shift

that begins with a single race and ends with a population that

is characterized by a strong majority race and moderately

sized minority population (slightly over 10 percent). The

Asian-to-Caucasian condition simulates a population shift

that begins with a single race (Asian) and progressively shifts

to equal inclusion of the two races (Asian and Caucasian).

As additional Caucasian pairs are included, the population

continues to shift toward a majority of Caucasians and a

minority Asian population.

A. Methods

The stimulus set for this experiment included faces of the

two races that were best represented in the database: Asians

and Caucasians. (See Table I for a listing of numbers of

images of people of these races). Note that “Asians” in this

study refer to people from the Far East (e.g., China, Japan,

Korea). Recall that the original similarity matrix consisted of

10882 entries containing the similarity scores for all possible

pairs of the images. The data for this experiment consisted

of a subset of those similarity scores. Specifically, they were

the set of similarity scores from same-gender and same-race

Caucasian and Asian face pairs.

Identity Match Data. The identity match data for both

conditions did not vary across the simulations. For the

Caucasian-to-Asian condition, the similarity scores for all

possible pairs of Caucasian identity matches (n = 2110)

were used to create the identity match distribution. For the

Asian-to-Caucasian condition, the similarity scores of all

possible pairs of Asian identity matches (n = 866) were

used to create the identity match distribution. These numbers

represent approximately three images of each person in the

database, allowing for approximately three pairings of a

person with him/herself.

Identity Non-Match Distribution. For the Caucasian-toAsian condition, the simulation began with the non-match

distribution consisting of the similarity scores from all possible gender-matched Caucasian pairs of different identities

(n = 257, 418). For the Asian-to-Caucasian case, the simulation began with the non-match distribution consisting of

the similarity scores from all possible gender-matched Asian

pairs of different identities (n = 34, 224).

In both cases, face pairs from the other race, were added

into the non-match distribution, 1000 at a time, and the
10−3 10−2 10−1 100 0

0.2

0.4

0.6

0.8

1

Demographic Matching:Moderately Difficult Pairs

False Alarm Rate

Hit Rate

No DM

DM Gender−Race

DM Race Only

DM Gender Only

10−3 10−2 10−1 100 0

0.2

0.4

0.6

0.8

1

 Demographic Matching:Difficult Pairs

False Alarm Rate

Hit Rate

No DM

DM Gender−Race

DM Race Only

DM Gender Only

10−3 10−2 10−1 100 0

0.2

0.4

0.6

0.8

1

ROC − Sandia Dataset 

False Alarm Rate

Hit Rate

No DM

DM Race only

DM Gender Only

DM Gender−Race

Fig. 2. ROC curves for the moderately difficult (left) and difficult (right) face pairs with different kinds of demographic control on the non-matched identity

pairs. Performance is best when the pairs are uncontrolled (No DM) and worst when the pairs are same-gender and same-race pairs (DM Gender-Race).

The gender-only and race-only controls yield performance in between the no-control and full control conditions. Analogous ROC curves for the Sandia

stimulus set appear in row 2 and support these findings using a database with different demographic characteristics.

verification rate at the 0.001 false alarm rate was computed.

B. Results

For completeness and clarity, before reporting the results

of the demographic mixing in the non-match distribution,

we first present the baseline performance of the algorithm

for the Asian versus Caucasian face pairs in Figure 4. The

top two panels indicate that algorithm performance for the

moderately difficult face pairs was marginally better for the

Asian faces (left panel). When the face pairs were difficult,

the performance was consistently better for the Caucasian

faces (right panel). We do not offer a strong interpretation of

this finding, because it is arbitrarily dependent on the three

algorithms chosen for the fusion. Two of these algorithms

were from Western countries and the third algorithm was

from a country in East Asia. In the bottom two panels of

Figure 4, we see a substantial shift in the similarity threshold

that produces a constant false alarm rate when the face race

differs. Again this indicates the importance of demographics

in the choice of a threshold value chosen to achieve a

constant false alarm rate.

The results from the demographic mixing of the nonmatch distribution appear in Figure 5. The top lines in the

figure show the performance for the moderately difficult

pairs and the bottom lines show performance for the difficult

pairs. The blue lines show verification rate at 0.001 false

accept rate as Asian face pairs are added into the non-match

distribution for Caucasian face pairs. The red lines show this

rate as Caucasian face pairs are added into the non-match

distribution for Asian face pairs. As noted, the Caucasian to

Asian condition approximates a case where a minority race

gradually becomes better represented in the population until

it reaches its maximum, in this case at 13.3 percent of the

population.

Figure 5 shows that the effect of this gradual increase in

representation of the minority race is a steady decline in the

verification rate at the .001 false accept rate. The Asian-toCaucasian condition approximates a case where a minority

race gradually becomes better represented in the population

until it reaches its equilibrium with the first race (50 percent

diversity). Ultimately, Caucasians become the majority race

(red lines on the graph). Two results are apparent from

this simulation. First, in contrast to the Caucasian-to-Asian

simulation, which shows a theoretically comparable situation
−10 −5 0 5

10−3

10−2

10−1

100

Demographic Matching : Moderately Difficult Pairs

Threshold

False Alarm Rate

No DM

DM Gender−Race

DM Race Only

DM Gender Only

−10 −5 0 5

10−3

10−2

10−1

100

Demographic Matching:Difficult Pairs

Threshold

False Alarm Rate

No DM

DM Gender−Race

DM Race Only

DM Gender Only

−20 −15 −10 −5 0 5 10

10−3

10−2

10−1

100

Threshold

False Alarm Rate

Sandia data 

No DM

DM Race only

DM Gender Only

DM Gender−Race

Fig. 3. False alarm rate is plotted as a function of threshold similarity score, using the four demographic control procedures. (Note that lower more

negative values indicate higher similarities.) The left graph shows the this function for the moderately difficult pairs and the right graph shows the function

for the difficult pairs. The graph below shows analogous data from the Sandia stimulus set. The threshold shifts toward values in the target distribution

(i.e., higher similarity scores) as the demographic controls tighten.

up to 13 percent diversity, verification rate increases in this

range. This suggests that the effect on verification rate in

these situations is not a general result, but rather, is likely to

depend both on the percentage diversity and on the algorithm

itself, which may process faces of different races in different

ways. The second result is the gradual continued increase in

verification rate as Caucasians become the majority race in

the non-match distribution. Over the full range of diversity

percentages, the verification rate at the .001 false alarm rate

varies by as much as 0.12.

V. DEMOGRAPHICALLY “REVERSED” IDENTITY MATCH

AND MISMATCH DISTRIBUTIONS

In the final experiment, we explore the case when the

population of face matches to be detected are of a different

race than the population against which these matches will be

compared.

We compared verification rate at the .001 false alarm rate

with the following four combinations of race in the identity

match and identity non-match distributions: a.) Caucasians

in the match distribution and Caucasians in the non-match

distribution (Caucasian-Caucasian); b.) Caucasians in the

match distribution and Asians in the non-match distribution

(Caucasian-Asian); c.) Asians in the match distribution and

Asians in the non-match distribution (Asian-Asian); and d.)

Asians in the match distribution and Caucasians in the nonmatch distribution (Asian-Caucasian).

Table IV lists the verification rates at the .001 false

accept rate for the four conditions. There was a sharp drop

in verification rate from the Caucasian-Caucasian condition

to the Caucasian-Asian case. This was true for both the

moderately difficult and difficult face pairs. Surprisingly, the

inverse pattern was found in comparing the Asian-Asian

condition to the Asian-Caucasian case. In this latter case, the

verification rate increased when the race of the non-matched

identity distribution differed from the race of the matched

identity.

An explanation of this might be found in the mean of

the similarity distributions for the match and non-matched

identity distributions or in the form of the distributions

themselves. A careful inspection of our data suggested both

factors at work. Differences in the separation of the various

match and non-match distribution means explained some, but

not all, of the verification rate data in Table IV. An explana-
10−3 10−2 10−1 100 0

0.2

0.4

0.6

0.8

1

False Alarm Rate

Hit Rate

ROC of Moderately Difficult Pairs

Caucasian

Asian

10−3 10−2 10−1 100 0

0.2

0.4

0.6

0.8

1

False Alarm Rate

Hit Rate

ROC of Difficult Pairs

Caucasian

Asian

−20 −15 −10 −5 0 5 10

10−4

10−3

10−2

10−1

100

Threshold

False Alarm Rate

False Alarm Rate of Moderately Difficult Pairs

Caucasian

Asian

−20 −15 −10 −5 0 5 10

10−4

10−3

10−2

10−1

100

Threshold

False Alarm Rate

False Alarm Rate of Difficult Pairs

Caucasian

Asian

Fig. 4. ROC curves that show the performance of the fusion algorithm on the Caucasian and Asian face pairs (top panels). There is a marginal advantage

for Asian faces in the moderately difficult pair condition and an inversion of this pattern for the difficult pairs. Consistent with previous experiments, the

bottom panels of the figure show shifts in the similarity threshold functions that produce a constant false alarm rate.

0 20 40 60 80 100

0

0.2

0.4

0.6

0.8

1

Diversification of Demographics 

Percent Diversity 

Hit rate at .001 False Accept rate

Percent Asian

 Percent Caucasian

Fig. 5. Verification rate at the 0.001 false accept rate is plotted as a function

of the diversification of the the non-match distribution. When the non-match

matrix begins with 100 percent Caucasians (blue), verification rate declines

as the representation of the Asian minority race increases to its maximum of

13.3 percent. When the non-match matrix begins with 100 percent Asians

in the matrix (red), the verification rate increases as the representation of

the Caucasians increases, equals Asians, and reverses.

TABLE IV

VERIFICATION RATE AT THE .001 FALSE ALARM RATE WITH REVERSED

DEMOGRAPHICS.

Match Non-Match Moderately Difficult Difficult

Caucasian Caucasian .66 .08

Caucasian Asian .49 .03

Asian Asian .70 .06

Asian Caucasian .85 .16

tion of the remaining differences must come, therefore, from

subtle aspects of the shapes of the distributions themselves.

VI. CONCLUSIONS

In summary, all measures of the performance of face

recognition algorithms rely both on the distribution of data

for identity matches and on the distribution of data for

mismatched identities. Traditionally, attempts to improve the

performance of face recognition algorithms have emphasized

methods that increase the degree of match between images

of the same person (e.g., by bridging differences in illumination). Less consideration has been given to the effects of

the composition of the non-match identity distributions in

producing stable estimates of algorithm performance. These
estimates are important for predicting how the algorithms

will perform in real world environments. In this study,

we show that differences in the treatment of demographic

diversity in the non-match distribution can radically alter the

our estimates of algorithm performance.

The results of this study point to the following factors as

determinants of performance. First, the demographic pairing

of non-matched identity items can affect both the overall

level of performance estimated and the choice of thresholds

for match/non-match decisions. If no demographic pairing

constraints are imposed, and if the database is diverse, the

ability of algorithms to recognize unique identities will be

over-estimated. This is because some part of the performance

will be based on face categorization (e.g., by gender or

race) rather than identity discrimination. Second, we show

systematic, but not general, effects on the verification rate

at the .001 false alarm rate when a non-match identity

distribution increases in racial diversity. Third, again using

the measure of verification rate at the .001 false alarm

rate, we demonstrate that comparisons of match and nonmatch distributions based on the same and different race

of faces can lead to substantial differences in performance

expectations.

These second and third findings are particularly troubling

because they suggest that it may be difficult to reliably

predict algorithm performance without good estimates of

the way the match and non-match identity distributions are

structured demographically. They further hint at difficulties

in choosing operating parameters, such as a threshold match

score, that will lead to stable false alarm rates under varying

population scenarios. Combined, the present findings pose

a new and pressing challenge for this literature to find a

method for tuning algorithm performance to the constantly

changing demographic environments in which these systems

must operate reliably. Again, returning to the context of an

airport or tourist attraction, the choice of an appropriate

non-match distribution and threshold may have to be reassessed periodically and adjusted as needed. The present

study offers the first quantitative evidence on the importance

of considering the demographic “background” in setting

performance expectations for face recognition systems in the

field.

VII. ACKNOWLEDGMENTS

The authors gratefully acknowledge funding from

TSWG/DOD for this work. PJP also thanks the Federal

Bureau of Investigation (FBI) for their support of this work.

Thanks are also due to Vaidehi Natu for helpful comments on

this paper. The identification of any commercial product or

trade name does not imply endorsement or recommendation

by the National Institute of Standards and Technology or the

University of Texas at Dallas.

REFERENCES

[1] P. J. Phillips, W. T. Scruggs, A. J. O’Toole, P. J. Flynn, K. W. Bowyer,

C. L. Schott, and M. Sharpe, “FRVT 2006 and ICE 2006 large-scale

results,” IEEE Trans. PAMI, vol. 32, no. 5, pp. 831–846, 2010.

[2] R. Gross, S. Baker, I. Matthews, and T. Kanade, “Face recognition

across pose and illumination,” in Handbook of Face Recognition, S. Z.

Li and A. K. Jain, Eds. Sringer, 2005, pp. 193–216.

[3] P. J. Phillips, P. J. Flynn, T. Scruggs, K. W. Bowyer, J. Chang,

K. Hoffman, J. Marques, J. Min, and W. Worek, “Overview of the face

recognition grand challenge,” in IEEE Computer Society Conference

on Computer Vision and Pattern Recognition, 2005, pp. 947–954.

[4] P. J. Phillips, H. Moon, S. Rizvi, and P. Rauss, “The FERET evaluation

methodology for face-recognition algorithms,” IEEE Trans. PAMI,

vol. 22, pp. 1090–1104, October 2000.

[5] P. J. Phillips, P. J. Grother, R. J. Micheals, D. M. Blackburn, E. Tabassi,

and J. M. Bone, “Face recognition vendor test 2002: Evaluation

report,” National Institute of Standards and Technology, Tech. Rep.

NISTIR 6965, 2003, http://www.frvt.org.

[6] R. S. Malpass and J. Kravitz, “Recognition for faces of own and other

race faces,” Journal of Personality and Social Psychology, vol. 13, pp.

330–334, 1969.

[7] R. K. Bothwell, J. C. Brigham, and R. S. Malpass, “Cross-racial

identification,” Personality & Social Psychology Bulletin, vol. 15, pp.

19–25, 1989.

[8] C. A. Meissner and J. C. Brigham, “Thirty years of investigating

the own-race bias in memory for faces: A meta-analytic review,”

Psychology, Public Policy, and Law, vol. 7, pp. 3–35, 2001.

[9] P. N. Shapiro and S. D. Penrod, “Meta-analysis of face identification

studies,” Psychological Bulletin, vol. 100, pp. 139–156, 1986.

[10] N. Furl, P. J. Phillips, and A. J. O’Toole, “Face recognition algorithms

and the other-race effect: computational mechanisms for a developmental contact hypothesis,” Cognitve Science, vol. 26, pp. 797–815,

2002.

[11] P. J. Phillips, F. Jiang, A. Narvekar, and A. J. O’Toole, “An otherrace effect for face recognition algorithms,” ACM Trans. Applied

Perception, vol. 8, 2010.

[12] P. J. Phillips, J. R. Beveridge, B. A. Draper, G. Givens, A. J. O’Toole,

D. Bolme, J. Dunlop, Y. M. Lui, H. Sahibzada, and S. Weimer, “An

Introduction to the Good, the Bad, and the Ugly Face Recognition

Challenge Problem,” in Proceedings, Ninth International Conference

on Automatic Face and Gesture Recognition, 2011.