Face Recognition Vendor Test 2002

Performance Metrics

Patrick Grother, Ross Micheals

and P. Jonathon Phillips

31 March 2003

The paper is published as NIST IR 6982 and will appear in

the Proceedings of the Fourth International Conference on

Audio-Visual Based Person Authentication, in June 2003.

The paper details the metrics used to quantify the performance of

the face recognition systems tested in FRVT 2002. The methods are

suited to any recognition evaluation, online or offline, technology

or scenario, for which complete similarity scores are archived.

The paper shows that the open-set identification problem known as

the watch list task is the general case: it requires systems to perform

1:N recognition with concurrent possibilities of false acceptance and

rejection. Two special cases are demonstrated: 1:1 Verification is

simply the watch list task with N=1; and closed-set identification

is that with no false accept rate.

The paper also presents the computation of standard error ellipses

used to show the effect of population variation on false accept and

false reject rates.

Face Recognition Vendor Test 2002 Performance

Metrics

Patrick Grother, Ross J. Micheals, and P. Jonathon Phillips

National Institute of Standards and Technology, Gaithersburg MD 20899, USA,

pgrother@nist.gov

Abstract. We present the methodology and recognition performance

characteristics used in the Face Recognition Vendor Test 2002. We refine

the notion of a biometric imposter, and show that the traditional measures of identification and verification performance, are limiting cases

of the open-universe watch list task. The watch list problem generalizes the tradeoff of detection and identification of persons of interest

against a false alarm rate. In addition, we use performance scores on

disjoint populations to establish a means of computing and displaying

distribution-free estimates of the variation of verification vs. false alarm

performance. Finally we formalize gallery normalization, which is an extension of previous evaluation methodologies; we define a pair of gallery

dependent mappings that can be applied as a post recognition step to

vectors of distance or similarity scores. All the methods are biometric

non-specific, and applicable to large populations.

1 Introduction

The evaluation protocol [2] used in FRVT 2002 [4] is a framework for the quantitative determination of the performance of recognition technologies using arbitrary biometrics. Specifically, it applies to either online (live human subjects)

or offline (stored imagery) testing as long as it produces output recognition data

that is available for subsequent analysis. This has the advantage that tests can

be conducted uniformly and fairly across participants, the results are repeatable,

and very large image sets can be tested expeditiously. We first present normalization in section 1.1 and then introduce the terminology of the FRVT 2002

protocol and define our performance metrics in section 2.

In the FRVT 2002 protocol a system is required to compare two biometric

signatures and report a scalar similarity score. A biometric signature can be an

arbitrary ensemble of pieces of imagery from an individual, for example a face

and a fingerprint, or face and voice audio sequence. In FRVT 2002, both single

face stills and video sequences were used. We use the term image to include

such modalities, unless stated otherwise. A similarity score is a measure of the

sameness of identity of the individuals appearing in the images. Without loss of

generality the protocol requires that images of the same person have a large similarity score. Systems reporting distance measures, where a small value indicates

sameness of identity, have their values negated before any processing.
2

The FRVT 2002 tests are structured around sets of images. An algorithm

compares all images in a query set, Q, with all images in a target set T . The

result is a similarity matrix whose ij-th element is the similarity between the

i-th element of T and the j-th element of Q. The matrix is computed and stored

in column order; each column corresponds to an unknown query image being

compared with all the known, enrolled, target images. In the general case the

matrix is rectangular, but for FRVT 2002 we used the special case T = Q.

This framework allows for great flexibility in arriving at quantitative results.

The sets T and Q are not normally used directly. Instead we consider virtual

image sets, a conceptual innovation first defined in the FERET protocol[3]. Here

a gallery, G, a subset of T , contains identically one signature per subject and

represents the set of images that have been enrolled in a biometric system.

Likewise a probe set, PG, is a subset of Q. Each of its images have a match

in the gallery, and represent a legitimate user. The images of a third set, the

imposter set PN , also a subset of Q, do not have a match in the gallery. This

set represents persons attempting to defeat a system. A match describes the

comparison of probe and gallery images of the same individual. A non-match

likewise arises from images of different persons.

The utility of this framework is that many different recognition experiments

are embedded within T and Q. All the results of FRVT 2002 are obtained from

the similarity elements corresponding to the rows defined by the subset G, and

the columns defined by PG and PN . Together these form a similarity matrix

S from which various performance statistics are computed. Disjoint gallery and

probe sets allow performance to be estimated on particular recognition tasks. For

example to explore the effect of subject ageing, a gallery containing the earliest

image of all persons is used with a probe set of more recent images.

This protocol evaluates recognition technologies rather than deployed systems. Particularly it ignores efficiency and performance when databases are partitioned. See Wayman [5] for a treatment of these issues.

1.1 Normalization

The FRVT 2002 protocol allows a normalization option. This is a post-processing

transform of similarity scores, that may exploit the fact that the gallery, unlike

the target set, contains only one image per person by definition. Specifically, a

vector, s, contains the column of elements sip from S corresponding to the single

probe p against gallery G. Normalization is defined as a function, f : RN → RN ,

mapping s to a new vector, t. For an algorithm that uses normalization, the final performance scores are computed over these transformed values. Notably the

normalization option is operationally realistic only if each probe is processed independently of all others. Certain unjustified performance gains may be realized

if normalization were allowed across probes.

Two classes of normalization functions were allowed in FRVT 2002. The

first is simply t = f1(s). The second also takes a matrix of similarities, SGG,

between all gallery pairs, whose off-diagonal elements contain information available to an operational system. The second form of normalization, then, is simply
3

t = f2(s, SGG). Notably, the use of this second form is impractical for even moderate gallery sizes because SGG imparts O(N2

) resource constraints. FRVT 2002

participants were given the option of submitting different f1 and f2 functions

for each of the recognition tasks: identification, verification and watch list, discussed in the next section. The functions were supplied to, and run by, NIST as

functions in an object file.

It should be noted that the use of normalization causes all searches to become

one-to-many operations. The verification task, detailed below, is usually considered a one-to-one search and is correspondingly efficient, but if normalization is

to be used then a gallery, of some size and content, must be constructed. If the

gallery is changed, normalization must be recomputed.

2 Performance Metrics

In FRVT 2002, we evaluate an algorithm on three related tasks: identification,

verification, and watch list, and we state separate appropriate statistics for each.

As described above, performance on each of these tasks is obtained solely from

the similarity values extracted from the similarity matrix and from the subject

identities. In a proctored test such as FRVT 2002, identity information is not

available to the recognition systems.

The watch list problem is a generalization of both identification and verification. For this task, a probe p is compared to a gallery which we term the watch

list. The probe ranks the gallery, so we state performance as an identification

rate. However a significant operational constraint is that a system should not

attempt identification of individuals not on the watch list. We must also, therefore, measure a false alarm rate. This makes clear that the generalized watch list

problem is defined over an open-universe.

In the next three subsections we formalize the watch list problem and show

that verification is the special case when the watch list size is 1, and identification

conversely is a closed universe watch list task.

2.1 Watch List

We measure the watch list performance using a watch list G and two probe

sets: PG with subjects who should be identified and, PN with true imposters

who should not throw an alarm. The former is used to state the detection and

identification rate equal as the fraction of probes in PG that are detected at or

above threshold t and recognized at rank r or better:

PDI (t, r) =

|{pj : rank(pj ) ≤ r, sij ≥ t, id(pj ) = id(gi)}|

|PG|

∀pj ∈ PG (1)

where the rank is defined as the number of watch list entries which have greater

than or equal similarity to the probe than the matching entry:
4

rank(pj ) = |{gk : skj ≥ sij , id(gi) = id(pj )}| ∀gk ∈ G. (2)

Throughout we use i and k to subscript gallery elements, corresponding to rows

of the similarity matrix, and j to subscript the probe sets corresponding to

columns of the matrix. In practice, this needs to be modified to handle tied

similarity values: we elected to use the mean of the lower and upper ranks of the

run of tied values:

2 rank(pj ) = |{k : skj ≥ sij , id(gi) = id(pj )}| + (3)

|{k : skj > sij , id(gi) = id(pj )}| + 1

The imposter set is used to compute the false alarm rate as the fraction of

probes from PN whose similarity to any gallery image is at or above threshold:

PF A(t) =

|{pj : maxi sij ≥ t}|

|PN |

∀pj ∈ PN ∀gi ∈ G (4)

2.2 Identification

Identification is a special case of the watch list task: If we mandate a closed

universe, then the false alarm rate is undefined and a pure identification rate

specifies performance. Formally for each probe p from PG we sort the similarity

scores against gallery G, and obtain the rank of the match. Identification performance is then stated as the fraction of probes whose gallery match is at rank r

or lower. If the set of probes with a close match is

C(r) = {pj : rank(pj ) ≤ r} ∀pj ∈ PG (5)

where the rank is defined as before. We now define the Cumulative Match Characteristic (CMC) to be the identification rate as a function of r:

PI (r) =

|C(r)|

|PG|

(6)

which we plot as the primary measure of identification performance. It gives an

estimate of the rate at which probe images will be classified at rank r or better. It

is a non-decreasing function of r. Although the CMC is most often summarized

with rank one performance, other points and the steepness of the curve are also

relevant operationally. One drawback of the characteristic is its dependence on

gallery size, |G|. For this reason we also plot identification performance at fixed

rank as a function of the gallery size.

2.3 Verification

The use of biometric systems for the verification task is perhaps more common

than identification. The operational model assumes that a probe pj is compared

with its matching gallery image and that the single similarity score is compared
5

1

10

100

Rank 1000 0.001

0.01

0.1

1

False Accept

Rate

0

0.2

0.4

0.6

0.8

1

Detection and

Identification Rate

Fig. 1. Identification rate as a function of rank and false alarm rate for a watch list

size of 3000. Note the weak dependence on rank, except at high false accept rates. The

horizontal plane gives the rank one closed-universe identification rate.

against a threshold to verify the individual or otherwise. Two types of error

can occur in this process: first a false accept in which an imposter claims an

identity and is matched by the system above threshold; and secondly a false reject

in which the system incorrectly matches the individual below threshold. This

method for accepting or rejecting a claim models the Neyman-Pearson observer.

A Neyman-Pearson model maximizes the verification rate for a constant false

accept rate[1].

The Receiver Operating Characteristic (ROC) is computed to quantify verification performance. It shows the tradeoff between the two types of error by

plotting estimates of the verification rate, PV (i.e. the true accept rate) against

the false accept rate, PF A as a parametric function of the prior operating threshold, t. The verification rate is the fraction of probes whose gallery match has

similarity greater than or equal to the threshold value, t:

PV (t) =

|{pj : sij ≥ t, id(gi) = id(pj )}|

|PG|

∀pj ∈ PG (7)

The false accept rate is computed over the set of imposters, PN , containing

those individuals who are not present in the gallery:

PF A(t) =

|{sij : sij ≥ t}|

|PN | |G|

∀pj ∈ PN , (8)
6

1

10

100

Gallery Size 1000 0.0001

0.001

0.01

0.1

1

False Accept

Rate

0

0.2

0.4

0.6

0.8

1

Detection and

Identification Rate

Fig. 2. Watch list performance. The rank one detection and identification rate is plotted

as a function of gallery size and false alarm rate.

where the denominator shows our use of all the non-matching similarities in

order to improve the fidelity of our PF A estimate. In reality an imposter would

usually claim just one identity, but our method of using all scores is realistic

unless an imposter has a known prior resemblance to a person.

Note that this definition differs from that for verification false accept rate.

Here we determine if there exists at least one gallery image more similar to

the imposter than t. This occurs more frequently than in verification where the

number of scoes above t between an imposter and all galleries are counted.

The use of a true imposter set PN contrasts to “round-robin” evaluations,

in which PG and PN are the same set. From an operational standpoint, this

models the case where a subject, already with legitimate access to the system

(they are in PG), attempts to gain access to the very same system, under a

different identity. There may be some specialized scenarios where this is a valid

model. However, we prefer to model the situation in which a person who does

not already have access to the system attempts verification. In this model, the

persons (not just the images) in PN are different from those in the gallery. The

rationale for using true imposters is that the non-match distributions estimated

from SGPN and SGPG may be different.

An empirical ROC is not a curve but a sequence of steps corresponding to

a set of operating points where PV and/or PF A changes. Because systems are

operated on the ROC’s convex hull which corresponding to PV changes, we use

the |PG| match values as thresholds. We sort these values, keep their unique

subset as ti, i = 1 . . . |PG| and insert the artificial threshold t0 = −∞. This
7

avoids the use of a much large number of non-matches |G||PN | in the ROC

calculation. The computation of PV proceeds by binning the number of match

values from SGPG that are in the range [ti,ti+1), and finally by accumulating all

of them. The same thresholds and procedure are used for PF A from SGPN .

2.4 Limiting Cases

Figures 1 and 2 show watch list performance as functions of watch list sizes, false

alarm rates and rank. They were generated from the similarity scores of a leading

FRVT 2002 participant. For galleries of size 1 the watch list measures reduce

to verification; the intersection of the surface and the (1,y,z) plane is the ROC

curve. The CMC curve is the plane (x,1,z) of figure 1 and similarly identification

performance as a function of gallery size is given by the intersection of the surface

and the (x,1,z) plane, as shown in figure 2. These two cases correspond to the

relaxation of the threshold, t → −∞ whence PF A → 1 then the detection and

identification rate becomes the identification rate, i.e. PDI (1, r) = PI (r).

2.5 Comparing Verification Performances

Frequently we seek to compare verification results, either between systems, or

across different data sets processed by the same system. Between systems this

may be accomplished by looking at the verification rate at fixed false accept

rates. This is appropriate because it is not possible, nor operationally feasible, to

set a uniform threshold across different systems that report on different ranges

and scales. However, for studies using just one system and many gallery and

probe sets, fixed PF A values correspond to different thresholds, which are not

operationally realizable. The correct approach is to set a single threshold t and

acknowledge that the PV (t) and PF A(t) are random variables across experiments.

Thus, to be able to compute variation in verification and false accept rates

across multiple galleries and probe sets, we must compute the ROC using a fixed

global set of thresholds. Formally, if R experiments use R different G, PG and

PN image sets, we extract the sorted union of R sets of match scores. These

thresholds will generally “oversample” each individual ROC. Thus we have R

(PV , PF A) pairs at each threshold, which we plot with an error ellipse that traces

two standard errors in the PV and PF A dimensions. The principal axes of this

ellipse are the eigenvectors of the covariance matrix of the R pairs. This is shown

in figure 3.

As a summary statistic we usually report the PV and PF A at a threshold that

gives PF A ≈ 0.01. We favor this over equal error rate (when PF A(t) = 1−PV (t))

because PF A = 0.01 is an operationally realistic number, while equal error rate

not only varies, but is usually higher.

3 Conclusion

The paper details the evaluation framework and scoring metrics used in the Face

Recognition Vendor Test 2002. We have outlined the concept of operational re-
8

0.7

0.75

0.8

0.85

0.9

0.95

1

0.001 0.01 0.1 1

Verification Rate

False Accept Rate

Fig. 3. Verification variation. The figure shows the results of computing ROC curves

from 23 disjoint populations of size 800.

alizability based on the real-world usage of systems with fixed operating thresholds. We show how variation in verification performance must be computed at

the same threshold. Motivated by operational relevance we have defined true

imposters and shown via the non-match distribution that their use is necessary

in evaluations. We define the watch-list scenario and show that verification and

identification are special cases of it.

References

1. K. Fukunaga. Statistical Pattern Recognition, chapter 3. Academic Press, second

edition, 1990.

2. R. J. Micheals, P. Grother, and P.J. Phillips. The NIST Human ID Evaluation

Framework. In Proceedings of the Fourth International Conference on Audio- and

Video-based Biometric Person Authentication, June 2003.

3. P. J. Phillips, H. Moon, S. A. Rizvi, and P. J. Rauss. The FERET Evaluation

Methodology for Face Recognition Algorithms. IEEE Trans. Pattern Analysis and

Machine Intelligence, 22:1090–1104, 2000.

4. P.J. Phillips, P. Grother, R. J. Micheals, D. M. Blackburn, E. Tabassi, and M. Bone.

Face Recognition Vendor Test 2002. Evaluation Report IR 6965, National Institute

of Standards and Technology, www.itl.nist.gov/iad/894.03/face/face.html, March

2003.

5. J. L. Wayman. Error-Rate Equations for the General Biometric System. IEEE

Robotics and Automation Magazine, pages 35–48, 3 1999.