FRVT

NIST Frequently Asked Questions Page 1 of 6

1

Face Recognition Vendor Test

Ongoing

Frequently Asked Questions (FAQs)

VERSION 1.1

Patrick Grother

Mei Ngan

Kayee Hanaoka

Information Access Division 

Information Technology Laboratory

Contact via frvt@nist.gov

November 6, 2019
FRVT

NIST Frequently Asked Questions Page 2 of 6

2 Table 1 – Questions Related to images and subjects

# Question Answer

1. You enumerate certain image types so 

that algorithms can alter their 

processing, will NIST provide examples 

of these?

In short, no. The properties of visa and passport photos are widely known because of 

their (approximate) conformance to ISO/IEC 19794-5:2005, so NIST will not provide 

those. A sample of mugshots is available in the MEDS dataset. A sample of 

photojournalism images is available in the IJB-A set, and perhaps LFW. Wild images 

can be obtained in volume from social media e.g. flickr. We consider wild images to 

include photos from both amateur and professional photographers. We cannot 

provide child-exploitation images and we advise developers that access and 

possession of such images is illegal in most jurisdictions. 

2. So why does NIST not provide training 

images?

NIST and the FRVT program do not train face recognition algorithms. We do not 

provide training data to software, and software is prohibited from adapting to any 

data we pass to the algorithms. Training of face recognition algorithms is not a turnkey operation; instead it is typically an extended process involving researchers 

curating suitable training sets, establishing architectures and hyperparameters, and 

running trials over days or weeks, and then evaluating the output. The result of such a 

process, which is often iterative, is usually a “trained model” i.e. static data and 

parameters that can be saved and provided to NIST as an integral part of the black-box 

recognition engine. NIST does not support training, because our tests seek to mimic 

operational reality and, there, algorithms are almost always shipped and used “as is” 

without any training or adaptation to customer data. The representation of the face, 

as described by the “model”, is fixed until the software is upgraded.

3. What range of subject ages are present 

in NIST image collections.

We will evaluate on images from children and adults, so the age range varies from a 

few months old to at least 90 years old.

4. Will a minimum and maximum face size 

be provided?

No. But note that if the image type label is “wild”, “photojournalism”, “unknown” or 

“child exploitation” then very widely divergent sizes are typical. For “mugshot” and 

“iso” types, then we intend to provide faces with interocular distance 40 <= IOD <= 

300 pixels.

5. What are the main characteristics of 

faces in child exploitation imagery?

The problems are five-fold: 1. Faces are from children aged anywhere from 0 to 17 

inclusive. 2. The faces can have very high roll angles, consistent with a subject lying 

down, in addition to unconstrained yaw and pitch. 3. The faces can be occluded by 

hair, limbs and other objects. 4. The faces can have very low to very high resolution. 

Some photos are similar to generic “Wild” photos.

6. Will NIST run tests on 3D images. We do not anticipate any tests with 3D images.

7. Will NIST compare images of two types, 

e.g. Wild with VISA.

Yes, because it is reflective of some operational use-cases, e.g. Passport e-Gates. NIST 

will additionally compare images within type e.g. mugshot vs. mugshot.

8.

9.

10.

11.

12.

13.

3

4
FRVT

NIST Frequently Asked Questions Page 3 of 6

5 Table 2 – Questions Related to Number and Schedule of Algorithm Submissions

# Question Answer

14. How many algorithms can 

we submit?

Unless otherwise noted in a specific FRVT track’s test plan, participants may send one submission 

as often as every four calendar months from the last submission for evaluation. NIST will evaluate 

implementations on a first-come-first-served basis, and quickly publish results.

15. Why does NIST limit the 

frequency of submissions

To limit the rate at which developers can tune to the test data.

To limit the computational resources NIST requires to run the tests.

16. Is there a deadline for 

submitting an algorithm?

No, FRVT will run indefinitely, so developers may submit their algorithms on a continuous basis.

17.

18.

19.

20.

21.

6

7
FRVT

NIST Frequently Asked Questions Page 4 of 6

8 Table 3 – Questions Related to Reporting of Results

# Question Answer

22. Where will NIST report 

results?

NIST will place PDF test reports on the FRVT website, and include HTML tables there also. NIST may 

include results in NIST interagency reports, academic publications, and in conference, workshop 

and seminar presentations.

23. NIST will produce a test 

report for an algorithm, 

what other algorithms will 

appear in the test report.

NIST will regenerate and republish test reports as soon as any new algorithm completes a test. So, 

for example, the report for a university’s X1 algorithm will be updated a year later when results 

from a company’s Q4 algorithm are available. This is intended to support comparison of 

algorithms.

24. How many algorithms will 

NIST list on the main FRVT 

website?

Two. NIST will designate algorithms as new or old. NIST will treat algorithm submissions as a FIFO 

queue. The two most recent algorithms will be regarded as “new”. NIST will run the new 

algorithms on all currently active tests. When an updated algorithm is submitted, the oldest 

algorithm will be assigned “old” status, and may not be run on new test datasets. So, for example, 

suppose a year from now NIST obtains a new database of passport photographs, NIST will run the 

two most recent (“new”) algorithms on it, but will not normally run the “old” algorithms. It is 

possible in this scenario for an old algorithm to outperform the more recent algorithms and not 

appear on the main results pages.

25. Will NIST report results for 

old algorithms?

NIST will archive all test reports on the FRVT website. NIST will not ordinarily list results on the 

main page, but reserves the right to do so.

26. What performance metrics 

will NIST report, beyond 

accuracy and speed.

NIST will report error rates, for whole datasets, for specific populations groups, and for specific 

types of images. We will also report processing durations, software and data sizes, template sizes, 

memory consumption. We reserve the right to report other metrics, for example on the biometric 

zoo. We may report any anomalous behavior. 

27.

28.

29.

30.

31.

32.

9

10

11
FRVT

NIST Frequently Asked Questions Page 5 of 6

12 Table 4 – Questions Related to Software Properties and Behavior

# Question Answer

What is a submission to 

FRVT?

A submission is software implementing the API and other requirements given in the particular 

FRVT test’s API document.

33. What is the order of NIST 

activity?

1. Validation. 2. Timing. 3. Full evaluation. 4. Report generation. 5. Publication. Completion of each 

step is a necessary condition for progression to the next.

34. What is the point of 

validation?

Validation serves solely to ensure software run at NIST gives identical outputs to software run by 

the developer. The validation images are not necessarily representative of images NIST will use. 

The images do not support training, nor testing.

35. What happens if we fail 

validation?

Usually validation succeeds: Both templates and scores are identical. If validation fails, we will 

email you with details of the observed differences. You may then modify and re-submit the 

algorithm, or you may request that NIST continues to test the algorithm ignoring the failed 

validation.

36. Will NIST evaluate face 

recognition software running 

on Windows or MacOS?

No. We run on a cluster of computers equipped with a linux variant. This is economical and 

reflective of many government and commercial back-ends. We anticipate the organizations should 

be able to separate their core algorithmic code from, from example access control or GUI-driven 

products.

37. What happens if the 

software is too slow?

NIST will send email with indicating the speed and ask you to resubmit a faster implementation. 

NIST will discontinue execution. Results will not be posted. We reserve the right to post that 

software was received and was unable to complete the test.

38. What happens if the 

software crashes?

NIST will send email with whatever diagnostic information we have, and will discontinue 

evaluation. We will ask you to resubmit a fixed implementation. If the revised software fails, we 

will repeat this process. However, if we judge that the software is too unreliable, we will ask you 

to do a more extensive software review and ask you to submit at a later time. 

39. What happens if the 

software does not make a 

template?

NIST will send email with whatever diagnostic information we have, and will discontinue 

evaluation. We will ask you to resubmit a fixed implementation.

NOTE: Given an input MULTIFACE, the createTemplate function must return a template even if the 

software cannot detect a face or chooses not to extract features. The function may return a nonzero error code as enumerated in the API document, but must always produce a template. See the

documentation for createTemplate in the API document.

40. What libraries can we 

supply, e.g. OpenCV or 

Caffé?

You may provide zero or more supporting libraries. These libraries should all reside in one lib 

directory with the main FRVT library that implements our API. The standard libraries can be seen 

in our linux distribution. Any other libraries should be sent to us.

41. Can our software contact 

servers?

No, your code must not attempt to access any device or resource except main memory, unless we 

explicitly specify it in the API. If NIST detects that software attempts to connect to, or open, or 

read, any server, socket, connection, port or other resource, we will stop all tests for an 

organization, delete all test reports. We may disclose the violation, and take additional steps.

42. How should we name our 

software submission?

The core implementation library shall be named according to the specifications in the 

corresponding API document. The provider identifier must be an appropriate, relevant, noninfringing name of the main provider of the software. If the provider identifier is not 

representative of the submitting organization, NIST reserves the right to reject the submission for 

evaluation.

43.

44.

45.

46.

47.

13

14
FRVT

NIST Frequently Asked Questions Page 6 of 6

15 Table 5 – Questions Related to Participation

# Question Answer

48. Is the participation 

agreement a requirement to 

get our algorithm tested in 

FRVT?

Yes, developers must send in a signed participation agreement for the FRVT track they would like 

to participate in. Additionally, 1) all correspondence from the provider to NIST must come from an 

organizational email address of the participating entity – no generic emails from gmail.com, 

mail.ru, qq.com, etc. will be accepted; 2) the point of contact listed in the participation agreement 

must specify an organizational email address; 3) the organization must have a non-trivial web 

presence.

49.

50.

51.

52.

53.

54.

55.

56.

57.

58.

59.

60.

61.

62.

16