Evaluation of Real-Time 

Face Recognition 

Technologies 

for Video Surveillance 

Applications

Dmitry Gorodnichy and Eric Granger

Science and Engineering Directorate

Video Surveillance and Biometrics Section

NIST International Biometric Performance Conference (IBPC 2012) 

Gaithersburg MD, March 5-9, 2012
Outline

PROVIT – evaluate state-of-the-art commercial 

technologies and academic systems for FRiVS: 

̶ public data sets for medium- to large-scale evaluation

̶ experimental protocols for different still-to-video and 

video-to-video surveillance applications, e.g.,

• screening of faces according to their resemblance to a 

wanted list

• matching a face across several video feeds

• fusion of face recognition from different cameras while tracking 

a person

̶ performance measures: transaction-based (P-R curve) 

and subject-based (biometric menagerie) analysis
Outline

1. Background – Face Recognition in Video Surveillance

– objectives and challenges

– where biometrics meets video surveillance

– academic and commercial solutions

2. Evaluation of Systems for FRiVS 

– publicly-available data sets and lab mock-up

– specialized performance metrics and protocols

– CBET: multi-order score analysis, threshold-validated analysis

3. TRL-based evaluation

– Issues with conventional performance evaluations

– Integrating FR into operational CCTV environment

– PROVE-IT (FRiV) methodology & results

– Preliminary TRL assessment
1) Face Recognition in Video Surveillance

A Generic System for FR in Video
Enhanced screening and 

situation analysis across a 

network of surveillance cameras

̶ automatically recognize and track 

individuals within dense and moving 

crowds, as found at major events and 

airports 

̶ determine if faces captured in video 

streams correspond to individuals of 

interest populating a restrained list of 

individuals

0.7 

0.6 

0.6 

0.2 

0.5 

input ROI

watchlist

1) Face Recognition in Video Surveillance
Problem statement

ROIs extracted from video frames (probes) are matched 

against facial model of individual of interest

• Still-to-video recognition: facial model of each individual

consists of 1+ templates extracted from a gallery of stills

Typical CBSA application: watchlist-based surveillance

• Video-to-video: facial model of each individual are

extracted from videos

Typical CBSA application: operator captures an individual of interest 

in a video stream and the system tracks him over a network of cameras

1) Face Recognition in Video Surveillance
Environments are complex and change over time due to: 

̶ low quality and resolution of video frames

̶ limited control of acquisition conditions – variation in poses, 

expressions, illumination, cooperation of individuals, occlusion...

̶ inter- and intra-class variability and noise in the feature space

̶ ageing and variation of interaction between sensor and individual

̶ facial models are often poor representatives of real faces

̶ highly skewed data distributions: very few positives (from individuals of 

interest) w.r.t. negative samples (from open world) 

Computational resources – video surveillance networks are 

comprised of a growing number of IP-based cameras

̶ transmit or archive massive quantities of data

̶ memory requirements: storage and retrieval of facial models

̶ processing time: matching ROIs against facial models

1) Challenges of FR in Video Surveillance
Biometric Setup

Faces captured in controlled environment (as in eGates with e-Passport) are much easier to recognize

Still images captures from these environments may 

provide:

̶ canonical face model adopted by ICAO’02 for 

passport-type documents

̶ high resolution (60 pixels between eyes)

̶ well positioned face (front-faced, eye-level) 

without occlusion (eye-glasses, scarf) 

̶ neutral facial expression

̶ high quality:

 no motion, blur, compression artifacts, etc

 in focus

 best possible illumination

1) Face Recognition in Video Surveillance
9 

1) Challenges of FR in Video Surveillance

Taxonomy of Surveillance Setups

Type 0: Cooperative Biometric setup (access control, eGate) 

Type 1: semi-constrained setup

– primary inspection lane (PIL) 

Type 2: unconstrained free-flow, one-at-time

– port of entry / chokepoint entry

Type 3: unconstrained free-flow, many-at-time 

– airport

Type 4: Outdoor (no lighting or structural constraints)

1 

2 

3 
Author Description Recognition Set Tracking Applications

Beveridge CSU Ellastic Graph 

Bunch Matching

Still-to-video, local Both No watch list screening

Zhou

2003

Simultaneous Face 

Tracking and Recognition

Still-to-video, videoto-video, hollisticClosed Yes access control

Ekenel

2007

Local Appearence-Based

Face Models

Video-to-video, 

hollistic

Open No access controll

Stallenkamp

2008

Local Appearence-Based

Face Models

Video-to-video, 

hollistic

Open No watch list screening 

Kamgar-Parsi

2011

Face Morphing to Boost 

Training Data

Still-to-video, local Open No watch list screening

Li

2005

TCM-kNN Still-to-still, hollistic Open No watch list screening

Connolly

2010

Evolving ensembles

using Dynamic PSO

Video-to-video, 

holistic

Closed No access control

Pagano

2011

Adaptive Ensemble of 

Detectors

Video-to-video, 

hollistic

Open Yes watch list screening

1) Face Recognition in Video Surveillance

Survey of Academic Solutions
1) Face Recognition in Video Surveillance

Survey of Commercial Technologies

Technology Vendor Typ

e 

Track Approach Applications

Verilook

Surveillance 

SDK

Neurotechnology SDK Multiple Still-to-video, 

video-to-video

Face anotation, watch list screening, 

enrollment from video, multi-modal 

biometrics

FaceR Animetrics SDK No Still-to-still Watch list screening, enrollment from 

video

FaceIT SDK L1 SDK No Still-to-still Watch list screening, multi-modal 

biometrics

PittPatt SDK Google* SDK Multiple Still-to-video, 

video-to-video

Face anotation, watch list screening, 

enrollment from video

FaceVACS Cognitec SDK Multiple Still-to-video, 

video-to-video

Face anotation, watch list screening, 

enrollment from video

Acsys FRS

SDK

Acsys SDK Multiple Video-tovideoFace anotation, watch list screening, 

enrollment from video

SureMatch 3D Genex App No Still-to-still Watch list screening

Notiface II FACE-TEK App No Still-to-still Watch list screening

Face First Face First App No Still-to-video Watch list screening
12

Public Data Sets for for medium- to large-scale evaluation

2) Evaluation of Systems for FRiVS
13

Data sets for FRiV - summary

datasets have been characterized according to: 

̶ demographics: distribution of individuals per session and 

in the entire dataset;

̶ complexity in scene: the systematic variations of 

illumination, motion, occlusion, expression and/or pose for 

some target application;

̶ capture properties: the number and type of cameras, 

duration of video sequences, frame rate and resolution. 

2) Evaluation of Systems for FRiVS
14

CMU – FIA (mono-modal, 1 face) 

• PIL: subjects mimicking passport checkpoint at 

airport

2) Evaluation of Systems for FRiVS
15

Chokepoint (mono-modal, 1 to 24 faces)

• CATSA checkpoint: subjects walking though 

portals

2) Evaluation of Systems for FRiVS
16

Performance metrics

Fundamental task under evaluation:

̶ independent, user-specific detection of an individual of 

interest among a restrained cohort of individuals

̶ data from a restrained cohort ≠ universal world model

2) Evaluation of Systems for FRiVS
17

Performance Metrics

Open-set FR problem with imbalanced class distributions 

(few positive samples from a restrained cohort)

– precision-recall space, and F-scores for transaction-based analysis 

Complex environment and uncertainty of facial models

– quality of acquired ROIs and tracks

– test for confidence or significance on quality estimates

Performance varies across a population of individuals, 

and some individuals are harder to recognize

– menageries – statistical tests to characterize individual

Growing complexity of surveillance networks

– analysis of time and memory complexity

2) Evaluation of Systems for FRiVS
18

Transaction-Based Analysis

Evaluation of detectors – count correct and incorrect 

decisions over a test set, and express performance trade-offs

using a curve or scalar metric

̶ Traditional: ROC or DET curves (accuracy, AUC, pAUC) 

̶ Dependent on class distributions and miss-classification costs: 

precision-recall (F-score), ROC isometrics, cost curves and others

2) Evaluation of Systems for FRiVS
19

Transaction-Based Analysis

Results on FIA data

2) Evaluation of Systems for FRiVS
20

Subject-Based Analysis

Doddington’s zoo – performance is assessed with 

different types of individuals in mind

̶ performance of face recognition systems may vary drastically from 

one individual to the next

̶ an analysis of these individuals and their common properties can:

• expose fundamental weaknesses in a biometric system

• schemes for user-specific thresholds, score normalization, and 

fusion

2) Evaluation of Systems for FRiVS
21

Transaction-Based Analysis

Results on FIA data: pAUC(10%) and F1-measure

(Pagano at al. IEEE WCCI 2012)

2) Evaluation of Systems for FRiVS
22

2) Evaluation of Systems for FRiVS

Comprehensive Biometrics Evaluation Toolkit 

(CDET)

Developed by CBSA-S&E for evaluation of biometric 

systems for border control for:

1. 1-to-N entry control applications, e.g., to investigate the risks of having 

non-confident matches in iris systems 

2. 1-to-M screening applications, e.g., to evaluate stand-off and iFR

technologies

Integrates best practices and recommendations, such as:

– all ISO-SC 37 / NIST metrics

– Multi-order score analysis

– subject-based analysis 

– “Non-confident” match analysis for fully-automated systems

– Threshold-validated ranking analysis

– Case studies (iris, face, voice)
23

3. TRL-Based Evaluation 

FRiV applications

 

Type 0 

(eGate) 

Type 1

 

Type 2

 

Type 3

 

Face Tracking

(in consecutive frames)

√ ? ? − 

Face Detection √ √ √ ? 

Face Grouping / Tagging

(across multiple feeds)

? ? ? − 

Instant “Watch List” 

Screening

? ? − − 

Forensic examination 

from video (off-line)

√ ? ? − 

Expression analysis √ ? ? − 

Face + Voice + Iris √ ? - − 

Video-to-video

face matching

√ ? ? − 

Soft biometrics 

(e.g., height)

? ? ? − 

Technology Readiness: Preliminary assessment 
24

Conclusions

• Current COTS and academic systems can be found useful 

for some FRiSV applications

• Post-processing and pre-processing (including Video 

Analytics) are critical for their success 

• Potential for new video-based (eg spatio-temporal 

recognition) techniques, as opposed to status-quo stillimage-based.

• There is no all-inclusive evaluation methodology for FRiVS

– conventional metric can be misleading

– for operational agency, TRL-based evaluation should prevail

• Ultimate metric - satisfaction of the end-user (border 

officer)!
25

Backup Slides 

25
26

Adaptive Multi-Classifier Systems

(Pagano et al., IEEE WCCI 2012)
27

27

Adaptive Multi-Classifier Systems

(Pagano et al., IEEE WCCI 2012) 

Classification and decision architectures for 

open-set FR
28

28

Adaptive Multi-Classifier Systems

(Pagano et al., submitted to IEEE WCCI 2012) 
29

29

Adaptive Multi-Classifier Systems

(de la Torre et al., IEEE WCCI 2012) 

Incremental learning of new data using L&C
30

30

Adaptive Multi-Classifier Systems 

(Connolly et al. PR2011) 

• Framework – a ‘swarm’ of incremental classifiers, a 

dynamic optimization module and a LTM:
31

31

Adaptive Multi-Classifier Systems

Adaptive Fusion: Incremental Boolean

Combination