Visual Analytics Dataset 

Generation

MARCIA KIMURA

REBECCA ERIKSON

NICK LOMBARDO

Pacific Northwest National Laboratory

Improving Biometric and Forensic Technology: The Future of Research Datasets
Improving Biometric and Forensic Technology: The Future of Research Datasets

Toyota Center in Kennewick, WA

January 30, 2015 2

 Live testing and 

collection with crowds

 GIS calibrated 

coordinates down to 3 

inches for hundreds of 

locations

Datasets from this location:

Human Tracking at a Distance

Facial Recognition in 

Live Crowds

Building Threat Scenarios

Multipurpose venue serving as a test-bed for wide 

range of security technologies
Improving Biometric and Forensic Technology: The Future of Research Datasets 

Dataset Collection Overview

January 30, 2015 3

Define 

Data 

Objectives

Define 

Testing 

Approach

Develop 

Test Plan

Venue 

Stakeholder

Approvals

Privacy 

Approvals

(PNNL/DHS)

Test Plan 

Approval

Dry Run

Recruit 

Role 

Players

Conduct 

Tests

Annotate 

Data

Data Set/Final 

Report 

Released

Analysis

Media 

Strategy
Improving Biometric and Forensic Technology: The Future of Research Datasets 

Simple Ideas for making a dataset useful 

for multiple purposes

January 30, 2015 4

Multiple cameras

Vary camera heights

Vary FOV 

Collect extra data

Background/empty video

Mugshots at multiple angles

Collect candid images 

Annotate extra information as time 

allows

Building Threat Scenario Data Collection
Data Collection Summary

Improving Biometric and Forensic Technology: The Future of Research Datasets January 30, 2015 5
Facial Recognition Video Dataset

“Non-Cooperative”

Operationally realistic corpus of video data to support the evaluation and 

enhancement of facial recognition systems technology

PNNL role players with public crowds in 5 indoor live events

1) one-way crowd flow

2) two-way crowd flow

3) linear and serpentine queues

147 hours of video data

11 cameras – consumer grade with SD memory 

Pixels on target

Varied pitch and yaw

24 fps at 1920 x 1080 

H.264 .mp4

Collected 2153 still photographs for “Watch List” of the 64 unique role players

Improving Biometric and Forensic Technology: The Future of Research Datasets January 30, 2015 6
Watchlist Composition

“Non-Cooperative”

January 30, 2015 7

16MP mugshots 

at 14 angles, 

repeated with 

eyeglasses if 

available

976 images of 64 individuals 

spanning up to 20 years

600 ppi scans ID images

Low resolution “port of entry” 

for 2 angles

Improving Biometric and Forensic Technology: The Future of Research Datasets 
FR Video Dataset Cameras

“Non-Cooperative”

Queue Cameras

Improving Biometric and Forensic Technology: The Future of Research Datasets January 30, 2015 8
FR Video Dataset Cameras

“Non-Cooperative”

Hall Cameras

Improving Biometric and Forensic Technology: The Future of Research Datasets January 30, 2015 9
FR Video Dataset Annotation

“Non-Cooperative”

14,401 annotated presentations

Ground truth annotation includes:

Role player alias

FOV entry/exit times of each role player

Qualitative assessment of

Face detection viability of presentation

Reason if low viability (occlusion, angle, etc.)

Duration of each best face presentation

Number of role players in FOV

Crowd density

Detailed report of video collection activities and results

Improving Biometric and Forensic Technology: The Future of Research Datasets January 30, 2015 10

3 separate presentations. 
Building Threat Video Dataset

“Resilient Buildings”

Video analytics systems and human operator evaluation for enhanced 

building security during observable threat events

Up to 12 role players in 

Event scenarios that include 

Active shooter (single and multiple weapons and persons)

Unauthorized entry 

Unauthorized parking

Leave-behind packages

Continuous video of background that contains no events 

Sparse video that contains occasional events

Variations in crowd density, dynamics, and clothing contrast

Variations in lighting conditions 

250 video clips (1-5 minutes each); >150 hours of raw video

8 cameras at various distances, elevations, and FOV, with some 

overlapping FOV

Data could be used for FR evaluations

Improving Biometric and Forensic Technology: The Future of Research Datasets January 30, 2015 11
Building Threat Video Dataset

“Resilient Buildings”

Video clip ground truth includes:

Role player alias

Video clip event scenario details (props/posture etc. as scripted)

Video clip length

Event time

Number of people in clip 

Lighting, weather, and other parameters

Improving Biometric and Forensic Technology: The Future of Research Datasets January 30, 2015 12
Standoff Tracking Video Dataset

“Microcorpus”

Scripted video data to support vendor development and testing of 

people tracking technology

Up to 27 role players in scenarios that include 

1 way, 2-way, and random crowd flow

Variations in crowd density, dynamics, and clothing contrast

Variations in lighting conditions (dawn through dusk, sunny and cloudy) 

Total of 231 video clips (1-3 minutes each); 25 hours of raw video

Recorded outdoors at 30 fps at 1920 x 1080 H.264 .mp4

3 cameras each at standoff distances from elevations of ~30 feet 

and partially overlapping fields of view

Video was collected in a geospatially-mapped region 

Improving Biometric and Forensic Technology: The Future of Research Datasets January 30, 2015 13
Standoff Tracking Video Dataset

“Microcorpus”

Video clip ground truth includes:

Video clip scenario

Video clip length

Number of role players

Light conditions

Contrast of attire

Crowd density

Props (bags, coolers, etc.)

People-to-people interactions

Extended annotation of select clips 

include ViPER formatted XML of 

individual bounding boxes that can 

be used in the F4DE Toolkit for 

performance evaluation of systems.

Improving Biometric and Forensic Technology: The Future of Research Datasets January 30, 2015 14
Summary

PNNL has access to a unique venue that can accommodate a wide 

range of video data sets needs, including those with live crowds

A repeatable process has been developed to enable a cost-effective 

collection of video data for forensic video analytic needs

Extensive experience in the annotation of data sets ensures that data 

sets can be used for benchmarking AND enhancement of video analytics 

technologies

Highly annotated non-cooperative, resilient building, and standoff tracking 

data sets are available for government partners (subject to DHS S&T PM 

approval)

Improving Biometric and Forensic Technology: The Future of Research Datasets January 30, 2015 15
Contact Information

PNNL Points of Contact

Nick Lombardo

PNNL Project Manager

nick.lombardo@pnnl.gov

Marcia Kimura

PNNL Project Manager

marcia.kimura@pnnl.gov

DHS Point of Contact

Patricia Wolfhope

DHS S&T Program Manager

patricia.wolfhope@dhs.gov

Improving Biometric and Forensic Technology: The Future of Research Datasets January 30, 2015 16