Exploratory Advanced Research . . . Next Generation Transportation Solutions

Automated Video Analysis

Analyzing Large Quantities of Transportation Research Data

be focused on the video from the camera pointed 

directly at the driver’s face and upper torso.

Overcoming a Data Bottleneck

The traditional way to extract features of interest 

from video is for technicians to physically sit in 

front of a monitor and manually scroll through 

frame by frame, logging the location where each 

feature is found. With approximately 1.2-million 

hours of video in the NDS dataset, it is estimated 

that it would take almost 600 technicians a full 

year, working 40 hours per week, to go through 

all the video. Automation of these feature extraction tasks would dramatically reduce the time 

and cost involved. 

In accordance, the EAR Program is funding 

a total of six projects, including the CMU research 

project, that are directly or indirectly focused on 

making the SHRP 2 NDS data more accessible. 

The objective of these research projects is to 

advance efficient and cost-effective methods 

and tools to analyze large amounts of safetyrelated video data. Such tools will reduce the 

time and costs required to pull needed features 

from these rich datasets, and thus expand 

the pool of researchers able and willing to use 

such data in their research. The CMU research 

project aims to address the challenge by automating the feature extraction so that the details 

that researchers need can be quickly and accurately extracted.

Machine-Learning Approach

The research team for this CMU study implemented a machine-learning–based approach. 

This approach feeds a multitude of features into 

learning algorithms that are trained on very 

large datasets and capable of learning with high 

accuracy exactly which features are important, 

which provide less useful data, and how features 

can be used in combination to accurately recognize and classify the desired targets. This means 

that the great diversity provided by large and 

complex datasets adds to the power and robust 

performance of the system.

The Exploratory Advanced Research Program Fact Sheet

Naturalistic Data

The SHRP 2 NDS involved almost 3,000 volunteers going about their regular activities with extensive instrumentation, including video and still 

cameras, a Global Positioning System, and many 

other sensors attached to their personal vehicles 

for up to 2 years. The NDS data will provide unparalleled insight into the connections between driver behaviors, road design, traffic, and many other 

topics that have previously been difficult, if not 

impossible, to directly investigate, or were limited 

in scale by the intimidating logistics of collecting 

and analyzing this much data. The major challenge is decoding the video data. For example, 

one researcher might be interested in how drivers 

react to particular elements of the overall roadway 

environment whereas another researcher might 

The second Strategic Highway 

Research Program (SHRP 2) Safety 

research area has produced unprecedented 

data for highway researchers in the form 

of the naturalistic driving study (NDS). 

With approximately 2 petabytes of data 

available to researchers, most of which is 

video data, new automated tools for data 

extraction and analysis are required. 

Developing a prototype tool was the goal 

of the Exploratory Advanced Research 

(EAR) Program project, “Machine Learning 

for Automated Analysis of Large Volumes 

of Highway Video.” The National Robotics 

Engineering Center, at Carnegie Mellon 

University (CMU), conducted this research, 

which was funded by the Federal Highway 

Administration (FHWA) in 2012.
Contextual Cues

To extract useful information from petabytes 

of data generated by moving or static video 

cameras, the research team integrated state-ofthe-art algorithms to exploit contextual clues, 

reduce the amount of expert effort required to 

guide learning, and manage the computational 

effort required. When humans interpret sensor 

data, they rely on contextual cues, such as inferring from a patch of image showing a car that it 

is likely to lie above an image of a roadway. Roads 

and vehicles offer many other similar contextual 

cues that can be exploited within a scene. 

Researchers traditionally encode these contextual relationships in a graphical model; however, 

predictions may be inaccurate in these models. 

This study focuses on a novel set of methods to 

automate the interpretation of ambiguous video 

data by building a sequence of context dependent predictions. Instead of a traditional graphical 

model, this approach defines an inference procedure (e.g., wheels are found near the corners of a 

car), which can then be used to classify features 

moving forward.

Data Challenges

The research team implemented powerful labeling 

tools to enable fast human analysis and labeling 

of data, anomaly detection to prioritize data 

access and focus labeler time on significant 

events, and semi-supervised learning to build 

predictive models and to minimize the amount of 

labeled data required. In addition, with datasets 

running into petabyte sizes, there is a critical 

need for computational systems that can be 

scaled to match. The researchers considered 

two different approaches for managing such 

computation- and data-intensive machine learning. One approach is parallelization, involving the 

distribution of work across multiple cores in a 

network. The other approach is a new class of 

techniques, known as anytime predictors, used to 

provide a sequence of results that improves as 

more time is provided to analyze test data.

What Is the Exploratory 

Advanced Research Program?

FHWA’s Exploratory Advanced Research 

(EAR) Program focuses on longer term, 

higher risk research with a high payoff 

potential. The program addresses underlying gaps faced by applied highway 

research programs, anticipates emerging 

issues with national implications, and reflects broad transportation industry goals 

and objectives. 

To learn more about the EAR Program, 

visit the Exploratory Advanced Research 

Web site at www.fhwa.dot.gov/advanced

research. The site features information on 

research solicitations, updates on ongoing 

research, links to published materials, 

summaries of past EAR Program events, 

and details on upcoming events.

Automated Video Analysis

Analyzing Large Quantities of Transportation Research Data 

Publication No. 

FHWA-HRT-15-070

HRTM-30/08-15(WEB)E 

Photo credit, page 1: 

Carnegie Mellon University 

A typical challenging situation 

for identifying discreet 

objects in a cluttered urban 

environment.

Future Applications

The primary outcome of this project is a software 

framework to support efficient application of 

machine-learning-based feature extraction techniques for very large sets of transportation video 

data, as well as specific target detectors to support 

various transportation analysis tasks. An additional 

benefit will be to expand industry understanding of 

a highly effective data-processing approach that 

can be widely used to process large volumes of 

future data. The goal is to be able to apply this 

approach to both moving targets (e.g., cars and 

pedestrians) and static targets (e.g., traffic signs).

Learn More

For more information on this project, contact 

Lincoln Cobb, FHWA Office of Safety Research 

and Development, at 202-493-3313 (email: lincoln.

cobb@dot.gov).