Exploratory Advanced Research . . . Next Generation Transportation Solutions

recognize objects that are not pedestrians and 

eliminate them from analysis, gaining real-time 

performance speed in hazard identification.

Layered-Processing Framework: 

Technical Approach

The goal of this EAR project is to produce a commercially adoptable, rapid, pedestrian-detection 

system with high accuracy and low false alerts. 

With this in mind, the system has already been 

tested to achieve a positive detection rate of 90 

percent at a distance of up to 35 m (114 ft), travelling at speeds up to 64 km/h (40 mi/h) under daylight and twilight conditions. The system does this 

using a layered framework, multi-cue multi-classifiers, and contextual scene understanding. The layered processing framework detects pedestrians 

and accurately and rapidly isolates them from other road objects. This is done by processing the layers through steps, each one increasing the certainty of detection and recognition. The project 

team aims to further enhance the speed and accuracy of the system and to issue different levels 

of warning, depending on the estimated time to 

collision.

Overcoming Challenges 

One of the major challenges on this project has 

been in the area of object recognition and context. 

The system has been developed to recognize the 

seven predetermined classes of roadway objects 

simultaneously, in real time at speeds up to 64 

km/h (40 mi/h), and then infer context from the 

scene. Although it is only possible to obtain partial 

and variable pose views of people and vehicles, 

the system has been successfully developed to 

generalize and handle multiple intra-class variations with limited examples and without models. 

Over the project period, the following technical 

challenges and objectives have been met: 

•	 Development of layered framework.

The Exploratory Advanced Research Program Fact Sheet

The Current State of Technology

At this time pedestrian detection can be achieved 

using fixed cameras pointing at predetermined 

zones, or through the use of vehicle-mounted camera systems. Both systems can detect pedestrians, 

while vehicle-mounted systems can also detect additional objects such as other vehicles, buildings, 

and zebra crossings. This EAR project moves beyond current systems and integrates two- and 

three-dimensional technology to provide cues for 

potential locations of pedestrians. 

Using multi-object classification, the system distinguishes seven classes of objects—ground, pedestrians, vehicles, buildings, bushes, trees, and 

other tall vertical structures such as light poles. The 

layered pedestrian-detection algorithm can then 

I

n 2005 alone, 64,000 pedestrians were 

injured and 4,882 were killed in the United States, with pedestrians accounting for 

11 percent of all traffic fatalities and 2 percent of injuries. The focus of “Layered Object Recognition System for Pedestrian 

Collision Sensing,” an Exploratory Advanced Research (EAR) Program project, 

is a real-time, in-vehicle, vision-only system to detect pedestrians and to determine potential collisions with high accuracy and minimal false alarms. Now 

nearing completion, the 2-year study 

launched by the Federal Highway Administration (FHWA), is being conducted by 

the Sarnoff Corporation in Princeton, New 

Jersey. 

Real-Time Pedestrian Detection 

Layered Object Recognition System for Pedestrian Collision Sensing
•	 Detection of pedestrians or other vulnerable road 

users at expected and unexpected locations.

•	 Discrimination between pedestrians, vehicles, 

and other roadside objects.

•	 Design of image processing methods for contrast 

enhancement to compensate for low light 

conditions.

•	 Demonstration of a real-time implementation of 

all of the above.

Future Efforts

“The success of this EAR project offers many benefits to roadway users,” says Wei Zhang at FHWA. 

“It has the potential to significantly reduce accidents 

and casualties between motor vehicles and other 

roadway users, such as bicycles and motorcycles, 

eventually at a price and performance point that 

crucially enables wide and mainstream adoption by 

vehicle manufacturers.” Zhang continues, “In addition, the high level of detection being developed 

could eventually enable more than just visual warning systems to be activated for pedestrian protection but could even be tied into activating passive 

safety systems. In fact, as decision quality increases, less expensive, single-use safety devices could 

be used with less risk of inadvertent deployment 

and subsequent additional cost to customers.”

Learn More

For more information on this EAR Program project, 

contact Wei Zhang at FHWA, 202-493-3317 (email: 

wei.zhang@dot.gov).

What Is the Exploratory 

Advanced Research Program?

FHWA’s Exploratory Advanced Research (EAR) 

Program focuses on long-term, high-risk research with a high payoff potential. The program addresses underlying gaps faced by applied highway research programs, anticipates 

emerging issues with national implications, 

and reflects broad transportation industry 

goals and objectives. 

To learn more about the EAR Program, visit 

the Exploratory Advanced Research Web site 

at www.fhwa.dot.gov/advancedresearch. The 

site features information on research solicitations, updates on ongoing research, links to 

published materials, summaries of past EAR 

Program events, and details on upcoming 

events. For additional information, contact 

David Kuehn at FHWA, 202-493-3414 (email: 

david.kuehn@fhwa.dot.gov), or Terry Halkyard 

at FHWA, 202-493-3467 (email: terry.halkyard@

fhwa.dot.gov).

Publication No. 

FHWA-HRT-10-022

HRTM-04/11-09(1M)E

Real-Time Pedestrian Detection 

Layered Object Recognition System for Pedestrian Collision Sensing