A System for Real-time
Detection and Tracking of Vehicles
from a Single Car-mounted Camera
Claudio Caraffi, Tomas Vojir, Jiri Trefny, Jan Sochman, Jiri MatasToyota Motor Europe

Center for Machine Perception, University of Prague

ITS Conference, Anchorage, 18 Sep 2012

An “easy” problem
Vehicle detection & tracking on motorways:• Only vehicle rear, rigid object• Limited street furniture, no pedestrian• Single camera• Variable lighting condition
Quantitative evaluation:• Extended dataset → expensive annotation• Laser-scanner for semi-automatic annotationof ~30000 frames. Dataset released.

Algorithm architecture• Target initialized by vehicle detector (Wald Boost)• KLT features tracking (Flock of Trackers)• LrD: Learn specific target and re-detect to correct
tracker drift (Randomized Forest)• Wald Boost detections are also used to correct
known targets → Precise bounding box• In case of tracker failure, WB & LrD try to recover
the target in a Kalman-filter predicted
position → long-term vehicle identity maintenanceVehicle Detection & Tracking

Vehicle detection and tracking

WaldBoost Detector 1• WaldBoost:• Sliding window algorithm• Sequence of weak classifier(AdaBoost)• Focus on time to decision• After each classifier, verify if
we can take a decision “notobject” (Asymmetric)

WaldBoost Detector 2•••5000 car images (and a few trucks…) training dataset
One billion negative samples
Performance ontest sequence:•

1.74 average classifiers evaluated per window for rearvehicle detection
Limits: longer time to decision in crowded traffic scenes.

•J. Sochman and J. Matas, “WaldBoost - Learning for Time Constrained Sequential Detection,” in CVPR 2005.
J. Trefny and J. Matas, “Extended Set of Local Binary Patterns for Rapid Object Detection,” in CVWW 10: Proceedings ofthe Computer Vision Winter Workshop 2010.

Vehicle detection and tracking

Flock of trackers (FOT) 1••••••

Target divided in cells
One KLT feature trackerper region
KLT feature pre-extraction skippedTracker evenly placed, “naturally”
converge to a good feature
Trackers are evaluated (in/outlier):- Previous: Forward-backward
KLT check, cost is twice- Neighbourhood consistency (Nh)- Markov model predictor (Mp)
- Norm. cross correlation (NCC)- Combination: 10% KLT time
Estimation of translation & scaling

Flock of trackers (FOT) 2• Fast computation (4.8 ms)

Comparison with recently published methods on public sequences• Effective for rigid objects• Failure in case of strong illumination changes
T. Vojir and J. Matas, “Robustifying the Flock of Trackers,” in Computer Vision Winter Workshop, 2011.

Vehicle detection and tracking

Learn & re-Detect• At target confirmation (after 3 detections)create a specific random forest classifier (RF):Insert as positive samples the current
target window and its affine warps• Keep updating the RF, collect:• Negative samples: RF detector firing
outside the tracked object• Positive samples: current target window if
similarity (NCC) > 0.75
• Limit: Similar vehicles in the scene
Z. Kalal, J. Matas, and K. Mikolajczyk, “P-N Learning: Bootstrapping Binary Classifiers by StructuralConstraints,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2010.

Scheduling (10 Hz objective)

Only one expensive operation per frame:• WB detector (every 3rd frame)• Random forest generation (frame after 1st det.)• LrD (other frames): correct position for eachtarget, collect negative samples for one target
Note: WB runs every 3 frames, 3 detections
to confirm a target → delay up to 0.9 s

Results 1

Quantitative evaluation:
Dataset and annotation• Acquired in December 2011 in cooperation
with VisLab, University of Parma, Italy• 28 clips for a total of ~27 minutes• Two 1024x768 color cameras,
IBEO 4-layer laser-scanner, ego-trajectory• Idea: Use laser-scanner data to extract
automatically vehicles• Project vehicles into the image to create an
approximated Ground Truth (GT’)

Laser-scanner detections
Easy scenario:• Extract surfaces perpendicular to the driving
direction, discard static objects• Project into the image using fixed calibration
parameters and a flat-ground assumption

Features of GT’• Consistent ID available• 3D position available• Object width estimated over full time ofobservation• Car – Truck classification based on width

Limits of GT’• No motorbike• Unreliable beyond 60-70 meters ( < 3 laserreflections)• Imprecise target side boundaries(quantization and noise)• No vehicle length (although possible)• No vehicle height (arbitrarily set)
• Static calibration insufficient for projection(oscillation and non-flat ground)Oscillation: best pitch matching

Vision algorithm

GT’ pitch correction

GT’ fixed calibration

Match GT’ ↔ Algorithm results• Given the mentioned limitations, we
introduce a custom overlap measurement

AREA
TERM

POSITION
TERM

Different sensor positions• A correct detection can be classified false positive• An object not visible in the image can be classified
false negativeDifferent sensor positions• The vehicle marked in yellow is
not considered in the statistics

Quantitative evaluation
Daylight:••••

Very few false positives
Delay of detection reduces “recall rate”
Poor results for trucks
Statistics beyond 70 meters not significant

Quantitative evaluation
Daylight:

Sunset:Results

Conclusions• Real-time reliable detection and tracking of cars
• Next steps: measure & reduce confirmation time
focus on trucks & other classesdifferent scenarios• Dataset is made public• Images available, ground truth + software Oct ’12• http://cmp.felk.cvut.cz/data/motorway/• … or google: “motorway dataset”

Thanks!

http://cmp.felk.cvut.cz/data/motorway/