Are we ready for Autonomous Driving?
The KITTI Vision Benchmark Suite
Andreas Geiger† Philip Lenz† Raquel Urtasun‡†

K ARLSRUHE I NSTITUTE OF T ECHNOLOGY‡TOYOTA T ECHNOLOGICAL I NSTITUTE AT C HICAGOAndreas Geiger(KIT)

Philip Lenz
(KIT)

Raquel Urtasun(TTI-C)The Challenge of Autonomous Cars

State of the art
Localization, path planning, obstacle avoidance
Heavy use of 3D laser scanner and detailed maps
Problems for computer vision
Stereo, optical flow, localization
Object detection, recognition and tracking
Semantic segmentation, 3D scene understanding
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

2

The Challenge of Autonomous Cars

State of the art
Localization, path planning, obstacle avoidance
Heavy use of 3D laser scanner and detailed maps
Problems for computer vision
Stereo, optical flow, localization
Object detection, recognition and tracking
Semantic segmentation, 3D scene understanding
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

2

The Challenge of Autonomous Cars

State of the art
Localization, path planning, obstacle avoidance
Heavy use of 3D laser scanner and detailed maps
Problems for computer vision
Stereo, optical flow, localization
Object detection, recognition and tracking
Semantic segmentation, 3D scene understanding
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

2

The Challenge of Autonomous Cars

3D Laserscanner

State of the art
Localization, path planning, obstacle avoidance
Heavy use of 3D laser scanner and detailed maps
Problems for computer vision
Stereo, optical flow, localization
Object detection, recognition and tracking
Semantic segmentation, 3D scene understanding
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

2

The Challenge of Autonomous Cars
Depth/reﬂectance map
for 3D localization

Detailed road map
for path planning

State of the art
Localization, path planning, obstacle avoidance
Heavy use of 3D laser scanner and detailed maps
Problems for computer vision
Stereo, optical flow, localization
Object detection, recognition and tracking
Semantic segmentation, 3D scene understanding
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

2

The Challenge of Autonomous Cars

State of the art
Localization, path planning, obstacle avoidance
Heavy use of 3D laser scanner and detailed maps
Problems for computer vision
Stereo, optical flow, localization
Object detection, recognition and tracking
Semantic segmentation, 3D scene understanding
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

2

The Challenge of Autonomous Cars

State of the art
Localization, path planning, obstacle avoidance
Heavy use of 3D laser scanner and detailed maps
Problems for computer vision
Stereo, optical flow, localization
Object detection, recognition and tracking
Semantic segmentation, 3D scene understanding
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

2

The Challenge of Autonomous Cars

State of the art
Localization, path planning, obstacle avoidance
Heavy use of 3D laser scanner and detailed maps
Problems for computer vision
Stereo, optical flow, localization
Object detection, recognition and tracking
Semantic segmentation, 3D scene understanding
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

2

The Challenge of Autonomous Cars

State of the art
Localization, path planning, obstacle avoidance
Heavy use of 3D laser scanner and detailed maps
Problems for computer vision
Stereo, optical flow, localization
Object detection, recognition and tracking
Semantic segmentation, 3D scene understanding
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

2

The KITTI Vision Benchmark Suite
Two stereo rigs (1392 × 512 px, 54 cm base, 90◦ opening)
Velodyne laser scanner, GPS+IMU localization
6 hours of recordings, 10 frames per second

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

3

The KITTI Vision Benchmark Suite
Two stereo rigs (1392 × 512 px, 54 cm base, 90◦ opening)
Velodyne laser scanner, GPS+IMU localization
6 hours of recordings, 10 frames per second

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

3

The KITTI Vision Benchmark Suite
Two stereo rigs (1392 × 512 px, 54 cm base, 90◦ opening)
Velodyne laser scanner, GPS+IMU localization
6 hours of recordings, 10 frames per second

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

3Sensor Calibration Challenges

Camera ↔ camera calibration



Velodyne ↔ camera registration oGPS ↔ Velodyne registration

Geiger et al., ICRA 2012
ICP + Hand-eye calibration

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

4Sensor Calibration Challenges

Camera ↔ camera calibration



Velodyne ↔ camera registration oGPS ↔ Velodyne registration

Geiger et al., ICRA 2012
ICP + Hand-eye calibration

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

4Sensor Calibration Challenges

Camera ↔ camera calibrationTC

TC

TVelodyne

Velodyne ↔ camera registration oGPS ↔ Velodyne registration

TGPS

Geiger et al., ICRA 2012
ICP + Hand-eye calibration

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

4Data Annotation Challenges

3D object labels: 22 Annotators
Occlusion labels: Mechanical Turk
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

5Data Annotation Challenges

3D object labels: 22 Annotators
Occlusion labels: Mechanical Turk
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

5Data Statistics

Number of Labels

150k

100k

50k

0

r
Ca

t
)
c
n
n
k
Va Truc stria itting yclis Tram Mis
e
C
s
d
(
e
P rsonPe

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

6Data Statistics

Number of Labels

150k

100k

50k

0

r
Ca

t
)
c
n
n
k
Va Truc stria itting yclis Tram Mis
e
C
s
d
(
e
P rsonPe

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

6Data Statistics

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

6Novel Challenges
Middlebury Stereo Evaluation – Version 2

Average errors: 2 − 3% (non-occluded regions)A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

7

Novel Challenges
Middlebury Stereo Evaluation – Version 2

Average errors: 2 − 3% (non-occluded regions)A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

7

Novel Challenges
Fast guided cost-volume filtering (Rhemann et al., CVPR 2011)

Middlebury, Errors: 2.7%

Error threshold: 1 px (Middlebury) / 3 px (KITTI)
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

8

Novel Challenges
Fast guided cost-volume filtering (Rhemann et al., CVPR 2011)

Middlebury, Errors: 2.7%

KITTI, Errors: 46.3%

Error threshold: 1 px (Middlebury) / 3 px (KITTI)
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

8

Novel Challenges
So what is the difference?

Middlebury

KITTI

Laboratory

Moving vehicle

LambertianSpecularities

Rich in texture

Sensor saturation

Medium-size label set

Large label set

Largely fronto-parallel

Strong slants

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

9Novel Challenges
So what is the difference?

Middlebury

KITTI

Laboratory

Moving vehicle

LambertianSpecularities

Rich in texture

Sensor saturation

Medium-size label set

Large label set

Largely fronto-parallel

Strong slants

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

9Novel Challenges
So what is the difference?

Middlebury

KITTI

Laboratory

Moving vehicle

LambertianSpecularities

Rich in texture

Sensor saturation

Medium-size label set

Large label set

Largely fronto-parallel

Strong slants

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

9Novel Challenges
So what is the difference?

Middlebury

KITTI

d = 16 px

d = 0 pxd = 150 px

d = 50 px
Laboratory

Moving vehicle

LambertianSpecularities

Rich in texture

Sensor saturation

Medium-size label set

Large label set

Largely fronto-parallel

Strong slants

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

9Novel Challenges
So what is the difference?

Middlebury

KITTI

Laboratory

Moving vehicle

LambertianSpecularities

Rich in texture

Sensor saturation

Medium-size label set

Large label set

Largely fronto-parallel

Strong slants

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

9

Stereo Evaluation
200 training images / 200 test images
sub.
sub.

sub.

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

10

Stereo Evaluation
200 training images / 200 test images
sub.
sub.

sub.

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

10

Stereo Evaluation
Particle Convex Belief Propagation (PCBP): Best Results

Errors: 0.5%

Errors: 0.5%

Natural scenes, lots of texture, no objects
A couple of wrong pixels at poles

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

10

Stereo Evaluation
Particle Convex Belief Propagation (PCBP): Worst Results

Errors: 19.5%

Errors: 21.1%

Inner city scenes, lots of objects
Textureless surfaces, sensor saturation, reflections

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

10Optical Flow Evaluation
200 training images / 200 test images
sub.

sub.

sub.

sub.

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

11Optical Flow Evaluation
200 training images / 200 test images
sub.

sub.

sub.

sub.

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

11Optical Flow Evaluation
Second Order Total Generalized Variation: Best Results

Errors: 0.5%

Errors: 0.5%

City scenes with slow motion (intersections)
Small flow vectors (< 30 px)

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

11Optical Flow Evaluation
Second Order Total Generalized Variation: Worst Results

Errors: 56.5%

Errors: 58.8%

Difficult lighting conditions, highway driving
Large flow vectors (> 150 px)

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

11SLAM Evaluation
22 sequences – 40 kilometers

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

12SLAM Evaluation
11 training sequences / 11 test sequences
sub.
sub.
sub.
sub.
sub.

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

12

Conclusion
Where are we now?
Realistic dataset with 3D ground truth
StereoOptical flow
SLAM
Object detection / orientation estimation

Complement existing benchmarks / reduce overfitting
Submit your results: www.cvlibs.net

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

13

Conclusion
Where are we now?
Realistic dataset with 3D ground truth
StereoOptical flow
SLAM
Object detection / orientation estimation

Complement existing benchmarks / reduce overfitting
Submit your results: www.cvlibs.net

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

13

Conclusion
Where are we now?
Realistic dataset with 3D ground truth
StereoOptical flow
SLAM
Object detection / orientation estimation

Complement existing benchmarks / reduce overfitting
Submit your results: www.cvlibs.net

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

13

Conclusion
Where are we now?
Realistic dataset with 3D ground truth
StereoOptical flow
SLAM
Object detection / orientation estimation

Complement existing benchmarks / reduce overfitting
Submit your results: www.cvlibs.net

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

13

Conclusion
Where are we now?
Realistic dataset with 3D ground truth
StereoOptical flow
SLAM
Object detection / orientation estimation

Complement existing benchmarks / reduce overfitting
Submit your results: www.cvlibs.net

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

13

Conclusion
But KITTI is much more ...
3D object tracking
Loop closure (SLAM)
Structure-from-Motion
Semantic segmentation (class labels)
3D scene understanding (layout and objects)
Use of maps

Lenz et al., IV 2011
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

14

Conclusion
But KITTI is much more ...
3D object tracking
Loop closure (SLAM)
Structure-from-Motion
Semantic segmentation (class labels)
3D scene understanding (layout and objects)
Use of maps

Williams et al., RAS 2009
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

14

Conclusion
But KITTI is much more ...
3D object tracking
Loop closure (SLAM)
Structure-from-Motion
Semantic segmentation (class labels)
3D scene understanding (layout and objects)
Use of maps

Agarwal et al., ICCV 2009
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

14

Conclusion
But KITTI is much more ...
3D object tracking
Loop closure (SLAM)
Structure-from-Motion
Semantic segmentation (class labels)
3D scene understanding (layout and objects)
Use of maps

Wojek et al., ECCV 2008
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

14

Conclusion
But KITTI is much more ...
3D object tracking
Loop closure (SLAM)
Structure-from-Motion
Semantic segmentation (class labels)
3D scene understanding (layout and objects)
Use of maps

Geiger et al., CVPR 2011 and NIPS 2011
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

14

Conclusion
But KITTI is much more ...
3D object tracking
Loop closure (SLAM)
Structure-from-Motion
Semantic segmentation (class labels)
3D scene understanding (layout and objects)
Use of maps

OpenStreetMap – The free Wiki World Map
A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

14Related Datasets and Benchmarks

Stereo / Optical Flow
EISATS
Middlebury
Make3D Stereo
LadickyKITTISLAM
TUM RGB-D
New College
Malaga 2009
Ford Campus
KITTI

settingsynthetic
laboratory
real
real
real
settingindooroutdooroutdooroutdooroutdoor#images
500
40
260
70
400
length
0.4 km
2.2 km
6.4 km
5.1 km39.2 km

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

ground truth
dense
dense
0.5 %
manual50 %metricX

X

15

Related Datasets and Benchmarks

Object Detection
Caltech 101
MIT StreetScenes
LabelMeETHZ Pedestrian
PASCAL 2011
DaimlerCaltech Pedestrian
COIL-100
EPFL Multi-View
Caltech 3D
KITTI

#cat.
101
9
4000
1
20
1
1
100
20
100
3

#labels/cat.
40-800
3k
60
12k
1k
56k
350k
72
90
144
1k - 40k

occlusion

3D

orientation

X
X
X
X

discrete
discrete
discrete
continuous

X
X
X

X

A. Geiger: The KITTI Vision Benchmark Suite – www.cvlibs.net

15