Vision for Vehicles

1 / 31A View of Computer Vision
David McAllester
Toyota Technologial Institute at Chicago (TTI-Chicago)

Joint work with
Koichiro Yamaguchi (TCRDL)
Michael James (TRINA)
Harry Yang (TTIC)
Raquel Urtasun (University of Toronto)

TTI-C

McAllester

Vision for Vehicles

2 / 31Do We Need Vision for Robotics?

Toyota Autonomous Car

Velodyne (Lidar)
TTI-C

McAllester

Vision for Vehicles

3 / 31The Value of Vision• Vision systems can provide depth information that very dense inspace and time.

• Vision systems can provide accurate motion information.

• Vision presents an important scientific challenge.

TTI-C

McAllester

Vision for Vehicles

4 / 31

Geometric vs. Semantic Vision
Rendering — converting scene geometry to images.
Inverse Rendering (low level vision) — converting images to scene
geometry (e.g., stereo, structure from motion, shape from shading).
Semantic Vision (high level vision) — vision involving conceptual
labels such as “car”, “bicycle”or “building” (e.g., image classification,object detection, semantic segmentation).
Both are important and should be done jointly.

TTI-C

McAllester

Vision for Vehicles

5 / 31

Why is Low Level Vision Important?

Autonomous Learning — learning from simply being in the world.
Learning without labeled data.

Human babies learn autonomously.

Stereo and motion seems central to autonomous learning.

TTI-C

McAllester

Vision for Vehicles

6 / 31

StereoCorrespondences between pixels from two cameras determine depth.
Unfortunately, the correspondences are surprisingly difficult forcomputers to see.
TTI-C

McAllester

Vision for Vehicles

7 / 31

Motion Stereo
Given two images of a Static Scene, with known camera positions,the correspondence of a given pixel lies along a known epi-polar line.
The position on the epi-polar line corresponds to depth in the image.

TTI-C

McAllester

Vision for Vehicles

8 / 31The Middlebury Sudio-Image Stereo Benchmark (2001)

TTI-C

Tsukuba

Venus

Cones

Teddy
McAllester

Vision for Vehicles

9 / 31The KITTI Driving-Scene Benchmark (2012)
Karlsruhe Institute of Technology (KIT)
Toyota Technological Institute at Chicago (TTI)
Andreas Geiger, Philip Lenz, Christoph Stiller, Raquel Urtasun

TTI-C

McAllester

Vision for Vehicles

10 / 31KITTI Stereo Leader Board, November 11, 2014
TeamZbontar and LeCun.
NYU
Yamaguchi et al.
TCRDL, TTIC
Vogel et al.
ETH Zurich
TU Darmstadt
Anonymous(TTIC)
Yamaguchi et al.
Yamaguchi et al.
..
.
64 entries total

Data*
S

Date
August 2014

Error Rate
2.61%

Time
100s

Proc.
GPU

S, MS

Feb. 2014

2.83%

35s

1 coreS, MS

March 2014

3.05%

300s

1 coreS

November 2014

3.30%

6s

6 cores

SS..
.

Sept 2014
March 2013
..
.

3.39%
3.40%..
.

2s
300s..
.

1 core4 cores..
.

* S means stereo pair, MS means motion stereo pair.
TTI-C

McAllester

Vision for Vehicles

11 / 31KITTI Motion Stereo Leader Board, November 11, 2014

Team

Data*

DateError RateTimeProc.

Vogel at al.
ETH Zurich
TU Darmstadt
Yamaguchi et al.
TCRDL, TTIC
Yamaguchi et al.
Vogel at al.
..
.
54 entries total

S, MS

March 2014

2.72%

300s

1 coreS, MSFeb. 2014

2.82%

35s

1 coreMS
S, MS..
.

Sept 2014
April 2013
..
.

3.39%
3.56%
..
.

11s
200s..
.

1 core4 cores..
.

* S means stereo pair, MS means motion stereo pair.

TTI-C

McAllester

Vision for Vehicles

12 / 31TCRDL &TTIC System: Slanted Plane Model
Birchfield and Tomasi, ICCV, 1999
Create superpixels on the left image.

Shown are SLIC superpixels, Achanta et al., 2010
Assume that in each superpixel the scene is a plane.
The problem is then to estimate the plane for each superpixel using thestereo data.
Slanted Plane models have dominated the Middelbury benchmark.
TTI-C

McAllester

Vision for Vehicles

13 / 31TCRDL & TTIC System: Overview• Slanted Plane Model.
• Discrete edge labels — each boundary between two superpixels islabeled as occlusion, hinge or coplanar.
• Segmentation, plane parameters and edge labels are inferred byjoint energy minization. The energy involves the following terms.
• The match energy for a setting of plane parameters is computedfrom SGM semi-sparse disparity values computed by the SGM
stereo vision algorithm (Hirschmuller, 2005).
• The smoothness energy between adjacent planes depends on theedge label between them.
• SLIC-like segmentation energy.
TTI-CMcAllester

Vision for Vehicles

14 / 31

Stereo Pairs and Motion Stereo Pairs
The system can be either with a single stereo pair (S), a single motion
stereo pair (MS), or both (S, MS).
In all three settings we use a variant of semi-global matching (SGM) toproduce an initial semi-dense depth estimate. Joint SGM for both Sand MS is nontrivial.
The semi-dense SGM estimate is then smoothed with a slanted-planemodel.
The smoothing algorithm is the same in all three cases.

TTI-CMcAllester

Vision for Vehicles

15 / 31

Slanted-Plane Smoothing by Alternating Optimization

We alternate between optimizing:• The segmentation.

• The segment plane parameters.

• The boundary labels.

TTI-C

McAllester

Vision for Vehicles

16 / 31

Depth-Aware SLIC Segmentation
We want to minize Eseg =Pp∈pixels Eseg (p, sp ).

Eseg (p, sp ) involves the position, color and depth of p and the mean
position, mean color and plane parameters of sp .
We use alternating minimization:
Initialize a set of segments (typically a grid).
repeatCompute segment parameters so as to minimize Eseg holding thesegmentation fixed.
Compute the segmentation sp so as to minimize Eseg holding theplane parameters fixed.
until Convergence
TTI-C

McAllester

Vision for Vehicles

17 / 31

Problems with Alternating-Minimization Segmentation

Individual segments become disconnected and must be converted toconnected segments in a post-processing phase.

The final segmentation is then not a local minimizer of the energy.

TTI-C

McAllester

Vision for Vehicles

18 / 31

Topology-Preserving Segmentation (ECCV 2014)
Initialize the stack to contain all boundary pixels
while the stack is not empty do
Take pixel p off the stack.
if valid connectivity(p) = 0 thencontinue
end if
sp = argminŝp ∈{Sq , q∈N4 (p)} Eseg (p, ŝp )if sp was changed thenUpdate Segment Parameters.
Push pixels in N4 (p) onto the stack.
end ifend while
TTI-C

McAllester

Vision for Vehicles

19 / 31KITTI Stereo Leader Board, November 11, 2014
Team
Zbontar and LeCun.
NYU
Yamaguchi et al.
TCRDL, TTIC
Vogel et al.
ETH Zurich
TU Darmstadt
Anonymous(TTIC)
Yamaguchi et al.
Yamaguchi et al.
..
.
64 entries total

Data*
S

Date
August 2014

Error Rate
2.61%

Time
100s

Proc.
GPUS, MSFeb. 2014

2.83%

35s

1 coreS, MSMarch 2014

3.05%

300s

1 coreSNovember 2014

3.30%

6s

6 coresSS..
.

Sept 2014
March 2013..
.

3.39%
3.40%..
.

2s300s..
.

1 core4 cores..
.

* S means stereo pair, MS means motion stereo pair.
TTI-C

McAllester

Vision for Vehicles

20 / 31

Moving Objects — Scene Flow

Two stereo pairs taken at different times should determine both thedepth and the 3D-motion of each point yielding a scene flow image(depth plus motion at each pixel).

TTI-C

McAllester

Vision for Vehicles

21 / 31

Problems with KITTIThe KITTI benchmark does not provide gound truth data for sceneswith moving objects.

This is becasue ground-truth depth must be constructed from Lidar
points taken at a time different from the time the images were taken
and this can only be done reliably for static scenes.

TTI-C

McAllester

Vision for Vehicles

22 / 31

Problems Even for Static Scenes
Even for static scenes the KITTI ground truth is not perfect:

The KITTI ground truth consists of (quite accurate) inferred labels.
Inferring ground truth for scene flow is too difficult — scene flow istypically evaluated on synthetic data (e.g. MPI Sintel, 2012).
TTI-C

McAllester

Vision for Vehicles

23 / 31

Avoiding Inferred Labels: Direct Lidar Evaluation
A system is scored by its ability to predict the flight time of a lidar
probe.

For scenfow evaluation the lidar probe is at a time different from theimage data.
TTI-C

McAllester

Vision for Vehicles

24 / 31

Evaluating Existing Software Packages
We have implemented a “predictor” for converting disparity images tolidar predictions.

StereoSPS-StrSGM
opencv-BM
opencv-SGBM
ELAS

TTI-C

Non-Occ
4.55%
5.84%
6.61%
9.54%
10.80%

All
5.79%
7.33%
7.88%
10.32%
11.09%KITTI-All
4.41%
6.60%
13.76%
9.13%
9.96%

McAllester

Vision for Vehicles

25 / 31

Other Evaluations are Possible
We will eventually implement predictors for converting the output ofeach of the following to lidar predictions.
• Image Segmentation.
• Optical Flow.
• Scene Flow.
• Monocular Depth Estimation.

Teams will be free to write their own predictors or select from a list ofdefault predictors provided by the evaluation site.
TTI-C

McAllester

Vision for Vehicles

26 / 31

Lidar Evaluation and Autonomous Learning
People can confirm vision by touching.
An autonomous car can confirm vision with lidar.
Direct lidar evaluation should allow autonomous learning of visionsystems.
Semantic vision should improve the accuracy of inverse rendering.
Direct lidar evaluation should therefore also support autonomous
learning of semantic vision.

TTI-C

McAllester

Vision for Vehicles

27 / 31

Deep Neural Networks (DNNs)

Over the last three years there has been enormous progress inapplications of DNN research.

The best systems (by a large margin) in speech recognition and image
classification are now based on DNNs.

There is now active research on the use of DNNs in viturally all
applications of machine learning.

TTI-C

McAllester

Vision for Vehicles

28 / 31

DNNs in ImageNet Image Classification

Year2012
2012
2013
2014

TTI-C

Team
U of Tokyo
U of Toronto (DNN)
NYU (DNN)
Google (DNN)

Error Rate
26.1%
15.3%
11.2%
6.7%

McAllester

Vision for Vehicles

29 / 31

PASCAL Object Detection: A Moore’s Law of AI?

2007
2008
2009
2010
2011
2012
2013 (DNN)
2014 (DNN)

TTI-C

bicycle
36.9
42.0
46.8
54.3
(58.1)(54.5)(56.3)

bus
23.2
23.2
43.8
54.2
(57.6)(57.1)(51.4)

car
34.6
32.0
37.2
49.1
(54.4)(49.3)(48.7)

motorbike
27.6
38.6
42.0
51.6
(58.3)(59.4)(59.8)

person
21.3
42.0
41.5
47.5
(51.6)(46.1)(44.4)

20 class average
17.1
22.9
27.9
33.7 (36.8)(40.9)(41.1)(43.2)(63.8)

McAllester

Vision for Vehicles

30 / 31KITTI Stereo Leader Board, November 11, 2014
Team
Zbontar and LeCun.
NYU (DNN)
Yamaguchi et al.
TCRDL, TTIC
Vogel et al.
ETH Zurich
TU Darmstadt
Anonymous(TTIC)
Yamaguchi et al.
Yamaguchi et al.
..
.
64 entries total

Data*
SDate
August 2014

Error Rate
2.61%

Time100sProc.
GPUS, MSFeb. 2014

2.83%

35s

1 coreS, MS

March 2014

3.05%

300s

1 coreS

November 2014

3.30%

6s

6 cores

SS..
.

Sept 2014
March 2013
..
.

3.39%
3.40%
..
.

2s
300s..
.

1 core4 cores..
.

* S means stereo pair, MS means motion stereo pair.
TTI-C

McAllester

Vision for Vehicles

31 / 31Summary• It seems likely that future inverse rendering systems will base boththe match energies and the image smoothing on DNN methods.
• It also seems likey that direct lidar evaluation will allow

autonomous learning of inverse rendering DNN systems.
• Autonomous learning of inverse rendering may lead toautonomous learning of semantic vision.
• I believe that there is still room for new theory in the area of

autonomous learning.

TTI-C

McAllester