Object Scene Flow for Autonomous Vehicles
Moritz Menze , Andreas Geiger
11Abstract

Leibniz Universität Hannover, MPI for Intelligent Systems, Tübingen
2

Qualitative Results

ModelE(s, o) =Xi∈Sϕ(s,o)+ii{z}|data

The data term ϕ consists of pairwise potentials which are evaluatedfor 3 pairs of images (see figure below)j∈O[ki = j] · Di(ni, oj )where the Iverson bracket restricts ϕ to the selected object.
To compute Di(ni, oj ) we leverage:smoothness

Our smoothness potential ψij (si, sj ) decomposes as:
depthorientmotion
ψij (si, sj ) = θ3 ψij (ni, nj ) + θ4 ψij (ni, nj ) + θ5 ψij (si, sj )with weights θ controlling the individual terms:• SGM disparity maps for both rectified pairs

100orientevaluating the difference of plane normals n (ψij )• Coherence of the assigned object indices is enforced by anmotionorientation-sensitive Potts model (ψij )Dataset0.00 - 0.19

0.19 - 0.38

0.38 - 0.75

0.75 - 1.50

1.50 - 3.00

3.00 - 6.00

6.00 - 12.00 12.00 - 24.00 24.00 - 48.00

48.00 - Inf

Qualitative Results. Each subfigure shows from top-to-bottom: disparity andoptical flow ground truth in the reference view, the disparity map (D1) and opticalflow map (Fl) estimated by our algorithm, and the respective error images using
the color scheme depicted in the legend. The four scenes below the horizontal line
are failure cases.

3D Points

Results on the ”Sphere” sequence

Annotation[7] [5] [9] [8] Ours
RMSE 2D Flow 0.63 0.69 0.77 0.63 0.55RMSE Disparity
3.8 3.8 10.9 2.84 2.58
RMSE Scene Flow 1.76 2.51 2.55 1.73 0.75
InputSegmentation

Superpixels

Before Optimization

After Optimization

Ground Truth. Disparity at t0 (left) and optical flow (right).

00345010203040

Number of iterations[2] Jan Cech, Jordi Sanchez-Riera, and Radu P. Horaud. Scene flow estimation bygrowing correspondence seeds. In CVPR, 2011.
[3] Heiko Hirschmüller. Stereo processing by semiglobal matching and mutualinformation. PAMI, 30(2):328–341, 2008.
[4] Michael Hornacek, Andrew Fitzgibbon, and Carsten Rother. SphereFlow: 6
DoF scene flow from RGB-D pairs. In CVPR, 2014.
[5] Frédéric Huguet and Frédéric Devernay. A variational method for scene flow
estimation from stereo sequences. In ICCV, 2007.

Disparity 1

Error D1

SGM

scene is approximated by planar superpixels from StereoSLICs = {si|i ∈ S}
• As opposed to existing works, we assume a finite number of rigidly
moving objects in the scene o = {oi|i ∈ O}
• Superpixels are parameterized by a plane n and an object index k:Tsi = (ni, ki)
T• Objects store their rigid motion parameters oj = (Rj , tj )• The superpixels inherit motion parameters from the assigned object

20[1] T. Brox and J. Malik. Large displacement optical flow: Descriptor matching invariational motion estimation. PAMI, 33:500–513, March 2011.

• The static background of the scenes is recovered from laser scanner data by removing all dynamic

Disparity Map• Following the idea of piecewise-rigid shape and motion [8], the 3D

40

Parameter Variation.
This figure shows the sceneflow errors of our methodon the proposed datasetwith respect to the numberof object proposals and50
MP-PBP iterations.

References• We annotated 200 training and 200 test scenes based on KITTI raw dataobjects and compensating for the vehicle’s ego-motion• Dynamic objects are captured by fitting detailed CAD models to• accumulated 3D laser point clouds in each frame• manually annotated 2D control points• SGM disparity maps

60

20

Number of objects• We propose a novel realistic scene flow dataset which includes dynamic objects

Representation40

2SF bg+fg occSF fg occ
SF bg occ

80

60

1

• We use max-product particle BP to jointly infer shape and motion

parameters with TRW-S for the inner loop
• Particles are drawn from normal distributions around the current
MAP solution and from neighboring superpixels

100

SF bg+fg occ
SF fg occ
SF bg occ

80

Inference

Results and Ground Truth. Estimated segmentation into moving objects (top),
optical flow (center) and proposed optical flow ground truth (bottom).

bg+fg
67.08
53.95
44.12
40.68
25.42
16.85
10.63

• Regularization of depth is achieved by penalizing differences in• We encourage the orientation of neighboring planes to be similar by• Sparse optical flow from feature matches

fg
64.03
59.11
44.77
53.03
37.11
33.71
28.76

Quantitative Results on the Proposed Scene Flow Dataset. This tableshows scene flow errors in %, averaged over all 200 test images. We provide
separate numbers for the background region (bg), all foreground objects (fg) as well
as all pixels in the image (bg+fg). Outliers are defined as those values exceeding
ground truth by at least 3 px and 5%.

depthdisparity at shared boundary pixels (ψij )• Dense Census features

bg
67.69
52.92
43.99
38.21
23.09
13.49
7.01

Huguet [5]GCSF [2]
SGM [3] + LDOF [1]
SGM [3] + Sun [6]SGM [3] + Sphere Flow [4]PRSF [8]Our full model

Error (%)

ϕi(si, o) =i∼j

ψ
(s,s)
ijij
|{z}

Smoothness Term

Data TermXX

Quantitative Results

Error (%)

We propose a novel model and dataset for 3D scene flow estimation
with an application to autonomous driving. As outdoor scenes often
decompose into a small number of independently moving objects, werepresent each element in the scene by its rigid motion parametersand each superpixel by a 3D plane and an index to the corresponding object. This minimal representation increases robustness andis formalized in a discrete-continuous CRF. Our model intrinsicallysegments the scene into its constituting dynamic components. We
demonstrate the performance of our model on existing benchmarks
and a novel realistic dataset with scene flow ground truth. We obtain this dataset by annotating 400 dynamic scenes from the KITTI
raw data collection using 3D CAD models for all vehicles in motion.
Our experiments reveal novel challenges which cannot be handled byexisting methods.

2Disparity 2

Optical FlowError D2

Error Fl

Illustration of our results for thesynthetic “Sphere” sequence [5].
Top-left to bottom-right: The left input
image of the first frame, our first disparity/error map, the obtained segmentation into different rigid body motions,the second disparity/error map, the superpixels we use, the recovered opticalflow/error map.

[6] Deqing Sun, Stefan Roth, and Michael J. Black. A quantitative analysis ofcurrent practices in optical flow estimation and the principles behind them.
IJCV, 106(2):115–137, 2013.
[7] Levi Valgaerts, Andres Bruhn, Henning Zimmer, Joachim Weickert, Carsten
Stoll, and Christian Theobalt. Joint estimation of motion, structure andgeometry from stereo sequences. In ECCV, 2010.
[8] C. Vogel, K. Schindler, and S. Roth. Piecewise rigid scene flow. In ICCV, 2013.
[9] Andreas Wedel, Clemens Rabe, Tobi Vaudrey, Thomas Brox, Uwe Franke, andDaniel Cremers. Efficient dense scene flow from sparse or dense stereo data. In
ECCV, 2008.

Contact Information
Web http://www.cvlibs.net/projects/objectsceneflow
Email menze@ipi.uni-hannover.de, andreas.geiger@tue.mpg.de