Object Scene Flow for Autonomous Vehicles
Moritz Menze1 , Andreas Geiger2
1 Leibniz Universität Hannover. 2 MPI Tübingen.

This paper proposes a novel model and dataset for 3D scene flow estimation with an application to autonomous driving. Taking advantage of thefact that outdoor scenes often decompose into a small number of independently moving objects, we represent each element in the scene by its rigid
motion parameters and each superpixel by a 3D plane as well as an index tothe corresponding object. This minimal representation increases robustnessand leads to a discrete-continuous CRF where the data term decomposes
into pairwise potentials between superpixels and objects. Moreover, ourmodel intrinsically segments the scene into its constituting dynamic components. We demonstrate the performance of our model on existing benchmarks as well as a novel realistic dataset with scene flow ground truth. We
obtain this dataset by annotating 400 dynamic scenes from the KITTI raw
data collection using detailed 3D CAD models for all vehicles in motion.
Our experiments also reveal novel challenges which cannot be handled byexisting methods.
Model: We focus on the classical scene flow setting where the input isgiven by two consecutive stereo image pairs of a calibrated camera and thetask is to determine the 3D location and 3D flow of each pixel in a reference frame. Following the idea of piecewise-rigid shape and motion [10],a slanted-plane model is used, i.e., we assume that the 3D structure of the Figure 1: Scene Flow Results on the proposed Scene Flow Dataset. Topscene can be approximated by a set of piecewise planar superpixels. As to-bottom: Estimated segmentation into moving objects with background in
opposed to existing works, we assume a finite number of ridigly moving transparent, optical flow results and proposed optical flow ground truth.
objects in the scene. Our goal is to recover this decomposition jointly withthe shape of the superpixels and the motion of the objects. We specify thebg
fg
bg+fg
problem in terms of a discrete-continuous CRF
Huguet [8]
67.69
64.03
67.08
GCSF [2]
52.92
59.11
53.95
E(s, o) = ∑ ϕi (si , o) + ∑ ψi j (si , s j )(1)i∈S | {z } i∼ j | {z }SGM [6] + LDOF [1]
43.99
44.77
44.12
datasmoothness
SGM [6] + Sun [9]
38.21
53.03
40.68
SGM
[6]+Sphere
Flow
[7]
23.09
37.11
25.42
where si determines the shape and associated object of superpixel i ∈ S, andPRSF
[10]
13.49
33.71
16.85
o are the rigid motion parameters of all objects in the scene.
Our full model
7.01
28.76
10.63
Dataset: Currently there exists no realistic benchmark dataset providing
dynamic objects and ground truth for the evaluation of scene flow or optical Table 1: Quantitative Results on the Proposed Scene Flow Dataset. Thisflow. Therefore, in this paper, we take advantage of the KITTI raw data [5] table shows scene flow errors in %, averaged over all 200 test images. We
to create a realistic and challenging scene flow benchmark with indepen- provide separate numbers for the background region (bg), all foreground
dently moving objects and annotated ground truth, comprising 200 training objects (fg) as well as all pixels in the image (bg+fg).
and 200 test scenes in total. The process of ground truth generation is especially challenging in the presence of individually moving objects since [1] T. Brox and J. Malik. Large displacement optical flow: Descriptor matching inthey cannot be easily recovered from laser scanner data alone due to thevariational motion estimation. PAMI, 33:500–513, March 2011.
rolling shutter of the Velodyne and the low framerate (10 fps). Our anno- [2] Jan Cech, Jordi Sanchez-Riera, and Radu P. Horaud. Scene flow estimation bygrowing correspondence seeds. In CVPR, 2011.
tation work-flow consists of two major steps: First, we recover the static[3]Andreas Geiger, Julius Ziegler, and Christoph Stiller. StereoScan: Dense 3D
background of the scene by removing all dynamic objects and compensatreconstruction in real-time. In IV, 2011.
ing for the vehicle’s egomotion. Second, we re-insert the dynamic objects
by fitting detailed CAD models to the point clouds in each frame. We make [4] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In CVPR, 2012.
our dataset and evaluation available as part of the KITTI benchmark.
[5] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision

meets robotics: The KITTI dataset. IJRR, 32(11):1231–1237, September 2013.
Results: As input to our method, we leverage sparse optical flow from
[6] Heiko Hirschmueller. Stereo processing by semiglobal matching and mutualfeature matches [3] and SGM disparity maps [6] for both rectified frames.
information. PAMI, 30(2):328–341, 2008.
We obtain superpixels using StereoSLIC [11] and initialize the rigid motion[7] Michael Hornacek, Andrew Fitzgibbon, and Carsten Rother. SphereFlow: 6parameters of all objects in the scene by greedily extracting rigid body moDoF scene flow from RGB-D pairs. In CVPR, 2014.
tion hypotheses using the 3-point RANSAC algorithm. By reasoning jointly [8] Frédéric Huguet and Frédéric Devernay. A variational method for scene flow
about the decomposition of the scene into its constituent objects as well as
estimation from stereo sequences. In ICCV, 2007.
the geometry and motion of all objects in the scene, the proposed model is [9] Deqing Sun, Stefan Roth, and Michael J. Black. A quantitative analysis of current practices in optical flow estimation and the principles behind them. IJCV,
able to produce accurate dense 3D scene flow estimates, comparing favor106(2):115–137, 2013.
ably with respect to current state-of-the-art as illustrated in Table 1. Our[10]
C. Vogel, K. Schindler, and S. Roth. Piecewise rigid scene flow. In ICCV, 2013.
method also performs on par with the leading entries on the original static[11]K.
Yamaguchi, D. McAllester, and R. Urtasun. Robust monocular epipolar flow
KITTI dataset [4] as well as on the Sphere sequence of Huguet et al. [8].
estimation. In CVPR, 2013.

This is an extended abstract. The full paper is available at the Computer Vision Foundation
webpage.