Object Scene Flow

Andreas Geiger
MPI Tübingen
Moritz Menze
University of HannoverWhat is Scene Flow?

2

“Scene flow is a dense three-dimensional vector field defined for eachpoint on every surface in the scene”[Vedula et al. 2005]

3

Scene Flow4

Application: Autonomous Driving[Wedel et al. 2010]5

Related Work: Methods
Variational Methods S. Vedula, P. Rander, R. Collins, and T. Kanade.
Three-dimensional scene flow. PAMI, 2005.
Segment Based (Local Rigidity)
 C. Vogel, K. Schindler, and S. Roth.
Piecewise rigid scene flow. ICCV, 2013.

6

“Most of the structures in the visual world
are rigid or at least nearly so.” [David Marr]

7

This Work

Observation:
Many scenes decompose into small number of rigid objects

Objects can be described by their shape and rigid motion

Dramatic reduction in the number of parameters:
4 × NSuperpixels + 6 × NObjects

8<<9 × NSuperpixels

Object Scene Flow9

Object Scene Flow10

Object Scene Flow
Let S and O denote the set of superpixels and objects
 Superpixel i ∈ S: s = (n , k )Tii i

Normal ni (nTi x = 1)
 Object index kiObject i ∈ O: oi ∈ SE (3)

Rigid motion parameters

Discrete-Continuous CRF Energy:XX
E (s, o) =ϕi (si , o) +ψij (si , sj )| {z }| {z }
i∈S




11

datasuperpixel variables s = {si |i ∈ S}
object variables o = {oi |i ∈ O}

Minimize E (s, o) using MP-PBP

i∼jsmoothness

Scene Flow Dataset

12

Related Work: Datasets with Ground Truth
Sphere Sequence [Huguet et al., ICCV 2007]13

Related Work: Datasets with Ground Truth
Auckland Synthetic Street Scene [Vaudrey et al., IVCNZ 2008]

14

Related Work: Datasets with Ground Truth
Middlebury Stereo Dataset [Scharstein et al., IJCV 2002]

15

Related Work: Datasets with Ground Truth
KITTI Stereo and Optical Flow [Geiger et al., CVPR 2012]16

Scene Flow Dataset


Accumulate static background from 7 laser scans (ICP+GPS)
Fit a 3D CAD model to each moving object ...
3D Points

SGM... using non-linear least squares minimization:X
E (ξ) =Et3D + EtSGM + Et2Dt∈{1,2}17Annotations

Disparity

Scene Flow Dataset

18

Experimental Results

19

Results on Synth. Sphere [Huguet et al., ICCV 2007]

RMSE 2D Flow
RMSE Disparity
RMSE Scene Flow
20

Valgaerts Huguet Wedel Vogel Ours
0.63
0.69
0.77
0.63 0.55
3.8
3.8
10.9
2.84 2.58
1.76
2.51
2.55
1.73 0.75

Results on Proposed Scene Flow Dataset

Huguet et al.
GCSF (Cech et al.)
SGM + LDOF (Brox et al.)
SGM + Classic+NL (Sun et al.)
SGM + Sphere Flow (Hornacek)
PRSF (Vogel et al.)Object Scene Flow

21bg67.9453.20
44.38
38.65
23.51
14.07
7.44

fg
bg+fg
69.63 68.25
66.39 55.57
54.77 46.25
60.72 42.62
50.48 28.36
48.49 20.26
45.30 14.25

Percentage of bad pixels



D1/D2: disparity error, Fl: flow error, SF: scene flow error

bg: background, fg: foreground, bg+fg: all image regions

Qualitative Results

22



GCSF [Cech et al., CVPR 2011]

Variational Scene Flow [Huguet et al., ICCV 2007]



SGM + LDOF [Brox et al., PAMI 2011]

SphereFlow [Hornacek et al., CVPR 2014]

PRSF [Vogel et al., ICCV 2013]



Object Scene FlowGCSF [Cech et al., CVPR 2011]23

Variational Scene Flow [Huguet et al., ICCV 2007]

23

SGM + LDOF [Brox et al., PAMI 2011]23

SphereFlow [Hornacek et al., CVPR 2014]

23

PRSF [Vogel et al., ICCV 2013]23

Object Scene Flow[23

GCSF [Cech et al., CVPR 2011]24

Variational Scene Flow [Huguet et al., ICCV 2007]

24

SGM + LDOF [Brox et al., PAMI 2011]24

SphereFlow [Hornacek et al., CVPR 2014]24

PRSF [Vogel et al., ICCV 2013]24

Object Scene Flow[24GCSF [Cech et al., CVPR 2011]25

Variational Scene Flow [Huguet et al., ICCV 2007]

25

SGM + LDOF [Brox et al., PAMI 2011]25

SphereFlow [Hornacek et al., CVPR 2014]

25

PRSF [Vogel et al., ICCV 2013]25

Object Scene Flow[

25

GCSF [Cech et al., CVPR 2011]26

Variational Scene Flow [Huguet et al., ICCV 2007]

26

SGM + LDOF [Brox et al., PAMI 2011]26

SphereFlow [Hornacek et al., CVPR 2014]

26

PRSF [Vogel et al., ICCV 2013]26

Object Scene Flow[26LIBVISO2 [Geiger et al.]2728