Resolving Stereo Ambiguities using Object Knowledge

Andreas Geiger
Fatma Güney
MPI for Intelligent SystemsTübingen

Stereo Matching

Task:

2



Given a rectified stereo image pairEstimate the disparity (=displacement) at each pixel

Stereo Matching

Task:

2



Given a rectified stereo image pairEstimate the disparity (=displacement) at each pixel

Stereo Matching

Energy Minimization Problem:
E =Xi

3

Edata (di ) +X

Esmooth (di , dj )i∼jdi : disparity of pixel i or plane parameters of superpixel i

i ∼ j: neighborsStereo Matching

Question:Is the stereo problem solved?

How can we address textureless or specular surfaces?

Key idea:
4Introduce top-down object knowledge into the process!

Stereo Matching

Question:Is the stereo problem solved?

How can we address textureless or specular surfaces?

Key idea:
4Introduce top-down object knowledge into the process!

Stereo Matching

Question:Is the stereo problem solved?

How can we address textureless or specular surfaces?

Key idea:
4Introduce top-down object knowledge into the process!

Stereo Matching

Question:Is the stereo problem solved?

How can we address textureless or specular surfaces?

Key idea:
4

Introduce top-down object knowledge into the process!

High-order Models in Computer Vision

5

Image Denoising

Image Segmentation[Roth & Black, 2009][Kohli, Ladicky & Torr, 2009]

Overview

6Overview

7

Initial Disparity Map8



CNN-based stereo matching cost [Zbontar et al., 2015]

Semi-global matching [Hirschmüller, 2008]

Left-right consistency checkSuperpixels and Semantic Segmentation

9StereoSLIC superpixels [Yamaguchi et al., 2013]



Associative Hierarchical Random Fields forvehicle vs. background segmentation [Ladicky et al., 2013]

Overview

10

Rapid Inverse Graphics

11

Goal: Generate plausible 3D geometries for a given semantic class

MCMC simulation based on 3D CAD models (∼8200 fps)

Semi-Convex Hull12

Displets

13Overview

14

Inference
Goal: Minimize Gibbs energy ...
XX
X
XX
D
ϕD
E=
ϕSi (ni ) +ψijS (ni , nj ) +ψki(dk , ni )k (dk ) +i∈S|i∼j{zData}|k∈D i∈Skk∈D{zSmoothness}|{zDisplets}|{zConsistency}... with respect to superpixel normals ni ∈ R3 and displets dk ∈ {0, 1}
Notation:

S: set of all superpixels

D: set of all displets

Sk : set of superpixels covered by displet kInference:

15

Max-Product Particle Belief Propagation (MP-PBP)

Inference
Goal: Minimize Gibbs energy ...
XX
X
XX
D
ϕD
E=ϕSi (ni ) +ψijS (ni , nj ) +ψki(dk , ni )k (dk ) +i∈S|i∼j{zData}|k∈D i∈Skk∈D{zSmoothness}|{zDisplets}|{zConsistency}... with respect to superpixel normals ni ∈ R3 and displets dk ∈ {0, 1}
Data Term:16

Penalizes deviation from initial disparity map (truncated `1 )

Measures photo-consistency between left and right image

Inference
Goal: Minimize Gibbs energy ...
XX
X
XX
D
ϕD
E=
ϕSi (ni ) +ψijS (ni , nj ) +ψki(dk , ni )k (dk ) +i∈S|i∼j{zData} |k∈D i∈Skk∈D{zSmoothness}|{zDisplets}|{zConsistency}... with respect to superpixel normals ni ∈ R3 and displets dk ∈ {0, 1}
Smoothness Term:17

Penalizes depth discontinuities at superpixel boundaries



Encourages similar orientations of neighboring superpixel planes

Inference
Goal: Minimize Gibbs energy ...
XX
XXX
D
ϕD
E=ϕSi (ni ) +ψijS (ni , nj ) +ψki(dk , ni )k (dk ) +i∈S|i∼j{zData}|k∈D i∈Skk∈D{zSmoothness} |{zDisplets}|{zConsistency}... with respect to superpixel normals ni ∈ R3 and displets dk ∈ {0, 1}
Displet Unary Term:Encourages image regions with semantic class label ck to be
explained by a displet of the corresponding class:ϕDk (dk ) = −θ3 [dk = 1] · (|[S = ck ] ∩ Mk | + κk )

18

Mk : foreground mask of displet k

S: semantic segmentation

Inference
Goal: Minimize Gibbs energy ...
XX
X
XXD
ϕD
E=
ϕSi (ni ) +ψijS (ni , nj ) +ψki(dk , ni )k (dk ) +i∈S|i∼j{zData}|k∈D i∈Skk∈D{zSmoothness}|{zDisplets} |{zConsistency}... with respect to superpixel normals ni ∈ R3 and displets dk ∈ {0, 1}
Displet Consistency Term:Encourages superpixels to agree with the geometry of the displets:Dψki(dk , ni ) = λki [dk = 1] · (1 − δ(ni , n̂k,zi ))19

λki : weight depending on distance to boundary:Results on KITTI Validation Set (Ablation)

Input Ω̂ (Interpolated)
Unary Only
Unary + Pair (Boundary)
Unary + Pair (Normal)
Unary + Pair
Unary + Pair + Occ
Unary + Pair + Disp
Unary + Pair + Occ + Disp

20

Reflective Regions
19.84 %
17.72 %
15.96 %
17.06 %
14.78 %
15.32 %
7.08 %
7.16 %All Regions
3.35 %
3.31 %
3.21 %
3.28 %
3.12 %
3.04 %
2.87 %
2.78 %

Metric: Percentage of erroneous pixels (> 3px disparity error)

Results on KITTI Test Set (Reflective Regions)

21Rank
1
2
3
4
5
6
7
8
9
10
1112..
.

MethodOur Method
VC-SF *
PCBP-SSSPS-StFl *CoP
SPS-StDDS-SS
PCBP
PR-Sf+E *StereoSLIC
MC-CNN
PR-Sceneflow *..
.

Out-Noc
8.40 %
11.58 %
14.26 %
14.74 %
15.30 %
16.05 %
16.23 %
16.28 %
17.85 %
18.22 %
18.45 %
19.22 %..
.

Out-All
9.89 %
12.29 %
18.33 %
18.00 %
19.15 %
19.34 %
19.39 %
20.22 %
20.82 %
21.60 %
21.96 %
22.07 %..
.

Avg-Noc
1.9 px
2.7 px
2.4 px
2.9 px
2.7 px
3.1 px
2.5 px
2.8 px
3.3 px
2.8 px
3.5 px
3.3 px..
.

Avg-All
2.3 px
2.8 px
3.9 px
3.6 px
4.1 px
3.6 px
3.0 px
4.4 px
4.0 px
3.6 px
4.3 px
4.0 px

62

ALE-Stereo

83.80 %

84.37 %

24.6 px25.4 px

Results on KITTI Test Set (All Regions)

22Rank1
234
56
789101112..
.

MethodOur Method
MC-CNNSPS-StFl *VC-SF *SPS-St
PCBP-SS
CoP
DDS-SS
StereoSLIC
PR-Sf+E *PCBP
PR-Sceneflow *..
.

Out-Noc
2.47 %
2.61 %
2.83 %
3.05 %
3.39 %
3.40 %
3.78 %
3.83 %
3.92 %
4.02 %
4.04 %
4.36 %..
.

Out-All
3.27 %
3.84 %
3.64 %
3.31 %
4.41 %
4.72 %
4.63 %
4.59 %
5.11 %
4.87 %
5.37 %
5.22 %..
.

Avg-Noc
0.7 px0.8 px0.8 px0.8 px0.9 px0.8 px0.9 px0.9 px0.9 px0.9 px0.9 px0.9 px..
.

Avg-All
0.9 px
1.0 px0.9 px0.8 px
1.0 px
1.0 px
1.1 px
1.0 px
1.0 px
1.0 px
1.1 px
1.1 px..
.

62

ALE-Stereo

50.48 %

51.19 %

13.0 px13.5 px

Qualitative Results: Success Cases

23

Qualitative Results: Success Cases

24Qualitative Results: Success Cases

25Qualitative Results: Success Cases

26

Qualitative Results: Failure Cases

27

Qualitative Results: Failure Cases

28

Qualitative Results: Video

29

Thank you!

30