Displets: Resolving Stereo Ambiguities using Object Knowledge
Fatma Güney, Andreas Geiger
MPI for Intelligent Systems, Tübingen

Experimental Results on KITTI

Displets

Inference

Displets: A representative set of disparity maps for a specific semantic class (e.g., car) conditioned on the image.

Minimizing E(n, d) is a non-convex, mixed continuousdiscrete optimization problem. We use max-product particle BP with TRW-S for the inner iterations.

0Illustration of Displet Influence λki

At each outer iteration, particles are sampled• around the previous MAP estimate• using neighbouring superpixels
160

Displet Illustration. Top: wireframe model, middle:
rendered disparity map, bottom: fitted superpixel planes.

Abstract• Current stereo methods often fail at textureless,

We sample 3D CAD model configurations ξkand associate each displet k ∈ D with• its class label ck

We introduce object knowledge for well-constrained object categories into a slanted-plane MRF. We leverage
inverse graphics to sample set of plausible object disparity maps given an initial semi-dense disparity estimate
and a rough semantic segmentation of the image. We
encourage the presence of these 2.5D shape samples (or“displets”) depending on how much their geometry andsemantic class agrees with the image observations.

Representation

Observation Model• a fitness score κk• a set of superpixels Sk

Stereo Matching using Displets
X XXXDSDS
ψki(dk , ni)E(n, d) =ϕk (dk ) +ϕi (ni) + ψij (ni, nj ) +i∼ji∈Si∈Sk∈Dk∈Dk {z{z}
|{z}
||}
{z}
|Data
Displets
Consistency
SmoothnessXData term

Displet Unary

We penalize deviations from an initial sparse disparitymap Ω̂ obtained using the method of [1]:We encourage image regions with semantic class label ck
to be explained by a displet of the corresponding class:S
ϕi (ni) =p∈Ri∩ Ω̂+• ω(ni, p) denotes the disparity of plane ni at pixel p.

• region Ri in the image

Local smoothness

Each displet k ∈ D is associated with:• random variable dk ∈ {0, 1} (presence/absence)

Our goal is to infer:• superpixel planes n = {ni|i ∈ S}
• displets d = {dk |k ∈ D}

Dϕk (dk ) = −θ3 [dk = 1] · (|[S = ck ] ∩ Mk | + κk )

ρτ1(ω(ni, p) − ω̂(p))

X• ρτ (·) is a robust l1 penalty ρτ (x) = min(x, τ ).

• random variable ni ∈ R describing a plane in 3D

We sample pose parameters from a model which• encourages displets to explain pixels in object regions• avoids occlusion of other objects

Importance of Different Terms.

We run one Markov chain for each combination of 3D
CAD models and object proposals using Metropoliswithin-Gibbs sampling with randomly chosen blocks.
We select the 8 most dominant modes after burn-in foreach model and combine all results to yield the final setof displets.

2020

CNN−Out−All
CNN−Out−Noc
SGM−Out−All
SGM−Out−Noc

15

100.0

We decompose the image into planar superpixels S.
Each superpixel i ∈ S is associated with:
3

Examples of Car Displets. Our displets cover the most likely 3D car
shapes given the initial disparity map and a semantic segmentation.

• the corresponding planes n̂k,1, . . . , n̂k,|Sk |reflecting or semi-transparent surfaces such as cars.
• Yet, as humans we are able to effortlessly extract
information about the geometry of objects even froma single image.
• Little is known about the importance of recognition
for stereo matching.

Reflective All
Input Ω̂ (Interpolated)
19.84 % 3.35 %Unary Only
17.72 % 3.31 %Unary + Pair (Boundary)
15.96 % 3.21 %17.06 % 3.28 %Unary + Pair (Normal)
14.78 % 3.12 %Unary + Pair
Unary + Pair + Occ
15.32 % 3.04 %7.08 % 2.87 %Unary + Pair + Disp
Unary + Pair + Occ + Disp 7.16 % 2.78 %Error (%)

Motivation. Using object knowledge, we encourage disparities toagree with plausible surfaces while simultaneously recovering 3D
geometry of the objects (bottom). This improves results (center)compared to current state-of-the-art stereo methods (top) [1].

We leverage the initial semi-dense disparity map, a semantic segmentation of the image as well as 3D CAD
models to sub-sample the space of displets using MCMC.

Ablation Study• S denotes the semantic segmentation.

Consistency between Displets and Superpixels

For adjacent superpixels, we penalize discontinuities atthe boundary Bij and encourage similar orientations:Sψij (ni, nj ) = θ1

For sampling displets, we leverage 3D CAD models
from Google Warehouse. To speed-up the rendering
process, we reduce details while preserving the overall
shape by fitting a “semi-convex hull” to the point
clouds of the original models:• Mk is the foreground mask of displet k.

Xp∈Bijθ2 ρ τ 3ρτ2 (ω(ni, p) − ω(nj , p)) +T
1 − |ni nj |/(knikknj k)

We ensure consistency by encouraging superpixels toagree with the geometry of the active displets:(b) Semi-Convex Hullthe boundary (see experiments).

0.6

0.8

1.0

15100123456(c) QSlim(d) MATLAB

Mesh Simplification. Comparison of our semi-convex hullalgorithm to QSlim and MATLAB’s reducepatch function.

78

Number of Models

Number of Proposals and Models. Influence of limiting thenumber of object proposals (left) and the variety of CAD models
used for generating the displets (right).

References

Quantitative Results
Rank Method Out-Noc Out-All Avg-Noc Avg-All
1 Our Method 8.40 % 9.89 % 1.9 px 2.3 px
2VC-SF* [2] 11.58 % 12.29 % 2.7 px 2.8 px
3 PCBP-SS [3] 14.26 % 18.33 % 2.4 px 3.9 px
4 SPS-StFl* [4] 14.74 % 18.00 % 2.9 px 3.6 px
..
..
..
..
..
11 MC-CNN [1] 18.45 % 21.96 % 3.5 px 4.3 px
Reflective Regions.

Dψki(dk , ni) = λki [dk = 1] · (1 − δ(ni, n̂k,zi))• λki denotes a weight which depends on the distance to0.4

CNN−Out−All
CNN−Out−Noc
SGM−Out−All
SGM−Out−Noc

Acceptance Probability

3D CAD Models(a) 3D Warehouse Model

0.2

Error (%)

Sampling Displet Proposals

Displet Influence. Less transparency indicates a higher penalty,e.g., we allow more deviation at the displet boundaries.

Rank Method Out-Noc Out-All Avg-Noc Avg-All
1 Our Method 2.47 % 3.27 % 0.7 px 0.9 px
2 MC-CNN [1] 2.61 % 3.84 % 0.8 px 1.0 px
3 SPS-StFl* [4] 2.83 % 3.64 % 0.8 px 0.9 px
4VC-SF* [2] 3.05 % 3.31 % 0.8 px 0.8 px
5SPS-St [4] 3.39 % 4.41 % 0.9 px 1.0 px
6 PCBP-SS [3] 3.40 % 4.72 % 0.8 px 1.0 px
All Regions.

[1] Jure Zbontar and Yann LeCun. Computing the stereo matching
cost with a convolutional neural network. In CVPR, 2015.
[2] Christoph Vogel, Stefan Roth, and Konrad Schindler.
View-consistent 3D scene flow estimation over multiple frames.
In ECCV, 2014.
[3] K. Yamaguchi, D. McAllester, and R. Urtasun. Robust
monocular epipolar flow estimation. In CVPR, 2013.
[4] Koichiro Yamaguchi, David McAllester, and Raquel Urtasun.
Efficient joint segmentation, occlusion labeling, stereo and flow
estimation. In ECCV, 2014.

Contact Information
Web http://www.cvlibs.net/projects/displets
Email {fatma.guney, andreas.geiger}@tue.mpg.de