Displets: Resolving Stereo Ambiguities using Object Knowledge
Fatma Güney, Andreas Geiger
MPI Tübingen

Stereo techniques have witnessed tremendous progress over the last decades,
yet some aspects of the problem still remain challenging today. Striking examples are reflecting and textureless surfaces which cannot easily be recovered using traditional local regularizers. In this paper, we therefore propose
to regularize over larger distances using object-category specific disparity proposals (displets) which we sample using inverse graphics techniques
based on a sparse disparity estimate and a semantic segmentation of theimage. The proposed displets encode the fact that objects of certain categories are not arbitrarily shaped but typically exhibit regular structures.
We integrate them as non-local regularizer for the challenging object class’car’ into a superpixel based CRF framework and demonstrate its benefits
on the KITTI stereo evaluation. At time of submission, our approach ranks
first across all KITTI stereo leaderboards.
Introduction: In this paper, we investigate the utility of mid-level processes such as object recognition and semantic segmentation for the stereo
matching task. In particular, we focus our attention on the reconstruction ofwell-defined objects for which the data term is weak and current methods
perform poorly, such as cars. Due to their textureless, reflective and semitransparent nature, those object categories represent a major challenge forcurrent state-of-the-art algorithms, as illustrated in Fig. 1 (top). In contrast,
as humans we are able to effortlessly extract information about the geometryof cars even from a single image thanks to our object knowledge and shape
representation. Inspired by this fact, we introduce object knowledge forwell-constrained object categories into a slanted-plane MRF and estimate adense disparity map. We leverage semantic information and inverse graphics to sample a set of plausible object disparity maps given an initial semidense disparity estimate. We encourage the presence of these 2.5D shapesamples (or displets) in our MRF formulation depending on how much theirgeometry and semantic class agrees with the observation. Intuitively, displets can be thought of as a representative finite subset of the infinitely large
set of possible disparity maps for a certain semantic category conditionedon the image. For example, car displets should cover the most likely 3D car
configurations and shapes given the two input images.

Figure 1: Resolving Stereo Matching Ambiguities: Current stereo methods often fail at reflecting or texturelesssurfaces (top, [5]). By using object
knowledge, we encourage disparities to agree with plausible surfaces (center). This improves results both quantitatively and qualitatively while simultaneously recovering the 3D geometry of the objects in the scene (bottom).

Results: Our experiments indicate that the proposed framework is able
to resolve stereo ambiguities on challenging stereo pairs from the KITTI
benchmark [1] as illustrated in Fig. 1 (center). At the same time our method
is able to extract 3D object representations which are consistent with theestimated disparity map and may serve as input to higher-level reasoning,see Fig. 1 (bottom) for an illustration. Table 1 and Table 2 show quantitative
results on the KITTI stereo benchmark at the time of submission using thedefault error threshold of 3 pixels. The numbers represent outliers (in %)and average disparity errors (in pixels). Methods marked with an asterisk
Model: We assume that the image can be decomposed into a set of planar are scene flow methods which use two or more stereo image pairs as input.
superpixels S and each superpixel i ∈ S is associated with a random variable
Rank
MethodOut-Noc
Out-All Avg-Noc
Avg-Allni describing a plane in 3D. D denotes the set of displets in the image and1
Our Method
2.47 %
3.27 %
0.7 px0.9 pxeach displet k ∈ D is associated with its class label ck , a fitness value, and2MC-CNN [5]2.61 %
3.84 %
0.8 px
1.0 pxa set of superpixels Sk ⊆ S on which it is defined. An additional random
3SPS-StFl* [4]2.83 %
3.64 %
0.8 px0.9 px
4VC-SF* [2]3.05 %
3.31 %
0.8 px0.8 pxvariable dk ∈ {0, 1}, which can be interpreted as auxiliary variable in a highSPS-St [4]3.39 %
4.41 %
0.9 px
1.0 px5order CRF, denotes the presence (dk = 1) or absence (dk = 0) of the displet in
6PCBP-SS [3]3.40 %
4.72 %
0.8 px
1.0 pxthe scene. Furthermore, we assume that we have access to a rough semantic
Table
1:
All
Regions
segmentation of the image. Our goal is to jointly infer all superpixel planeparameters ni as well as the presence or absence of all displets dk in theMethod
Out-Noc
Out-All
Avg-Noc
Avg-AllRank
1
Our Method8.40 %
9.89 %
1.9 px
2.3 pxscene. We specify our CRF in terms of an energy function
E(n, d)=∑ϕiS (ni ) +i∈S∑

ψiSj (ni , n j ) +i∼ j∑ ϕkD (dk ) + ∑ ∑ ψkiD (dk , ni )k∈Dk∈D i∈Sk

234..
.
11VC-SF* [2]PCBP-SS [3]SPS-StFl* [4]..
.
MC-CNN [5]11.58 %
14.26 %
14.74 %..
.
18.45 %

12.29 %
18.33 %
18.00 %..
.
21.96 %

2.7 px
2.4 px
2.9 px..
.
3.5 px

2.8 px
3.9 px
3.6 px
4.3 px

Table 2: Reflective Regions
where n = {ni |i ∈ S} and d = {dk |k ∈ D} and i ∼ j denotes the set ofadjacent superpixels in S. In addition to the data term ϕiS (ni ) and pairwise
constraints ψiSj (ni , n j ), we introduce long-range interactions into our modelusing displets: The unary potential ϕkD (dk ) encourages image regions withsemantic class label ck to be explained by a displet of the correspondingclass. Furthermore, ψkiD (dk , ni ) ensures that the displet and the associatedsuperpixels in the image are consistent.
This is an extended abstract. The full paper is available at the Computer Vision Foundation
webpage.

[1] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous
driving? The KITTI vision benchmark suite. In CVPR, 2012.
[2] Christoph Vogel, Stefan Roth, and Konrad Schindler. View-consistent 3D scene
flow estimation over multiple frames. In ECCV, 2014.
[3] K. Yamaguchi, D. McAllester, and R. Urtasun. Robust monocular epipolar flow
estimation. In CVPR, 2013.
[4] Koichiro Yamaguchi, David McAllester, and Raquel Urtasun. Efficient joint segmentation, occlusion labeling, stereo and flow estimation. In ECCV, 2014.
[5] Jure Zbontar and Yann LeCun. Computing the stereo matching cost with a convolutional neural network, 2014.