Joint 3D Object and Layout Inference
from a single RGB-D Image
Andreas Geiger1
1

Chaohui Wang1,2

Max Planck Institute for Intelligent Systems, Tübingen, Germany
2
Université Paris-Est, Marne-la-Vallée, Paris, France

Abstract. Inferring 3D objects and the layout of indoor scenes from asingle RGB-D image captured with a Kinect camera is a challenging task.
Towards this goal, we propose a high-order graphical model and jointly
reason about the layout, objects and superpixels in the image. In contrast
to existing holistic approaches, our model leverages detailed 3D geometry using inverse graphics and explicitly enforces occlusion and visibility
constraints for respecting scene properties and projective geometry. We
cast the task as MAP inference in a factor graph and solve it efficiently
using message passing. We evaluate our method with respect to several
baselines on the challenging NYUv2 indoor dataset using 21 object categories. Our experiments demonstrate that the proposed method is able
to infer scenes with a large degree of clutter and occlusions.

1

Introduction

Robotic systems (e.g., household robots) require robust visual perception in order to locate objects, avoid obstacles and reach their goals. While much progresshas been made since the pioneering attempts in the early 60’s [33], 3D scene understanding remains a fundamental challenge in computer vision. In this paper,
we propose a novel model for holistic 3D understanding of indoor scenes (Fig. 1).
While existing approaches to the 3D scene understanding problem typically infer
only objects [16, 17] or consider layout estimation as a pre-processing step [25],our method reasons jointly about 3D objects and the scene layout. We explicitly
model visibility and occlusion constraints by exploiting the expressive power ofhigh-order graphical models. This ensures a physically plausible interpretationof the scene and avoids undercounting and overcounting of image evidence.
Following [17,25,38], our approach also relies on a set of 3D object proposals
and pursues model selection by discrete MAP inference. However, in contrast
to previous works, we do not fit cuboids to 3D segments in a greedy fashion.
Instead, we propose objects and layout elements by solving a set of “inversegraphics” problems directly based on the unary potentials in our model. This
allows us to take advantage of the increasing availability of 3D CAD models
and leads to more accurate geometric interpretations. We evaluate the proposed
method in terms of 3D object detection performance on the challenging NYUv2

2

Andreas Geiger and Chaohui Wang

Fig. 1. Illustration of our Results. Left-to-right: Inferred objects, superpixels(red=explained), reconstruction (blue=close to red=far) and semantics with color code.

dataset [38] and compare it to [25] as well as two simple baselines derived fromour model. Our code and dataset are publicly available1 .

2

Related Work

3D indoor scene understanding is a fundamental problem in computer vison andhas recently witnessed great progress enabled by the increasing performance ofsemantic segmentation and object detection algorithms [6, 10] as well as theavailability of RGB-D sensors. Important aspects of this problem include 3D
layout estimation [15, 36], object detection [35, 39], as well as semantic segmentation [13, 32]. A variety of geometric representations have been proposed, including cuboids [17, 25, 46], 3D volumetric primitives [8, 47], as well as CAD
models [1,24,35,39]. While the problem has traditionally been approached using
RGB images [1, 8, 23, 36, 46] and videos [42], the availability of RGB-D sensors [30] and datasets [38] nourish the hope for more accurate models of thescene [11, 16, 18, 47]. Towards this goal, a number of holistic models have been
proposed which take into account the relationship between objects (often represented as cuboids) and/or layout elements in the scene [4,22,37,45]. While CRFs
provide a principled way to encode such contextual interactions [43], modelingvisibility/occlusion rigorously is a very challenging problem [37, 41].
The approach that we present is particularly related to several recent works
which model the 3D scene using geometric primitives (e.g., cuboids) [17,25]. Despite their promising performance, these works ignore some important aspects
in their formulation. In [25], a pairwise graphical model is employed to incorporate contextual information, but visibility constraints are ignored, which leads
to overcounting of image evidence. In [17], undercounting of image evidence isaddressed by enforcing “explained” superpixels to be associated with at least one
object. However, occlusions are not considered (e.g., an object which explains asuperpixel might be occluded by another object at the same superpixel), which
can lead to implausible scene configurations. Besides, semantic labels and related
contextual information are ignored.
While 3D CAD models have been primarily used for object detection [24,35, 39, 48], holistic 3D scene understanding approaches typically rely on simpler
1

http://www.cvlibs.net/projects/indoor_scenes/

Joint 3D Object and Layout Inference from a single RGB-D Image3

cuboid models [17, 25]. In this work, we leverage the precise geometry of CAD
models for holistic 3D scene understanding. The advantages are two-fold: First,
we can better explain the depth image evidence. Second, it allows for incorporating visibility and occlusion constraints in a principled fashion.

3

Joint 3D Object and Layout Inference

We represent indoor scenes by a set of layout elements (e.g., “wall”, “floor”,“ceiling”) and objects (e.g., “chairs”, “shelves”, “cabinets”). Given an RGB-D
image I partitioned into superpixels S, our goal is to simultaneously infer all
layout and object elements in the scene. In particular, we reason about the type,semantic class, 3D pose and 3D shape of each object and layout element. Towards
this goal, we first generate a number of object and layout proposals given theobserved image I (see Section 3.4), and then select a subset of layout elements
and objects which best explain I and S via MAP inference in a CRF.
More formally, let L and O denote the set of layout and object proposals,
respectively. Each proposal ρi = (ti , ci , mi , ri , zi ) (i ∈ L ∪ O) comprises thefollowing attributes: the proposal type ti ∈ {layout, object}, its semantic classci ∈ {mantel, . . . , other}, a 3D object model indexed by mi ∈ {1, . . . , M }, theimage region ri ⊂ I which has generated the proposal, as well as a set of poseparameters zi which characterize pose and scale in 3D space. For each proposal,
the semantic class variable ci takes a label from the set of classes corresponding
to its type ti ∈ {layout, object}. We pre-aligned the scene with the camera
coordinate axis using the method of Silberman et al. [38] and assume that layout
elements extend to infinity. Thus, for ti = layout, mi indexes a 3D plane model,
and zi comprises the normal direction and the signed distance from the camera
center. For ti = object, mi indexes one of the 3D CAD models in our dataset or
a 3D cuboid if no CAD model is available for an object category. Furthermore,zi comprises the 3D pose (we only consider rotations around the up-vector). andscale parameters of the object, i.e., zi ∈ R3 × [−π, +π) × R3+ .
We associate a binary random variable Xi ∈ {0, 1} with each layout/object proposal ρi , taking 1 if scene element i is present and 0 otherwise. To
impose visibility/occlusion constraints and avoid evidence undercounting, we
also associate a binary random variable Xk (k ∈ S) with each superpixel k tomodel if the superpixel is explained (Xk = 1) or unexplained (Xk = 0). A valid
scene configuration should explain as many superpixels as possible while at thesame time satisfying Occam’s razor, i.e., simple explanations with a small number of layouts and objects should be preferred. We specify our CRF model on
X = {Xi }i∈{L∪O∪S} in terms of the following energy
E(x|I) =

X
i∈L

+

φL
(xi |I) +| i {z }Xk∈S

layoutXi∈Oκk (xck )| {z }φO (xi |I) +| i {z }occlusion/visibility

object+Xk∈S

φSk (xk ) +| {z }

superpixel

X

O,O
ψij
(xi , xj ) +

i,j∈O

|

{z

object-object}

X
i∈L∪O,k∈S

X
i∈L,j∈OS
ψik(xi , xk |I)|{z}occlusion/visibilityL,Oψij(xi , xj )|{zlayout-object

}

(1)

4

Andreas Geiger and Chaohui Wang

where xck = (xi )i∈ck denotes a joint configuration of all variables involved inOclique ck . The unary potentials φL
i and φi encode the agreement of proposal iSwith the image, and φk adds a penalty to the energy function if superpixel k is notS
explained by any object or layout element. The pairwise potentials ψik
and thehigh-order potentials κc ensure consistency between the scene and superpixels
while respecting visibility and occlusion constraints. Contextual information such
L,O
O,O
.
and ψij
as relative pose or scale is encoded in ψij

3.1

Unary Potentials

We assume that each proposal ρi originates from a candidate image region ri ⊂ I
which we use to define the layout and object unary potentials in the following.
Details on how we obtain these proposal regions will be specified in Section 3.4.
Layout Unary Potentials: We model the layout unary terms as
LφL
hL (ρi ) + bL xi
i (xi |I) = w(2)

where wL and bL are model parameters that adjust the importance and biasof this term and hL (ρi ) captures how well the layout proposal fits the RGB-D
image. More specifically, we favour layout elements which agree with the depth
image and occlude as little pixels as possible, i.e., we assume that the walls,
floor and ceiling determine the boundaries of the scene. In particular, we definehL (ρi ) as the difference between the count of pixels occluded by proposal ρi andthe number of depth inliers wrt. all pixels in region ri .
Object Unary Potentials: Similarly, we define the object unary terms as
OφO
hO (ρi ) + bO xi
i (xi |I) = w(3)

where hO (ρi ) captures how well the object fits the RGB-D image: We consider
an object as likely if its scale (last 3 dimensions of zi ) agrees with the scale ofthe 3D object model si , its rendered depth map agrees with the RGB-D depth
image and its re-projection yields a region that maximizes the overlap with theregion ri which has generated the proposal. We assume a log-normal prior forthe scale si , which we learn from all instances of class ci in the training data.
Superpixel Unary Potentials: For each superpixel k we defineφSk (xk ) = wS (1 − xk )(4)

where wS ≥ 0 is a penalty assigned to each superpixel k which is not explained.
This term encourages the explanation of as many superpixels as possible. Notethat without such a term, we would obtain the trivial solution where none ofthe proposals is selected. Due to the noise in the input data and the approximations in the geometry model we enforce this condition as a soft constraint, i.e.,
superpixels may remain unexplained at cost wS , cf. Fig. 1.

Joint 3D Object and Layout Inference from a single RGB-D Image

3.2

5

Visibility and Occlusion Potentials

To ensure that the selected scene elements and superpixels satisfy visibility andSocclusion constraints we introduce the potentials κk and ψik.
High-Order Consistency Potentials: κk (xck ) is defined as:P∞ if xk = 1 ∧ i∈L∪O xi = 0
κk (xck ) =(5)0 otherwise
Here, the clique ck ⊆ {k} ∪ L ∪ O comprises the superpixel k and all proposals
i ∈ L ∪ O that are able to explain superpixel k. In practice, we consider asuperpixel as explained by a proposal if its rendered depth map is within athreshold (in our case 0.2 m) of I for more than 50% of the comprised pixels.
Note that Eq. 5 ensures that only superpixels which are explained by at least
one object can take label xk = 1.
Occlusion Potentials: Considering κk (xck ) alone will lead to configurations
where a superpixel is explained by objects which are themselves occluded byother objects at the same superpixel, thus violating visibility. To prevent thisSsituation, we introduce pairwise occlusion potentials ψik
between all scene elements i ∈ L ∪ O and superpixels k ∈ S
∞ if xi = 1 ∧ xk = 1 ∧ “i occludes k”
Sψik (xi , xk |I) =(6)0 otherwise
where “i occludes k” is true if for more than 50% of the pixels in superpixel k thedepth of the rendered object i is at least 0.2m smaller than the corresponding
depth value in I. In other words, we prohibit superpixels from being explained
if one or more active scene elements occlude the view.
3.3

Context Potentials

We also investigate contextual cues in the form of pairwise relationships between
object and layout elements as described in the following.
Object-Object Potentials: The pairwise potential between object i and j ismodeled as the weighted sum
X
O,O
t
ψij
(xi , xj ) =wt ψij(xi , xj )(7)t∈{p,s,ovlp}twhere ψij
is a feature capturing the relative pose, scale or overlap between
object i and object j. We encode the pose and scale correlation between objects
conditioned on the pair of semantic classes. For the pose, let distij (zi , zj ) androtij (zi , zj ) denote the distance and the relative rotation (encoded as cosine
similarity) between object i and j, respectively. For each pair (c, c0 ) of semantic
classes, we estimate the joint distribution ppc,c0 (distij , rotij ) from training datausing kernel density estimation (KDE). The relative pose potentials betweenpa pair of objects are then defined by the negative log-likelihood ψij(xi , xj ) =6Andreas Geiger and Chaohui Wang

− xi xj log ppci ,cj (distij (zi , zj ), rotij (zi , zj )). Similarly, we consider scale by thenegative logarithm of the relative scale distribution between semantic classes cisand cj as ψij(xi , xj ) = − xi xj log psci ,cj (sij ). Here, the relative scale sij is definedas the difference of the logarithm in scale and psci ,cj (sij ) is learned from training
data using KDE. To avoid objects intersecting each other, we further penalize
the overlapping volume of two objects
V (ρi ) ∩ V (ρj ) V (ρi ) ∩ V (ρj )ovlp+
ψij (xi , xj ) = xi xj
V (ρi )V (ρj )
where V (ρ) denotes the space occupied by the 3D bounding box of proposal ρ.
Layout-Object Potentials: Regarding the pairwise potential between layout
i and object j, we consider the relative pose and volume exclusion constraints
in analogy to those for the object-object potentials specified above:
L,O
p
ovlp
ψij
(xi , xj ) = wp ψij(xi , xj ) + wovlp ψij(xi , xj )

(8)

p
ovlp
Here, ψij
denotes the log-likelihood of the object-to-plane distance and ψij
penalizes the truncation of an object volume by a scene layout element.

3.4

Layout and Object Proposals

As discussed in the previous sections, our discrete CRF takes as input a set oflayout and object proposals {ρi }. We obtain these proposals by first generating
a set of foreground candidate regions {ri } using [3, 12] and then solving the“inverse graphics problem” by drawing samples from the unary distributions
specified in Eq. 2 and Eq. 3 for each candidate region ri .
Foreground Candidate Regions: For generating foreground candidate regions, we leverage the CPMC framework [3] extended to RGB-D images [25].
Furthermore, we use the output of the semantic segmentation algorithm of [12]as additional candidate regions. While [3] only provides object regions, [12] additionally provides information about the background classes wall, floor and ceiling.
In contrast to existing works on RGB-D scene understanding which often relyon simple 3D cuboid representations [17, 25], we explicitly represent the shapeof objects using 3D models. For indoor objects such data becomes increasingly
available, e.g., searching for “chair”, “sofa” or “cabinet” in Google’s 3D Warehouse returns more than 10, 000 hits per keyword. In our case, we make use of
a compact set of 66 models to represent object classes with non-cuboid shapes.
Proposals from Unary Distributions: Unlike [17, 25], we do not fit thetightest 3D cuboid to each candidate region for estimating the proposal’s poseparameters as this leads to an undesirable shrinking bias. Instead, we sample
proposals directly from the unary distributions specified in Section 3.1 using
Metropolis-Hastings [9, 26], leveraging the power of our 3D models in a generative manner. More specifically, for each layout candidate region, we draw samples
from pL (zi , mi ) ∝ exp −φL (zi , mi |I) and for each object
candidate region wedraw samples from pO (zi , mi ) ∝ exp −φO (zi , mi |I) . Here, the potentials φL

Joint 3D Object and Layout Inference from a single RGB-D Image

7and φO are defined as the right hand sides of Eq. 2 and Eq. 3, fixing xi = 1.
Note that for proposal generation φL and φO depend on the pose and model parameters while those arguments are fixed during subsequent CRF inference. By
restricting zi to rotations around the up-axis we obtain an 8-dimensional sampling space for objects. For layout elements the only unknowns are the normal
direction and the signed distances from the camera coordinate origin.
We randomly choose between global and local moves. Our global moves sample new pose parameters directly from the respective prior distributions which
we have learned from annotated objects in the NYUv2 training set [11]. Modesof the target distribution are explored by local Student’s t distributed moves
which slightly modify the pose, scale and shape parameters. For each candidate
region ri we draw 10, 000 samples using the OpenGL-based 3D rendering engine
librender presented in [14] and select the 3 most dominant modes.
3.5

Inference

Despite the great promise of high-order discrete CRFs for solving computer vision problems [2, 43], MAP inference in such models remains very challenging.
Existing work either aims at accelerating message passing for special types of potentials [7,20,27,31,40] or exploits sparsity of the factors [19,21,34]. Here, we explore the sparsity in our high-order potential functions (cf., Eq. 5) and recursively
split the state space into sets depending on whether they do or do not containany special state as detailed in the supplementary material. The class of sparsehigh-order potentials which can be handled by our recursive space-partitioning isa generalization of the pattern-based potentials proposed in [21, 34]. In contrast
to [21, 34], our algorithm does not make the common assumption that energy
values corresponding to “pattern” states are lower than those assigned to allother states as this assumption is violated by the high-order potential in Eq. 5.
For algorithmic details, we refer the reader to the supplementary material.

4

Experimental Results

We evaluate our method in terms of 3D object detection performance on thechallenging NYUv2 RGB-D dataset [38] which comprises 795 training and 654
test images of indoor scenes including semantic annotations. For evaluation,
we use the 25 object and layout (super-)categories illustrated in Fig. 1 andleverage the manually annotated 3D object ground truth of [11]. We extract
400 superpixels from each RGB-D image using the StereoSLIC algorithm [44],adapted to RGB-D information and generate about about 100 object proposals
per scene. The parameters in our model (wL = 1, bL = 0, wO = 1.45, bO = 1.3,
wS = 1.3, wp = ws = 0.001, and wovlp = 100) are obtained by coordinate
descent on the NYUv2 training set and kept fixed during all our experiments.
Evaluation Criterion: We evaluate 3D object detection performance by computing the F1 measure for each object class and taking the average over alloverallprintermonitorcurtainboard

blindsmicrowave

oven

chest

refrigerator

chair

sofa

cabinet

shelf

table

headboard

bed

sink

bathtubcounter

toiletAndreas Geiger and Chaohui Wang

mantel8#obj[25] - 8 Proposals[25] - 15 Proposals[25] - 30 Proposals
Base-Det-Cuboid
Base-NMS-Cuboid
NoOcclusion-Cuboid
NoContext-Cuboid
FullModel-Cuboid
Base-Det-CAD
Base-NMS-CAD
NoOcclusion-CAD
NoContext-CAD
FullModel-CAD

10 126 30 36 25 169 23 455 242 534 228 703 137 42 29 40 111 91 50 81 25 3187
0
4 27 12 0 13 0
8
13
3
16
8
5
0 0 0 13 5 3 8 0 7.90
0
3 27 10 0 11 0
7
11
3
19
8
4
0 0 11 11 6 3 6 0
7.71
0
3 24 11 12 10 0
7
10
3
18
9
5
0 0 11 11 5 3 6 0
7.61
08
3 2 13 12 5
8
3
6
6
4
14 14 7 33
2 1 4 25.80
0
3 16 00 51 6 11814 12
724 10 6 0 10 7 2 7 4 11.93
05
8 3 22 51 7 15
917 17 10 21 17 0 0
6
6 2 1 5 13.68
0
9
7 2 27 51 6 17
7
18 16
6
21 23 5 04
2 1 5 6 13.38
06
8 3 23 51 7 15
8
18 17
7
24 21 0 06
6 2 6 5 13.45
0
8 13 2 11 10 5 10468914 14 7 45
3 4 4 17.66
0
2 43 30 48 6 16
914 21 15 23 14 5 66
5 2 5 4 15.05
0
4 52 4 25 49 0 21
917 30 18 24 24 0 00
6 4 3 0 17.57
0
8 47 4 28 45 7 23
820 28 20 25 22 0 4
2
4 5 4 0 18.61
04 61 4 31 55 7 24 10 19 33 18 27 24 0 01
6 3 5 0 19.22[25] - 8 Proposals[25] - 15 Proposals (vis)[25] - 30 Proposals (vis)
FullModel-CAD (vis)

0
0
0
0

4
6
5
7

27 12 0 13
33 10 0 12
30 11 12 11
61 8 31 56

0
0
0
7

8
10
9
25

13
13
12
13

3
6
6
21

16
23
22
31

8
10
10
18

5
0
8
0
9
0
26 16

0
0
0
0

0 13 5 3 8
16 14 10 5 10
16 13 9 5 10
0
2 11 5 6

0
0
0
0

7.90
10.12
9.96
20.47

Table 1. 3D Detection Performance on 21 Object Classes of NYUv2. The firstpart of the table shows results for [25], our baselines and our full model (FullModelCAD) when evaluating the full extent of all 3D objects (i.e., including the occluded
parts) in terms of the weighted F1 score (%). The second part of the table shows F1scores when evaluating only the visible parts. See text for details.

classes, weighted by the number of instances. An object is counted as true positive if the intersection-over-union of its 3D bounding box with respect to theassociated ground truth 3D bounding box is larger than 0.3. This threshold ischosen smaller than the 0.5 threshold typically chosen for evaluating 2D detection [5] as the 3D volume intersection-over-union criterion is much more sensitive
compared to its 2D counterpart.
Ablation Study: In this section, we evaluate the importance of the individualcomponents in our model. First, we compare our method when using CAD models vs. using only simple Cuboid models as object representation. As illustrated
in Table 1, we obtain a relative improvement in F1 score of 42.2% when using
CAD object models in our full graphical model (FullModel-CAD vs. FullModelCuboid), highlighting the importance of accurate 3D geometry modeling for thistask. Next, we compare our full model with versions which exclude the occlusion (NoOcclusion) or context (NoContext) terms in our model. From Table 1,
it becomes evident that the occlusion term is more important than context,
improving the F1 score by 9.1%. Adding the contextual relationship improves
performance by 3.2%. Finally, Fig. 2 displays the 3D detection performance ofour model with respect to the number of proposals (first subfigure) and superpixels (second subfigure) evaluating objects to their full extent (blue) or only
the visible part (red) by clipping all bounding boxes accordingly.
Baselines: In this section, we quantitatively compare our method against a
recently published state-of-the-art algorithm [25] and two simpler baselines derived from our full model: For our first baseline (Base-Det), we simply threshold
our unary detections at their maximal F1 score calculated over the training set.
Our second baseline (Base-NMS) additionally performs greedy non-maximum-

20

15
10
Visible Parts
Full Extent

5010Visible Parts
Full Extent

5
0

0

20

40

60

#Proposals

80

Base-Det-CAD
Base-NMS-CAD
FullModel-CAD

0.4

15

0.3
0.2
0.1

500

#Superpixels

1000

0.3
0.2
0.1
0

0
0

9

Base-Det-Cuboid
Base-NMS-Cuboid
FullModel-Cuboid

0.4
Precision

25

20

Precision25

F1 score

F1 score

Joint 3D Object and Layout Inference from a single RGB-D Image0

0.1

0.2 0.3
Recall

0.4

0

0.1

0.2 0.3
Recall

0.4

Fig. 2. 3D Object Detection. From left-to-right: Performance of full model wrt.
number of proposals and wrt. number of superpixels. Precision-recall curves of thebaselines wrt. the full model when using 3D CAD models and cuboid primitives.

suppression, selecting only non-overlapping objects from the proposal set. As our
results in Table 1 show, our method yields relative improvements in F1 scoreof 149.4% and 27.2% wrt. Base-Det-CAD and Base-NMS-CAD, respectively.
Furthermore, the third and fourth plot of Fig. 2 show the performance of thebaselines in terms of precision and recall when varying the detection threshold.
We further compare our method to [25] as their setup is most similar toours and their code for training and evaluation is available. As [25] is only able
to detect the visible part of objects and has been trained on a ground truth
dataset biased towards cuboids, we re-train their method on the more recent
and complete NYUv2 ground truth annotations by Guo et al. [11] clipped to
the visible range and report results for different number of proposals (8, 15, 30).
For a fair comparison, we evaluate only the visible parts of each object (visible,
lower part of the table). On average, we double the F1 score wrt. [25]. The
differences are especially pronounced for furniture categories such as bathtub,
bed, table, cabinet, sofa and chair, showing the benefits of leveraging powerful
3D models during inference. Furthermore, we note that the performance of [25]drops with the number of proposals while the performance of our method keeps
increasing (Fig. 2), which is a favorable property considering future work atlarger scales. For completeness, we also show the performance of [25] on theunclipped bounding boxes (first rows of Table 1).
Qualitative Results: Fig. 3 visualizes our inference results on a number ofrepresentative NYUv2 test images. Each panel displays (left-to-right) the inferred object wireframe models, virtual 3D renderings and the corresponding
semantic segmentation. Note how our approach is able to recover even complex shapes (e.g., chair in row 1, right column) and detects heavily occluded 3D
objects (e.g., bathtub and toilet in row 5, right column). The two lower rows
show some failure cases of our method. In the top-left case, the sink is detected
correctly, but intersects the volume of the containing cabinet which is removed
from the solution. For most other cases, either the semantic class predictions
which we take as input are corrupt, or the objects in the scene do not belong to
the considered categories (such as person, piano or billiard table). However, notethat even in those cases, the retrieved explanations are functionally plausible.
Furthermore, flat objects are often missed due to the low probability of theirvolume intersecting the ground truth in 3D. Thus (and for completeness) we10Andreas Geiger and Chaohui Wang

Fig. 3. Inference Results. Each subfigure shows: Object wireframes, rendered depth
map and induced semantic segmentation.

also provide an evaluation of the objects projected onto the 2D image (similar
to the one carried out in [25]) in our supplementary material.
Runtime: On average, our implementation takes 119.2 s for generating proposals (∼ 6, 000 samples/second via OpenGL), 7.9 s for factor graph construction
and 0.7 s for inference on an i7 CPU running at 2.5 Ghz.

5Conclusion

In this paper, we have proposed a model for 3D indoor scene understanding from
RGB-D images which jointly considers the layout, objects and superpixels. Our
experiments show improvements with respect to two custom baselines as well asa state-of-the-art scene understanding approach which can be mainly attributed
to two facts: First, we sample more accurate 3D CAD proposals directly fromthe unary distribution and second, the proposed model properly accounts for
occlusions and satisfies visibility constraints. In the future, we plan to address
more complete scene reconstructions, e.g., obtained via volumetric fusion in order
to increase object visibility and thus inference reliability. Furthermore, we plan
to extend our model to object based understanding of dynamic scenes from
RGB/RGB-D video sequences by reasoning about 3D scene flow [28, 29].

Joint 3D Object and Layout Inference from a single RGB-D Image

11

References
1. Aubry, M., Maturana, D., Efros, A., Russell, B., Sivic, J.: Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models. In: CVPR
(2014)
2. Blake, A., Kohli, P., Rother, C.: Markov Random Fields for Vision and Image
Processing. MIT Press (2011)
3. Carreira, J., Sminchisescu, C.: CPMC: automatic object segmentation using constrained parametric min-cuts. PAMI 34(7), 1312–1328 (2012)
4. Choi, W., Chao, Y.W., Pantofaru, C., Savarese, S.: Understanding indoor scenes
using 3D geometric phrases. In: CVPR (2013)
5. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The
pascal visual object classes (voc) challenge. IJCV 88(2), 303–338 (2010)
6. Felzenszwalb, P.F., Girshick, R.B., McAllester, D.A.: Cascade object detection
with deformable part models. In: CVPR (2010)
7. Felzenszwalb, P.F., Mcauley, J.J.: Fast inference with min-sum matrix product.
PAMI 33(12), 2549–2554 (2011)
8. Fouhey, D.F., Gupta, A., Hebert, M.: Data-driven 3D primitives for single image
understanding. In: ICCV (2013)
9. Gilks, W., Richardson, S.: Markov Chain Monte Carlo in Practice. Chapman &Hall (1995)
10. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR (2014)
11. Guo, R., Hoiem, D.: Support surface prediction in indoor scenes. In: ICCV (2013)
12. Gupta, S., Arbelaez, P., Malik, J.: Perceptual organization and recognition of indoor scenes from RGB-D images. In: CVPR (2013)
13. Gupta, S., Girshick, R., Arbeláez, P., Malik, J.: Learning rich features from RGB-D
images for object detection and segmentation. In: ECCV (2014)
14. Güney, F., Geiger, A.: Displets: Resolving stereo ambiguities using object knowledge. In: CVPR (2015)15. Hedau, V., Hoiem, D., Forsyth, D.: Recovering the spatial layout of cluttered
rooms. In: ICCV (2009)
16. Jia, Z., Gallagher, A., Saxena, A., Chen, T.: 3D-based reasoning with blocks, support, and stability. In: CVPR (2013)
17. Jiang, H., Xiao, J.: A linear approach to matching cuboids in RGB-D images. In:
CVPR (2013)
18. Kim, B., Xu, S., Savarese, S.: Accurate localization of 3D objects from RGB-D
data using segmentation hypotheses. In: CVPR (2013)
19. Kohli, P., Ladicky, L., Torr, P.H.S.: Robust higher order potentials for enforcing
label consistency. IJCV 82(3), 302–324 (2009)
20. Kohli, P., Pawan Kumar, M.: Energy minimization for linear envelope MRFs. In:
CVPR (2010)
21. Komodakis, N., Paragios, N.: Beyond pairwise energies: Efficient optimization forhigher-order MRFs. In: CVPR (2009)
22. Lee, D., Gupta, A., Hebert, M., Kanade, T.: Estimating spatial layout of rooms
using volumetric reasoning about objects and surfaces. In: NIPS (2010)
23. Lim, J.J., Khosla, A., Torralba, A.: FPM: Fine pose parts-based model with 3D
CAD models. In: ECCV (2014)
24. Lim, J.J., Pirsiavash, H., Torralba, A.: Parsing IKEA objects: Fine pose estimation.
In: ICCV (2013)12

Andreas Geiger and Chaohui Wang

25. Lin, D., Fidler, S., Urtasun, R.: Holistic scene understanding for 3D object detection with RGB-D cameras. In: ICCV (2013)
26. Mansinghka, V., Kulkarni, T., Perov, Y., Tenenbaum, J.: Approximate bayesian
image interpretation using generative probabilistic graphics programs. NIPS 2013(2013)
27. Mcauley, J.J., Caetano, T.S.: Faster algorithms for max-product message-passing.
JMLR 12, 1349–1388 (2011)
28. Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: CVPR (2015)29. Menze, M., Heipke, C., Geiger, A.: Joint 3d estimation of vehicles and scene flow.
In: ISA (2015)30. Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J.,
Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A.: Kinectfusion: Real-time dense
surface mapping and tracking. In: ISMAR (2011)31. Potetz, B., Lee, T.S.: Efficient belief propagation for higher-order cliques using
linear constraint nodes. CVIU 112(1), 39–54 (2008)
32. Ren, X., Bo, L., Fox, D.: RGB-(D) scene labeling: Features and algorithms. In:
CVPR (2012)
33. Roberts, L.G.: Machine perception of three-dimensional solids. Ph.D. thesis, Massachusetts Institute of Technology (1963)
34. Rother, C., Kohli, P., Feng, W., Jia, J.: Minimizing sparse higher order energyfunctions of discrete variables. In: CVPR (2009)35. Satkin, S., Hebert, M.: 3DNN: viewpoint invariant 3D geometry matching for scene
understanding. In: ICCV (2013)
36. Schwing, A., Urtasun, R.: Efficient exact inference for 3D indoor scene understanding. In: ECCV (2012)
37. Schwing, A.G., Fidler, S., Pollefeys, M., Urtasun, R.: Box in the box: Joint 3Dlayout and object reasoning from single images. In: ICCV (2013)
38. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support
inference from RGB-D images. In: ECCV (2012)
39. Song, S., Xiao, J.: Sliding shapes for 3D object detection in depth images. In:
ECCV (2014)
40. Tarlow, D., Givoni, I.E., Zemel, R.S.: Hop-map: Efficient message passing withhigh order potentials. In: AISTATS (2010)
41. Tighe, J., Niethammer, M., Lazebnik, S.: Scene parsing with object instances andocclusion ordering. In: CVPR (2014)
42. Tsai, G., Xu, C., Liu, J., Kuipers, B.: Real-time indoor scene understanding using
bayesian filtering with motion cues. In: ICCV (2011)
43. Wang, C., Komodakis, N., Paragios, N.: Markov random field modeling, inference& learning in computer vision & image understanding: A survey. CVIU 117(11),1610–1627 (2013)
44. Yamaguchi, K., McAllester, D., Urtasun, R.: Robust monocular epipolar flow estimation. In: CVPR (2013)
45. Zhang, H., Geiger, A., Urtasun, R.: Understanding high-level semantics by modeling traffic patterns. In: ICCV (2013)
46. Zhang, Y., Song, S., Tan, P., Xiao, J.: PanoContext: A whole-room 3D context
model for panoramic scene understanding. In: ECCV (2014)
47. Zheng, B., Zhao, Y., Yu, J.C., Ikeuchi, K., Zhu, S.C.: Beyond point clouds: Scene
understanding by reasoning geometry and physics. In: CVPR (2013)
48. Zia, M., Stark, M., Schiele, B., Schindler, K.: Detailed 3D representations for object
recognition and modeling. PAMI 35(11), 2608–2623 (November 2013)