Supplementary Material:
Joint 3D Object and Layout Inference
from a single RGB-D Image
Andreas Geiger1
1

Chaohui Wang1,2

Max Planck Institute for Intelligent Systems, Tübingen, Germany
2Université Paris-Est, Marne-la-Vallée, Paris, FranceAbstract. In this supplementary document we illustrate the 3D CAD
model database we use, give details on the sparse factor-to-variable message calculation and provide additional quantitative results in terms ofreconstruction, segmentation and 2D/3D detection performance. Furthermore, we show additional qualitative results and comparisons withrespect to Lin et al. [10] on randomly selected scenes from the NYUv2dataset.
In addition, the supplementary video visualizes the proposals, the inference process and our estimates in the form of wireframe projections and3D popups for several scenes. To illustrate how inference progresses, weshow the estimated max-marginal configuration at different iterations ofthe belief propagation algorithm. The video can be viewed with the VLC
media player, available at: http://www.videolan.org/vlc/.

1

Related Work and 3D CAD Model Database

Table 1 shows a table comparing related methods for scene understanding to theproposed method in terms of their properties. Fig. 1 shows scaled (i.e., normalized) versions of the 3D CAD models we collected from Google 3D Warehouse1 .
For consistency with the object annotations from [3], we selected the same 3Dmodels for the object classes bed, table, shelf, sofa and chair. Note that thisdoesn’t affect our evaluation as we compute the intersection-over-union of the3D bounding box associated with the estimated 3D models with respect to thecorresponding ground truth 3D bounding box.

1https://3dwarehouse.sketchup.com/

Our model

Lin et al. [10]Jia et al. [5]Jiang et al. [6]Kim et al. [7]Aubry et al. [1]Satkin et al. [13]Lim et al. [9]Hedau et al. [4]Schwing et al. [14]Pero et al. [11]Choi et al. [2]Andreas Geiger and Chaohui Wang

Zhang et al. [16]2Input360° 2D 2D 2D 2D 2D 2D 2D 3D 3D 3D 3D 3D#Object classes 16 6 8 1 1 8 1 2 9 1 11 21 21#Layout classes 1 5 5 5 5 – – – 4 – – – 3#Scene classes– 3 – – – – – – – – – 12 –3 3 3 3 333 3 3Context model
Visibility model3 3 33 3 333 33Occlusion model
CAD models
3 3 3333 3
3 3 3 3
Beyond MHWTable 1. Related Work. This table gives an overview over the most related work onindoor scene understanding with 3D models. “Beyond MHW” expresses the fact thatthe object orientations are not limited to the Manhattan world assumption.

Fig. 1. 3D Model Database. For each object category which can’t be well represented by a cuboid model we use one out of 66 CAD models collected from Google 3D
Warehouse. The colors correspond to the different semantic categories.

Joint 3D Object and Layout Inference from a single RGB-D Image

23Sparse Factor-to-Variable Message Calculation

Finding the minimum of E(x|I) with respect to x in the main paper is an
NP hard problem and we employ max-product loopy belief propagation to find
an approximate solution. For numerical stability we follow common practiceand make use of the equivalent min-sum formulation where messages between
variable i and the factor corresponding to clique c are passed as follows:X
fc (xc ) +
mtc→i (xi ) = minmt−1(1)j→c (xj )−ixc

mti→c (xi ) =j∈c\{i}X

mt−1
c0 →i (xi )(2)c0 ∈Ni \{c}Here, Ni denotes all factors involving variable i and x−ic denotes all variables
involved in clique c except variable i. For high-order cliques the computational
complexity of computing the factor-to-variable message in Eq. 1 is O(DN ) ingeneral, where N is the number of variables in the clique and D denotes thenumber of labels2 . As the size of our high-order cliques easily exceeds N = 20 or
N = 30, brute force calculation is clearly intractable. We approach this problem
by taking advantage of the sparsity in our high-order potentials which can bewritten as3
ξ(x) if x ∈ S
f (x) =(3)0 otherwise
where ξ(x) : DN → R is an arbitrary function and S is a set of special stateswith |S|  2N . Note that in contrast to the high-order potentials consideredin [8, 12], we allow negative as well as positive values for ξ(x). The latter case isrequired by our superpixel-object consistency potentials (see paper, Eq. 5).
In Algorithm 1, we provide an efficient recursion (an illustration of which
is given below using a simple example) to exactly compute factor-to-variable
messages (Eq. 1) for the class of potentials specified in Eq. 3 given a limited
computational budget. Without loss of generality, we assume that the message
is passed to the first node in the clique. For other cases we simply switch thetarget variable with the first variable in the clique. Moreover, we assume that thevariables in the clique under consideration are indexed from 1 to N for clarity
of presentation.
To illustrate the idea, let us look at the simple case of a clique c with N = 3
binary variables (i.e., D = 2). We first consider the trivial case where no special
state is present, i.e., |S| = 0 and f (·) ≡ 0. In this case, Eq. 1 can be computed
efficiently by swapping the min and the sum operators: we obtain mtc→1 (·) ≡2For clarity, here we assume that all variables have the same number of labels D while
our algorithm has no such restriction.
3We define f (x) = 0 for non-special states as we can always add a constant to f (x)without changing the optimization problem.

4

Andreas Geiger and Chaohui Wang

Algorithm 1 Sparse Factor-to-Variable Message Passing
Input: Factor: fc (·), Target variable i = 1, Input messages: mt−1i→c (·) ∀i ∈ {2, . . . , N }Output: Factor-to-variable message: mtc→1 (·)∀i ∈ {2, . . . , N } : µi ← minx (mt−1. precompute all min-messagesi→c (x))∀x : mtc→1 (x) ← +∞. initialize output message to +∞y ← vector(N ). y will be set during the recursion
UpdateMessage(y,0). start recursion
procedure UpdateMessage(y,k)
if k < N then. partition contains ≥ 1 state
if k = 0 ∨ Tk (y) ∩ S 6= ∅ then . k = 0 or Tk (y) contains ≥ 1 special state
for all d ← 1, . . . , D do
. for all sub-partitions
yk+1 ← d. specify sub-partition
UpdateMessage(y,k+1). continue recursion
else. compute minimum for partition Tk (y)mtc→1 (y1 ) = min mtc→1 (y1 ), gk (y). gk (·) is defined in Eq. 5else. update minimum forsingle state TN (y)P
mtc→1 (y1 ) = min mtc→1 (y1 ), f (y) + i∈{2,...,N } mt−1i→c (yi )t−1with min-messages defined as µi = minx mi→c(x) . Next, let us
consider the presence of a single special state: S = {s}, s = (1, 1, 1)T . Let
Pi∈{2,3} µi

Tk (y) = {x|xi = yi , i ≤ k} ⊂ {1, . . . , D}N(4)denote the subset of states for which the first k variables in x are equal to thosein y. For N = 3 this allows us to partition the state space into 4 disjoint sets
T1 (0) ∪ T2 (1, 0) ∪ T3 (1, 1, 0) ∪ {s} and we obtain mtc→1 (·) as(mtc→1 (x) =g1 (0)

min g2 (1, 0), g3 (1, 1, 0), ξ(s) + if x = 0otherwiset−1i∈{2,3} mi→c (si )Pwhere gk (y) denotes the minimum of partition Tk (y),gk (y) =Xi∈{2,...,k}t−1mi→c(yi ) +Xµi(5)i∈{k+1,...,N }which can be computed in linear time. This partitioning naturally suggests arecursive implementation which we specify in Algorithm 1: we iterate over thevariables involved in the factor, update the minimum using the pre-computed
min-messages {µi } if no special state is included in the current partition, andcontinue the recursion otherwise. By adaptively exploring the state space, thisreduces the time complexity for computing sparse high-order factor-to-variable
messages from O(DN ) to O(DN 2 ). In the presence of multiple special states
the complexity depends on the distribution of these states and the worst-casecomplexity is O(|S|DN 2 ).

Joint 3D Object and Layout Inference from a single RGB-D Image

35

Additional Quantitative Resultsoverall

printermonitorboard

curtainmicrowave

blindsrefrigerator
oven

chest

sofa

chair

cabinet

shelf

table

headboardbathtubbed

sinkcounter

toiletmantel2D Object Detection In addition to the 3D object detection results shown in
Table 2 of the paper submission we investigate the performance of our method
on 2D object detection, i.e., we project all 3D bounding boxes into the image andevaluate detection performance using the traditional 2D intersection-over-union(IOU) criterion. Due to the large degree of clutter and occlusion in the scenes,
we use a IOU threshold of 30% for considering an object detection as correct.
As shown in Table 3 (unclipped case) and Table 3 (clipped case), our method
performs about equally for CAD models and cuboids, due to the perspective“flattening” of the results. However, note that for both, the clipped as well as
the unclipped case, our method compares favorably with respect to the approachof Lin et al. [10].

#obj
11 127 30 40 25 170 23 466 253 544 229 751 143 43 30 45 112 98 52 100 30 3322
[10] - 8 Proposals
0 28 42 17 0 36 7 25 29 32 37 25 26 7 0 11 32 28 20 34 11 27.46[10] - 15 Proposals 17 32 41 15 7 32 5 28 25 30 36 26 32 12 10 25 27 24 21 28 6 27.48[10] - 30 Proposals
0 29 36 16 29 28 6 27 23 29 33 26 30 16 6 26 24 22 21 28 0 26.40
Base-Det-Cuboid
0 20 12 3 17 14 23 12 7 11 12 10 19 21 12 9 7 8 8 12 3 11.40
Base-NMS-Cuboid
0 29 49 6 7 67 25 20 28 38 46 29 28 20 6 15 32 33 15 28 7 31.07
NoOcclusion-Cuboid 0 29 49 9 22 65 13 27 32 43 46 38 27 29 6 0 26 30 19 19 11 34.54
NoContext-Cuboid 0 36 51 3 32 62 13 29 32 42 43 33 28 34 11 4 10 10 10 21 6 32.33
FullModel-Cuboid
0 27 49 9 23 66 14 27 32 44 46 34 31 32 6 0 27 31 18 23 5 34.29
Base-Det-CAD
0 20 13 4 13 11 23 12 8 11 12 13 19 21 12 9 12 8 8 12 3 12.05
Base-NMS-CAD
0 26 47 8 0 63 25 23 28 38 43 26 29 24 11 14 22 33 15 26 7 30.02
NoOcclusion-CAD
0 26 52 8 31 67 0 28 26 42 53 30 28 32 6 4 9 31 18 20 0 32.40
NoContext-CAD
0 32 54 6 33 61 14 31 27 42 50 33 31 30 6 4 12 10 10 20 0 32.60
FullModel-CAD
0 27 57 7 31 73 15 32 27 44 55 29 32 30 6 4 10 33 16 23 0 33.97
FullModel-CAD-GT 60 59 86 24 70 91 41 61 69 78 89 47 78 83 76 68 20 63 68 51 51 63.92

overallmonitorprinterboard

curtainblindsmicrowave

refrigerator
ovenchest

chair

sofa

cabinetshelftableheadboardbedbathtubsink

toiletcountermantelTable 2. 2D Detection Unclipped. Evaluation in terms of F1 score (%). See text
for details.

#obj
10 126 30 36 25 169 23 455 242 534 228 703 137 42 29 40 111 91 50 81 25 3187
[10] - 8 Proposals 0 29 45 17 0 36 20 26 28 32 36 25 27 7 6 16 30 25 21 36 13 27.54[10] - 15 Proposals 18 33 43 16 7 32 22 28 24 30 35 26 33 9 5 31 25 22 19 27 6 27.31[10] - 30 Proposals 0 29 38 16 29 28 22 27 21 29 32 26 31 13 6 29 23 20 18 28 0 26.25
FullModel-CAD0 26 61 8 31 74 7 32 26 41 53 29 33 30 6 5 7 30 17 24 0 33.35

Table 3. 2D Detection Clipped. Evaluation in terms of F1 score (%). See text fordetails.

6Andreas Geiger and Chaohui Wang

4

Additional Qualitative Results

The following pages show additional qualitative results on the NYUv2 test set.
We compare our estimates against ground truth and the results of Lin et al. [10].
Each subfigure shows: Ground truth, results of [10], our results (top-to-bottom)
in form of 3D popups, re-projections, depth maps and semantic segmentations(left-to-right) using the semantic color coding scheme from Fig. 1 in the mainpaper. For the ground truth we directly show the depth channel of the RGB-D
image in the corresponding row of the reconstruction column. A legend illustrating what is shown in the individual subplots is given in Fig. 2Fig. 2. Additional Qualitative Results Legend.

Joint 3D Object and Layout Inference from a single RGB-D Image

78

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

910Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

11

12

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

13

14

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

15

16

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

17

18

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

19

20

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

21

22

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

23

24

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

25

26

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

27

28

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

29

30

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

31

32

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

33

34

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

35

36

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

37

38

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

39

40

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

41

42

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

43

44

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

45

46

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

47

48

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

49

50

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

51

52

Andreas Geiger and Chaohui Wang

Joint 3D Object and Layout Inference from a single RGB-D Image

53

References
1. Aubry, M., Maturana, D., Efros, A., Russell, B., Sivic, J.: Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models. In: CVPR
(2014)
2. Choi, W., Chao, Y.W., Pantofaru, C., Savarese, S.: Understanding indoor scenes
using 3D geometric phrases. In: CVPR (2013)
3. Guo, R., Hoiem, D.: Support surface prediction in indoor scenes. In: ICCV (2013)
4. Hedau, V., Hoiem, D., Forsyth, D.: Thinking inside the box: Using appearance
models and context based on room geometry. In: ECCV (2010)
5. Jia, Z., Gallagher, A., Saxena, A., Chen, T.: 3D-based reasoning with blocks, support, and stability. In: CVPR (2013)
6. Jiang, H., Xiao, J.: A linear approach to matching cuboids in RGB-D images. In:CVPR (2013)
7. Kim, B.S., Kohli, P., Savarese, S.: 3D scene understanding by Voxel-CRF. In: ICCV(2013)
8. Komodakis, N., Paragios, N.: Beyond pairwise energies: Efficient optimization forhigher-order MRFs. In: CVPR (2009)
9. Lim, J.J., Khosla, A., Torralba, A.: FPM: Fine pose parts-based model with 3D
CAD models. In: ECCV (2014)
10. Lin, D., Fidler, S., Urtasun, R.: Holistic scene understanding for 3D object detection with RGB-D cameras. In: ICCV (2013)
11. Pero, L.D., Bowdish, J., Kermgard, B., Hartley, E., Barnard, K.: Understanding
bayesian rooms using composite 3D object models. In: CVPR (2013)
12. Rother, C., Kohli, P., Feng, W., Jia, J.: Minimizing sparse higher order energyfunctions of discrete variables. In: CVPR (2009)13. Satkin, S., Hebert, M.: 3DNN: viewpoint invariant 3D geometry matching for scene
understanding. In: ICCV (2013)
14. Schwing, A.G., Fidler, S., Pollefeys, M., Urtasun, R.: Box in the box: Joint 3Dlayout and object reasoning from single images. In: ICCV (2013)15. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support
inference from RGB-D images. In: ECCV (2012)
16. Zhang, Y., Song, S., Tan, P., Xiao, J.: PanoContext: A whole-room 3D context
model for panoramic scene understanding. In: ECCV (2014)