Title: JOINT OBJECT AND RELATIONAL NET-WORK FOR 3D PREDICTION

Abstract: We propose an approach to predict the 3D shape and pose for the objects present in a scene. Existing learning based methods that pursue this goal make independent predictions per object, and do not leverage the relationships amongst them. We argue that reasoning about these relationships is crucial, and present an approach to incorporate these in a 3D prediction framework. In addition to independent per-object predictions, we predict pairwise relations in the form of relative 3D pose, and demonstrate that these can be easily incorporated to improve object level estimates. We report performance across different datasets (SUNCG, NYUv2), and show that our approach significantly improves over independent prediction approaches while also outperforming alternate implicit reasoning methods.

1. INTRODUCTION:

A single 2D image can induce a rich 3D perception. When we look at an image, we can reason about its 3D layout, the objects in the image, their shape, extent, relationships etc.. This is really surprising given that going from a 2D projection to a 3D model is inherently ill-posed. How are we able to solve this problem? Humans rely on regularities in the 3D world in order to do sothis helps us discard many improbable solutions in 3D and reason about more likely ones. This regularity exists at the scene level -indoor scenes have roughly perpendicular walls; object levelchairs have similar shapes; and in local object relationships -chairs are close to tables, monitors are on top of tables etc.. A decade ago, a lot of work in computer vision focused on using all the three levels of regularities. For example, a lot of work focused on object-centered 3D prediction (Fidler et al., 2012) , scene-level 3D prediction (Hedau et al., 2009) , and multi-object 3D reasoning (Jiang et al., 2012) . However, in recent years, since the advent of ConvNets, a vast majority of computer vision approaches do not leverage these object-object relationships, and instead reason about each object independently.

In this paper, we attempt to take a holistic view of the 3D prediction problem and note that solving the 3D prediction problem would require incorporation of all the three cues. We believe there are three fundamental questions that need to be answered to design this holistic architecture: (a) What is the right representation for object level 3D prediction?; (b) How do we represent object-object relationships and how do we predict them from pixels?; (c) Finally, how to incorporate objectobject relationships with object-level modules. This paper builds upon the recent success in (a) and investigates how to model relationships and incorporate them into our 3D prediction framework.

So, how do we model relationships and estimate them from pixels? There is a whole spectrum of possible approaches. On one end of the spectrum is a complete end-to-end approach. Some examples of these include Interaction Networks (Battaglia et al., 2016) or Graph Convolutional Networks (Kipf & Welling, 2016). Both these methods provide a mechanism for object features in the scene to effect each other, thereby allowing an implicit modeling of relationships among them. However, as we show in our experiments, these end-to-end approaches disregard the structural information which might be crucial for modeling the relationships. The other end of spectrum is to use category-based image-agnostic pairwise priors (Zhao & Zhu, 2011) to model relationships. A drawback is that these priors are often too strong to generalize and it is better to learn them (Jiang et al., 2012) .

When it comes to the final question of how does one incorporate relationships to improve 3D prediction, the answer is even murkier. One classical approach is to use graphical models such as CRFs (Lafferty et al., 2001; Schwing et al., 2012) . However, these classical approaches have usually provided little improvements over object-based approaches. Our key insight is to incorporate structural information in end-to-end systems. Specifically, we model and predict pairwise relationships in the translation, rotation and scale space. One advantage of using this structured relationship space is that the incorporation of relationships into object-level estimates is simple yet effective. But how do we predict these pairwise relationships from pixels? Our paper investigates several design choices and proposes a simple architecture. Our method demonstrates significant improvement in performance across multiple metrics and datasets. As we show in our experiments, this modeling of relationships in this structured space provides a huge 6 point AP improvement in detection settings over current state-of-the-art 3D approaches. We will release our code for reproducibility.

2. RELATED WORK:

Dating back to the first thesis written on computer vision (Roberts, 1963), inferring the underlying 3D structure of an image has been a long-standing goal in the field. A common characteristic amongst these approaches is the reasoning at a per-object level. While the object-centric nature is certainly desirable as a representation, we argue that reasoning independently for each object to infer this representation is not, as it does not allow leveraging the relationships between the entities in a scene. We propose a method that also uses object-centric representations but goes beyond independent reasoning per object.

We are of course not the first to pursue reasoning about relationships between entities in a scene. We use the instance encoder to create an embedding for each object (instance) in the scene. The instance decoder uses this embedding to predict a pose for each object independently. The relative encoder takes each pair of instances and their embeddings to output an embedding for the pair. The relative decoder predicts a relationship (relative pose) between the pairs. We combine these relative and perobject predictions to predict final pose estimates for each object in an end-to-end differentiable framework.

relation inference as the end goal, we instead aim to leverage these for a per-instance prediction task. In the context of incorporating relations for such per-instance prediction, there are two alternate ideologies. On the one hand, approaches pursuing 3D scene inference or generation (Zhao & Zhu, 2011; Lin et al., 2013; Choi et al., 2013; Fisher et al., 2012; 2011; Huang et al., 2018; Jiang et al., 2018 ) typically incorporate pairwise (or higher order) relations via explicit class-based priors regarding possible configurations and optimize predictions to adhere to these. This approach of explicitly modeling relations as a prior imposes the same constraints across all scenes, independent of the structure in the image, and is therefore not flexible enough and has difficulties in scaling up to arbitrary relations across arbitrary objects.

The alternate ideology for incorporating relationships is to eschew any explicit structure for these relations, and instead implicitly capture these via architectural changes to the CNNs, thereby allowing the features of objects to influence one another (Battaglia et al., 2016; Kipf & Welling, 2016) . While this a generally applicable mechanism, it does not leverage several aspects regarding the structure of the problem -for 3D inference, specific relations like relative position, orientation are very relevant, and can be used in specific ways to influence per-instance predictions. Our approach leverages some aspects of both these ideologies -unlike the classical prior based approaches, we learn and infer these relations in a image-dependent context via a CNN, and unlike the purely implicit methods, we are more explicit about the structure and meaning of these relations.

3. APPROACH:

Our goal is to predict the 3D pose and shape for all the objects in a scene. We observe that in addition to the visual cues per object, reasoning about relationships across them can further help our predictions, in particular for the 3D pose -a chair would be in front of a table, and of a compatible relative size, and therefore even if we are uncertain about the pose of one of these objects, e.g., due to occlusion, these relationships can enable us to make accurate predictions.

We operationalize this insight in our method (see figure 1 ) that leverages both -independent perobject predictions alongwith predictions regarding relationships between them. We infer the final estimates for all the objects in the scene by integrating these two. We first formally describe the object-centric representations pursued and briefly review a recent per-instance prediction approach in section 3.1. We then introduce the relative representations in section 3.2 and present our network architecture that enables predicting these in section 3.3. In section 3.4 we discuss how these relative predictions are combined with the independent per-object predictions to yield the final 3D estimates for the objects. We show that optimizing the combination of these estimates (i.e., final estimate) in a differentiable end-to-end framework helps improve the final per-object predictions.

3.1. INSTANCE SPECIFIC REPRESENTATIONS AND INFERENCE:

We output the 3D pose of an object by predicting its shape in a canonical frame, and its scale, translation and rotation in camera frame. The shape is parametrized as a 32 3 volumetric occupancy grid in a canonical space where the objects are upright, front-facing, normalized to a unit cube. The translation t ∈ R 3 and (logarithm of) anisotropic scale s ∈ R 3 , and normalized quaternion q indicate the position, size and orientation of the object respectively. Following prior work (Tulsiani et al., 2018) , we parametrize the rotation prediction as a classification task among fixed bins.

We build upon the recent work by (Tulsiani et al., 2018) : they pursue a similar per-object representation, but make independent predictions across objects. We use their approach to obtain the independent predictions for each object. We briefly review their prediction framework but refer the reader to the paper for details about the representation and architecture. Their method uses an architecture similar to Fast R-CNN (Girshick, 2015) , where the input image is encoded via convolutional layers, and for each object bounding box, an RoI pooling layer crops the corresponding features. These per-object features, in conjunction with coarse image feature and an encoding of the bounding box coordinates, are encoded to an instance-specific bottleneck representation from which the corresponding shape and pose are predicted.

We adopt a similar architecture, illustrated in figure 2 , with the introduction of spatial coordinates as additional feature channels (see section 3.3) to obtain per-object (unary) predictions. We note that these predictions are obtained independently across objects, and do not incorporate reasoning across them. However, unlike previous work, these are subsequently integrated with relative predictions (section 3.4) to obtain final estimates.

3.2. RELATIONS AS RELATIVE REPRESENTATIONS:

Given an image of a scene, we can infer that two chairs nearby might be of a similar size, a laptop kept above a desk, and a television facing the bed etc.. Thus, relative pose between objects captures an important aspect of their relationship in the scene.

Concretely, given two objects, say A and B, we infer the relative pose from A to B. The relative pose, akin to the absolute 3D pose, is factored into the relative translation, scale, and direction. The relative translation is defined as the difference of the absolute locations of the object in the camera frame t AB = t B -t A . Similarly, the relative scale is simply the ratio of the two object sizes, or equivalently a difference in logarithms s AB = s B -s A . Finally, we predict a relative direction dAB , i.e., the direction of object B in the frame of object A: dAB ∝ ( RA ) t AB , where dAB is normalized to unit norm. We note that this parametrization, unlike relative rotation, helps us overcome some ambiguities due to symmetries, e.g., the relative direction from a chair to a table in front of it is unambiguous, even if the table is symmetric.

3.3. NETWORK ARCHITECTURE:

Our network has two branches -a per-object (unary) prediction module, and a relative prediction module. The architecture of both these branches is shown in figure 2 . The former, as described in section 3.1, simply computes a per-object encoding and subsequently makes per-object predictions. The relative prediction module is then tasked with inferring the relative pose (section 3.2) for every pair of objects in the scene.

As depicted in figure 2 , the relative prediction module takes as input: a) the encoding of both objects, and b) an encoding of the larger (union) bounding box containing both objects. Since this larger box may contain several additional objects beyond the ones of interest, we additionally give as input two channels with binary masks indicating the source and target bounding box extents respectively (denoted as 'object location image'). We also find it beneficial to concatenate the normalized per-pixel spatial coordinates as additional features, as it allows the network to easily reason about the absolute spatial location of the bounding box(es) under consideration. Finally, similar to the parametrization in section 3.1, we frame the relative direction prediction as a classification task among 24 bins, where the bins are computed by clustering relative directions across the training instances.

3.4. COMBINING PER-OBJECT AND RELATIVE PREDICTIONS:

We saw in section 3.1 that we can obtain independent per-object predictions for the 3D shape and pose, and introduced the relative pose predictions and architecture in section 3.2 and section 3.3. We denote by (t n , s n , q n ) the per-object (unary) predictions for the translation, log-scale, and rotation respectively for the n th object, and similarly (t mn , s mn , d mn ) denote the relative pose predictions from the m th to the n th object. Using these, we show that we can obtain final per-instance predictions (t * n , s * n , q * n ) that incorporate both, the unary and relative predictions.

Translation and Scale Prediction. The relative predictions give us linear constraints that the final predictions should ideally satisfy. As an example, we would want t * n -t * m = t mn . This can equivalently be expressed as A mn t * = t mn , where A mn is a sparse row-matrix with the m th and n th entries as (-1, 1), and t * denoting the final translations for all the N objects. We can similarly express all pairwise linear constraints as At * = t rel , where t rel denotes all the relative predictions, and A is the appropriate sparse matrix.

In addition to satisfying these linear constraints, we would also like the final estimates to be close to the unary predictions. We can therefore incorporate both, the relative and the unary constraints via a system of linear equations, and solve these to obtain the final estimates.

EQUATION

Here X + denotes the Moore-Penrose inverse of matrix X, and λ indicates the relative importance of the unary estimates. We can therefore obtain the final translation predictions t * that integrate both, unary and relative estimates. We note that the final estimates are simply a linear function of the unary predictions t and the relative predictions t rel , and it is therefore straightforward to propagate learning signal from supervision for t * to these predictions. While the description here focused predicting translation, as we represent scale using logarithm of the sizes, similar linear constraints apply. We can therefore similarly compute final predictions for the scales across objects s * .

Rotation Prediction. While the incorporation of unary and relative predictions can be expressed via linear constraints in the case of translation and scale, a similar closed form update does not apply for rotation because of the framing as a classification task and non-linearity of the manifold. Instead, we update the likelihood of the unary predictions based on how consistent each rotation bin is with the relative estimates.

We denote as R b the rotation matrix corresponding to the b th rotation bin, and use ∆(R, d, t) to measure how inconsistent a predicted rotation R is w.r.t the predicted relative direction distribution d and relative translation t (see appendix for details). Using these, we can compute the (unnormalized) negative-log likelihood distribution over possible rotations as follows:

EQUATION

This update to compute q * can equivalently be viewed as a single round of message passing, with the messages being of an explicit rather than an implicit form.

Training Details. We described above how the independent per-object predictions are analytically combined with the relative pose predictions to obtain the final estimates, and note that this integration process allows us to propagate learning signal from supervision on the final estimates back to the unary and relative predictions. Our training happens in two steps. In the first step, we train both unary and relative predictions independent of each other. We formulate the loss-function for each network similar to (Tulsiani et al., 2018) . Specifically, we use regression losses for shape encoding, the (absolute and relative) translation and scale, and classification losses for the rotation and relative direction prediction. Note that as some objects might be rotationally symmetric, we allow multiple 'correct' bins for these and maximize the maximum probability across these. In the second step, after a few epochs, we train the whole model in joint manner. We add similar losses for the final pose predictions that are computed using both, unary and relative estimates. During inference, given the unary and relative predictions, we simply compute the final pose predictions via the optimization process described above. Additional details on optimization are provided in the appendix.

4. EXPERIMENTS:



4.1. EXPERIMENTAL SETUP:

Datasets: We use the SUNCG dataset (Song et al., 2017 ) which provides many diverse and detailed 3D models for houses. Following (Zhang et al., 2017; Tulsiani et al., 2018) , we use the 2D renderings of the houses and the corresponding parsed 3D house model information to get roughly 550k image and 3D supervision pairs. We follow the setup in (Tulsiani et al., 2018) and split this data into train (70%), val (10%) and test (20%); with objects classes -bed, chair, desk, sofa, table, and tv.

We also use the NYUv2 (Silberman et al., 2012) dataset which consists of 1449 real world images, and use the annotations by (Guo & Hoiem, 2013) to finetune the network trained on SUNCG using the same subset of object categories. This dataset has lower resolution images and serves well to check the generalization of our approach to real data. As the NYUv2 annotations use the same small set of 3D shapes across train and test images, we do not evaluate shape prediction.

Metrics: (Tulsiani et al., 2018) propose different metrics to measure the quality of various prediction components -detection, rotation, translation and scale. They also propose different thresholds δ for each of these components which are used to count a prediction as a true positive in the detection setting. We use these metrics and refer the reader to the appendix for a summary review.

Baselines: We use the following baseline methods:

• Factored3D: This method from (Tulsiani et al., 2018) reasons about each object independently and serves as a baseline to see how our relationship reasoning can improve performance. • GCN: We use Graph Convolutional Networks (Kipf & Welling, 2016) to perform implicit relational reasoning. We use the Object Encoder (Figure 2 ) to obtain object embeddings o i for each object and then use a 2-layer GCN to obtain the final object embeddings for each object. These are then used to predict the 3D pose and shape for the objects. • InteractionNet: We use the method from (Battaglia et al., 2016) as an alternate way to perform implicit reasoning over the object embeddings. We compute an 'effect' embedding e AB for each ordered tuple (o A , o B ) via a learned MLP, and update the object embeddings by aggregating these as o A + max B (e AB ), and use these for per-object predictions.

Note that while the latter two baselines can implicitly reason about relationships, they ignore the underlying structure (how relative translations affect translation etc.).

Image GT Factored3D Ours Image GT Factored3D Ours

Figure 3 : We visualize sample prediction results in the setting with known ground-truth boxes. Our method produces better estimates than the baseline, e.g., bottom right where the distance between the chairs and table is predicted correctly, and the pose of the yellow chair is corrected. Best viewed in color.

4.2. USING GROUND TRUTH BOXES:

We first analyze all methods in the setting where we are given the ground truth bounding boxes. In this setting, we can analyze just the 3D prediction quality without the additional variance introduced due to imperfect detection. During training, we train all the methods on ground-truth boxes as well as object proposals (obtained using (Zitnick & Dollár, 2014 )) which have an IOU ≥ 0.7 and at test time, we evaluate them only on ground truth boxes. We evaluate the final (combined) output of our method. As shown in We visualize sample prediction results on the NYUv2 dataset in the setting with known ground-truth boxes. Our method produces better estimates than the baseline, e.g., for the top left image we see that the chairs are better aligned by our method as compared to the baseline. We use the ground-truth meshes for visualization due to lack of variability in shape annotations for learning. We show the median and mean error (lower is better) and the % of samples within a threshold (higher is better). also worth noting that adding any form of relationships modeling, like in GCN or InteractionNet, gives a boost over doing per-object predictions like in Factored3D. Our structured reasoning and inference about object relationships provides a boost over other pairwise models such as GCN and InteractionNet. Our performance gain over the baseline holds across SUNCG and NYUv2 datasets demonstrating the generalization of our method.

Qualitative Results: In figure 3 , we show a few results of our method and the baseline (Fac-tored3D). We see that our method can correct many error modes (relative positions and poses of objects) compared to the baseline. We observe similar trends on the NYUv2 dataset (figure 4 ).

Image GT Factored3D Ours Image GT Factored3D Ours 

4.3. DETECTION SETTING:

In this setting, we test the learned models using detections from a pre-trained object detector (taken from (Tulsiani et al., 2018) ). Thus, we can now evaluate the robustness of these methods when the Table 3 : We study the effect of combining the unary and relative predictions at various stages. The multitask model does not combine them, whereas the next two models combine them either at train or test time. Our method that jointly optimizes these predictions (and their combination) at both train and test time shows the biggest improvement.

input object boxes are not pristine. In Table 2 , we see the mean Average Precision values for different criteria used to define true positives (see appendix for metrics). As an example, in the first column a true positive satisfies IOU (box2d) threshold with the ground truth, is close in scale, rotation, translation, shape thresholds. Each subsequent column examines one criteria so we can analyze their accuracy with detections. We see (in the first column) that our method provides a significant gain of 6 points in mAP over the baseline. Relationship modeling (GCN and InteractionNet) provides benefit in the detection setting too. However, our structured reasoning outperforms these methods, with most of the gains coming from predicting higher quality scale and translation. We visualize some predictions showing the difference between our approach and the baseline in figure 5 . Furthermore, in Table 4 we evaluate the our method on the NYUv2 in the detection setting, and we achieve similar trends in performance with respect to the baseline. Due to increased difficulty of the task on NYU we observe relative lower mAP scores.

4.4. EFFECT OF COMBINATION AND OPTIMIZATION:

Combining the unary and relative predictions provides benefit at the training time because it allows the model to modify its unary and relative predictions so that they are more 'compatible' with each other. We quantify this in Table 3 using the SUNCG dataset. We report the performance of a purely multi-task version ('MT') that only predicts the unaries and relative pose values, without ever combining them. The 'MT + combine test only' combines these during inference (but not training). Finally, the 'MT + combine train only'. We also compare to a method that only uses this combination at train time, and finally report our full method as a reference. We see that combining at either train or test alone performs better than pure multi-task learning. This shows the importance of the unary and relative predictions interacting with each other. Combining these at both train and test time, and jointly optimizing like in our method performs the best. We proposed a method to incorporate relationship based reasoning in the form to relative pose estimates for the task of 3D scene inference. While this allowed us to significantly improve over existing approaches that reason independently across objects, numerous challenges still remain to be addressed. In our approach, we only leveraged pairwise relations among the objects in a scene, and it would be interesting to pursue incorporating higher order relations. We also relied on a synthetic dataset with full 3D supervision to train our prediction networks, thereby limiting direct applicability to datasets without 3D supervision. Towards overcoming this, it might be desirable to combine our approach with parallel efforts in the community to use 2D reprojection losses (Garg et al., 2016) or leverage domain adaptation techniques (Hoffman et al., 2017) .

:



Figures and Tables:

Figure 1: (a) Approach Overview:We study the problem of layout estimation in 3D by reasoning about relationships between objects. Given an image and object detection boxes, we first predict the 3D pose (translation, rotation, scale) of each object and the relative pose between each pair of objects. We combine these predictions and ensure consistent relationships between objects to predict the final 3D pose of each object. (b) Output: An example result of our method that takes as input the 2D image and generates the 3D layout.

Figure 2: Approach Details:We use the instance encoder to create an embedding for each object (instance) in the scene. The instance decoder uses this embedding to predict a pose for each object independently. The relative encoder takes each pair of instances and their embeddings to output an embedding for the pair. The relative decoder predicts a relationship (relative pose) between the pairs. We combine these relative and perobject predictions to predict final pose estimates for each object in an end-to-end differentiable framework.

Figure 4:We visualize sample prediction results on the NYUv2 dataset in the setting with known ground-truth boxes. Our method produces better estimates than the baseline, e.g., for the top left image we see that the chairs are better aligned by our method as compared to the baseline. We use the ground-truth meshes for visualization due to lack of variability in shape annotations for learning.

Figure 5: We visualize sample prediction results in the detection setting. Our method produces better estimates than the baseline, e.g., in the first image the television set and the table are better placed. The colors only indicate separate instances, and do not correspond between the ground-truth and the predicted representations.



Table content: <table/>

1, our method generates higher quality predictions across both translation and scale where it can outperform the baselines on the mean, median and % error. It is

Table content: <table><tr><td>Image</td><td>GT</td><td>Factored3D</td><td>Ours</td><td>Image</td><td>GT</td><td>Factored3D</td><td>Ours</td></tr></table>

Median Mean (Err ≤ 30 • )% Median Mean (Err ≤ 0.2)% We use ground truth boxes at test time and compare the 3D prediction performance of the methods.

Table content: <table><tr><td/><td/><td colspan="3">Translation (meters)</td><td colspan="2">Rotation (degrees)</td><td/><td>Scale</td></tr><tr><td colspan="5">Dataset Median Mean (Err ≤ 0.5m)% SUNCG Method Factored3D 0.28 0.39 79.5 InteractionNet 0.28 0.37 80.0 GCN 0.26 0.38 79.3</td><td>4.56 19.91 4.58 20.19 4.60 20.45</td><td>86.4 86.4 86.0</td><td>0.16 0.11 0.11</td><td>0.25 0.19 0.20</td><td>58.4 68.6 69.1</td></tr><tr><td/><td>Ours</td><td>0.23</td><td>0.33</td><td>84.0</td><td>4.58 19.82</td><td>86.6</td><td>0.10</td><td>0.19</td><td>69.7</td></tr><tr><td/><td>Factored3D</td><td>0.49</td><td>0.62</td><td>51.0</td><td>14.55 42.55</td><td>63.8</td><td>0.37</td><td>0.40</td><td>18.9</td></tr><tr><td>NYUv2</td><td colspan="2">Interaction Net 0.45 GCN 0.45</td><td>0.59 0.60</td><td>56.2 55.6</td><td>13.34 38.7 14.22 41.63</td><td>67.6 65.7</td><td>0.36 0.37</td><td>0.39 0.40</td><td>20.1 19.2</td></tr><tr><td/><td>Ours</td><td>0.41</td><td>0.54</td><td>60.9</td><td>14.00 39.60</td><td>67.0</td><td>0.33</td><td>0.38</td><td>21.7</td></tr></table>

We report the mean Average Precision (mAP) values for the detection setting for SUNCG. In each column, we vary the criteria used to determine a true positive. This helps us analyze the relative contribution of each component (translation, rotation, scale) to the final performance.

Table content: <table><tr><td colspan="4">Under review as a conference paper at ICLR 2019</td><td/><td/><td/><td/><td/></tr><tr><td colspan="2">Method</td><td>all</td><td colspan="4">box2d + trans box2d + rot box2d + scale</td><td/><td/></tr><tr><td colspan="2">Factored3D</td><td>21.72</td><td>49.28</td><td>62.77</td><td>33.56</td><td/><td/><td/></tr><tr><td colspan="3">Interaction Net 26.42</td><td>50.26</td><td>61.75</td><td>41.66</td><td/><td/><td/></tr><tr><td>GCN</td><td/><td>24.76</td><td>48.63</td><td>61.61</td><td>39.97</td><td/><td/><td/></tr><tr><td>Ours</td><td/><td>27.76</td><td>54.38</td><td>62.72</td><td>41.83</td><td/><td/><td/></tr><tr><td/><td colspan="3">Translation (meters)</td><td colspan="2">Rotation (degrees)</td><td/><td>Scale</td><td/></tr><tr><td>Method</td><td colspan="8">Median Mean (Err ≤ 0.5m)% Median Mean (Err ≤ 30 • )% Median Mean %(Err ≤ 0.2)%</td></tr><tr><td>Multi-task (MT)</td><td>0.25</td><td>0.35</td><td>81.9</td><td>4.55 19.33</td><td>86.7</td><td>0.11</td><td>0.20</td><td>68.2</td></tr><tr><td>MT + combine test only</td><td>0.23</td><td>0.34</td><td>83.5</td><td>4.51 19.30</td><td>86.9</td><td>0.11</td><td>0.19</td><td>68.4</td></tr><tr><td>MT + combine train only</td><td>0.25</td><td>0.36</td><td>82.0</td><td>4.63 20.02</td><td>86.1</td><td>0.11</td><td>0.19</td><td>68.9</td></tr><tr><td colspan="2">MT + combine train &amp; test (Ours) 0.23</td><td>0.33</td><td>84.0</td><td>4.58 19.82</td><td>86.6</td><td>0.10</td><td>0.19</td><td>69.7</td></tr></table>

We report the mean Average Precision (mAP) values for the detection setting for NYU. Note that the threshold for scale is different from the one used in SUNCG Table2. Added after reviews 5 DISCUSSION AND FUTURE WORK

Table content: <table><tr><td>Method</td><td>all</td><td colspan="3">box2d + trans box2d + rot box2d + scale</td></tr><tr><td>Factored3D</td><td>5.30</td><td>17.17</td><td>20.36</td><td>28.36</td></tr><tr><td colspan="2">Interaction Net 7.57</td><td>19.92</td><td>20.39</td><td>30.93</td></tr><tr><td>GCN</td><td>6.49</td><td>17.39</td><td>21.85</td><td>30.42</td></tr><tr><td>Ours</td><td>8.49</td><td>21.16</td><td>22.81</td><td>31.91</td></tr></table>

Peer review (iclr2019):
 This paper proposed a 3D scene parsing that takes both objects and their relations into account, extending the Factor3D model proposed by Tulsiani et al 18. Results are demonstrated on both synthetic and real datasets. The paper is in general well written and clear. The approach is new, the results are good, the experiments are complete. However, I am still lukewarm about the paper and cannot champion it. I feel the paper interesting but not exciting, and it’s unclear what we can really learn from it. Approach-wise, the idea of using pair-wise relationship as an inductive bias is getting popular. This paper demonstrated that it can be used for scene parsing, too, within a neural net. This is good to know, but not surprising given what have been demonstrated in the extensive literature in the computer graphics and vision community. In particular, the authors should discuss many related papers from Pat Hanrahan’s group and Song-Chun Zhu’s group (see some examples below). Apart from that, this paper doesn’t have an obvious technical innovation that can inspire future work. This is different from Factor3D, which is the first voxel-based semantic scene parsing model from a single color image, with modern neural architecture. The results are good, but are on either synthetic data, or using ground truth bounding boxes. Requiring ground truth boxes greatly restricts the usage of these models. Would that be possible to include results under the detection setting on NYU-D or Matterport 3D? The authors claimed that the gain of 6 points is significant; however, a simple interaction net achieves a gain of 5 points, so the technical contribution of the proposed model is not too impressive. In general, I’m on the border but leaning slightly toward rejection, because this paper is very similar to Tulsiani et al, and the proposed innovation has been explored in various forms in other papers. A minor issue: - In fig 5. The object colors are not matched for GT and Factor3D and ours. Related work Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image. ECCV’18. Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars. IJCV’18. Characterizing Structural Relationships in Scenes Using Graph Kernels. SIGGRAPH’11. Example-based Synthesis of 3D Object Arrangements. SIGGRAPH Asia’12.