Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper introduces a new model for obtaining object-level 3D scene representations from images. The proposed method models 3D scenes by parsing objects one-by-one into structured representations of shape, texture, and poses. A differentiable renderer incorporated in this pipeline enables self-supervised training as well as test-time image generation.

The method is interesting and novel, though general ideas are borrowed from Park et al. 2019, Sitzmann et al. 2019 and Wang et al. 2020. The paper is well-written and easy to follow for the most part. Also, the experiments performed have supported the claims from the authors. 

Pros: 
- This is the first work to jointly learn the tasks of object instance detection, instance segmentation, object localization, and inference of 3D shape and texture in a single RGB image through self-supervised scene decomposition.
- The idea of incorporating a renderer into the parsing pipeline is interesting and meaningful. In this way, the method is able to perform self-supervised decomposing of the scene.
- The authors perform extensive experiments and ablation studies on the clevr and ShapeNet dataset, which have proved the effectiveness of the proposed pipeline on these datasets and provided useful information on designing such kind of pipeline.

Cons:
- Although the authors provide preliminary results on real-world images, the extension of this pipeline to real-world scenes is doubtful:
  - The number of objects is fixed/given for the framework; the increasing number of objects in complex scenes leads to more ill-conditioned problems, which will make the self-supervised decomposition extremely hard.
  - Scene conditions (e.g. light) are not modeled or overly simplified (e.g. background) within this pipeline, and the differentiable renderer used (or even the sota differentiable renderers) are not able to sufficiently model complex real scene conditions.
  - It would be appreciated if you could include more complex shapes in real-world images. For example chairs/toy cars as in the ShapeNet.

Minor Comments & Questions:
- The `Change Shape` example for ShapeNet in Figure 1 does not seem to be obvious. 
- Figure 2 is not clear enough at first glance. You can add some texts in Figure 2 to make it more self-contained and understandable, e.g. "g_O: object encoder", "g_{bg}: background encoder", etc. Also, the `Top` and `Bottom` is clearly separated. You should consider to split and reorganize these two figures.
- The method section is slightly verbose and can be better organized.
- Is it possible to render new views given the parsed scene representation? This seems to be possible but not mentioned in the paper.
review_title: Interesting idea for scene parsing; more discussions on real-world use needed

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors proposed a method for 3D scene inference, which jointly does object instance detection, instance segmentation, object localization, and 3D shape and texture inference. The authors designed an autoencoder-like network such that it can be trained in a self-supervised way from RGBD images. Some limitations:
- The authors claimed they are the first to do these tasks jointly, and they did not compare their method to any existing work, but only tested on the Clever and ShapeNet dataset. However, there are existing depth/normal estimation algorithms which can serve as baselines.
- It is not justified the superiority of the design of the method. Did it model the relationships between objects? Why not using an object detector combined with a 3D shape/texture reconstruction neural network?
- The authors did not test their method on real data. Only tested on the Clever and ShapeNet dataset.
- The pipeline figure (Figure 2) is very difficult to understand. Might be better with more text illustrations in the figure, not only notations.

---Post-Rebuttal---

Thank the authors for their response. I now agree with the authors and other reviewers that the authors' approach has its novelty (self-supervised, rendering, etc.), and the ablation study in Table 1 is reliable to prove each component is useful. Therefore, I increase my rating by 1.
review_title: Interesting work on 3D scene inference but has space for improvement

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Weaknesses

- All experiments are synthetic. Although synthetic experiments are common in the field, I see this as a weakness because the method heavily relies on image reconstruction loss on rendered objects. Real scene are very different due to both high level (more clutter, occlusion, number of objects) and low level (shading, less predicable texture) differences from the examples used in the synthetic experiments and will likely require different constraints to solve. The model has access to dense ground truth depth maps too, which is difficult to obtain. The motivation for self-supervised learning is to take advantage of unlabeled in-the-wild images, so considering all of those things, I think the synthetic RGB-D assumption hurts this paper more than an average paper.

- There's much to learn from the object count experiment (Table 1 and Page 6), but I found the results confusing. According to Page 5 which says "each dataset contains scenes with a specific number of objects which we choose from two to five", the network was trained to model a specific number of objects separately in each experiment. If you evaluate a three-object trained network on images containing two objects ("obj=3/2" in Table 1, 10th row), one would expect it to suffer from false positives, but the quantitative results do not show this. The results are almost the same as "obj=3/3". Why is this? Did it learn to solve this problem, or were they somehow not included in the evaluation? I think there's some resemblance to EM, and it would matter a lot if you knew the number of true clusters beforehand, and how over or under-estimations are handled, in understanding the model's behavior.

- Lack of basic baselines. Currently there's no way to know how difficult this dataset is, and how well a fully supervised model would perform.

-  Potentially missing details, additional questions, and suggestions:
     - In RGB reconstruction loss and evaluation, do you also count the background, or is this on ground truth object region only?
     - Page 2 says "By representing 3D geometry explicitly, our approach naturally respects occlusions and collisions between objects and facilitates manipulation of the scene within the latent space." This part was confusing to me. How does it avoid collision of objects in 3D? My understanding is that this is not part of equation 1.
     - Figure 2 is confusing. I think a few captions in the image would have helped a lot.
     - In Table 2, would you attribute the gap between seen and unseen instances more to the shape, or the texture of the object?

- One suggestion is, would you be able to render the scene from a novel viewpoint, since you know the depth, shape and pose of each object? And then evaluate against ground truth rendering from that viewpoint? Being able to do such an experiment could make the synthetic setting more useful.  

### Strengths

- Ablation study in Table 1 shows that all of the loss terms are important. This justifies the design and I would consider it an insightful discovery.  For example, the ground truth depth already globally provides sharp object boundaries, but in order to reproduce them, both shape and ground plane loss terms were needed. So the model provides the right constraints in which all of those factors are considered.

- "How do you turn single-object 3D reconstruction models into priors, for scene-level inference?" is still a very open question, and more research is needed in this area. The fact that this methods requires such a prior could be a weakness to someone who prefers to see less assumptions made in favor of generalizability. But I believe objects have good enough scientific foundation to be beneficial in the long run.

### Justification of rating

Despite the weaknesses I listed, I feel that this paper is above threshold, albeit marginal. Other researchers can still learn from the design choices made even if it turns out that those constraints don't work well in real scenes. I understand that this is a difficult task, and having a synthetic model as a reference point would ultimately help, assuming the code for reproducing the experiments will be fully released, to serve as a baseline.
review_title: Review of paper #794

Meta-review: The paper proposes learning of 3D object representation from images. The pretraining used assumes it can generate implicit 3D models for the objects, and then objects are detected in multi-object scenes without further supervision. Reviewers raised concerns regarding experiments being conducted only on synthetic data. Authors are encouraged to try out their approach on real data, to demonstrate the benefits of their solution.
Decision: Reject