Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper introduces a complete pipeline to represent a scene with 3D model of moving objects and immovable background by a single image. The approach utilize NOCS-SDF, RGB-LF and surfel-based differentiable rendering to optimize the full geometry of movable object instances with the help of a object prior database. The decomposition of the scene enables the approach to edit the movable objects and re-render the scene.
strengths_and_weaknesses: Strengths
- The idea decomposing the scene to dynamic objects and static scene is quite commonly exploited. However, the representation of the object this paper developed has a good geometry representation. It can also recover the surface of object based on a prior object database. The combination of SDF and LF MLP makes sense from the perspective of theory and shows promising result. 
- This paper introduces the surfel-based differentiable renderer into this field for the first time. Comparing with NeRF, the surfel-based differentiable renderer is more geometrical consistent. It will not collapse when you generalize this approach to unseen scene.

Weakness
- The effectivenes of transformer-based feature extractor needs to be validated. Usually transformer-based architecture requires more data to keep the same performance as CNN. Is the transformer-based feature extractor really better than CNN in this task? Also, it would be misleading to use CNN instead of Transformer in the Figure 1. 
- It is not fair to compare this method with NeRF since NeRF deals with novel view synthesis. The fundamental principle of NeRF and this approach have a large discrepancy. It would be fair to compare it with monocular depth estimation with editable moving objects, which shares the same representation of this approach. You can compare different object representation under the framework of monocular depth estimation.
- The details of object reasoning block are missing in the paper and supplementary material.
- The authors claim this approach can apply to real data without domain adaptation. It would be nice if a quantitative result of the performance difference between VKITTI dataset and KITTI dataset.
summary_of_recommendation: This work solve the monocular scene reconstruction by scene decomposition of dynamic objects and static scene. By introducing NOCS-SDF and RGB-LF based optimization and surfel-based differentiable render, the novel representation and rendering of these movable instances makes this approach generalizable to unknown scene (but not for unknown objects). The theory of this approach is solid and the overall design is reasonable for robotics application. 
issues: - Will the transformer-based feature extractor harm the performance of the depth estimation module? We have not seen many literature addressing this issue, and you are actually sharing the representation with two tasks: Depth and NOCS. A ablation study which shows the advantage of transformer-based feature extractor will be nice to have.
- Why this approach can apply to real data without domain adaptation? Do you mean by similar scene like the example you showed in the paper: VKITTI and KITTI? If the scene changes, will it still be able to have a strong sim-to-real performance without domain adaptation? 
- The missing loss functions in the Section 3.2 can make the reader without related background knowledge hard to understand.
- Can we have a way to show the advantage of partial shape recevery by quantitative analysis? This seems a crucial step to obtain a good final object reconstruction with differential render.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a holistic single-shot scene reconstruction system that separately recovers the depth of the background and the 3D geometry and appearance of the objects.
strengths_and_weaknesses: Strength:
- Results are strong.
- It's an interesting idea to use SDF features to encode the prior dataset and fit the reconstructed partial object with the learned SDF decoder.

Weakness:
- The structure of section 3.2 is very messy:
  - You have to read 3.2.1 and 3.2.2 before you can understand the first part of 3.2. Because this part is not high-level enough and many unexplained terms appear in this part.
  Please briefly introduce the terms like PriorDB feature, 0-isosurface projection, and Lsymm loss.
  - 158: The distance between which point clouds? Is the ground truth full point cloud of the objects available? Or partial point clouds are compared?
- 207: Fig 5 does not have an "a
- It makes sense to fit SDF of prior shapes because cars' shapes are close. But the luminance field can be very different across the shapes. For example, in Figure 5, the rendered color of the second car on the right side of the road is orange, while in partial reconstruction, it's white.
- No ablation study is conducted. For example, w/ or w/o the 3 components in 2D alignment. 
- Few qualitative results are shown in paper or supplementary material. No qualitative results of novel view synthesis of PixelNeRF and the proposed methods are shown, which makes the discussion not convincing.
summary_of_recommendation: The idea is pretty cool. But the lack of qualitative results and the poor writing harms the quality of the paper. So I recommend rejecting the paper. But if the author can give a concrete rebuttal, I'll definitely reconsider.

Update: The authors' responses address my questions. So I would like to change my recommendation to 'weak accept'.
issues: As stated in weakness.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method for reconstructing, and decomposing a scene from a single RGB image. The main idea is to use a decomposition network to simultaneously predict normalized object coordinates a depth map of the background. The object poses can then be easily computed using PnP and refined using a database of object models and an optimization procedure. The results show that the proposed approach generalizes well and achieves excellent performance on view synthesis, depth estimation and object detection in this domain.

strengths_and_weaknesses: **Strengths** 
- The paper is very well written from start to finish. It is clear that the authors have spent a lot of effort on writing and polishing the paper. The paper is easy to follow and describes the method in sufficient detail (there is, however, some information that is missing related to the data used, see below).
- The setup of the problem is very interesting and is the way I think perception problems should be addressed. By this I mean instead of just doing black-box depth prediction / object detection, the scene is decomposed in a more “symbolic” manner and the combination of each of these elements helps to describe the overall scene. This allows the method to be used for multiple tasks at the same time in an interpretable manner which is not often the case with deep learning approaches.
- The evaluation is targeted and supports the claims made in the paper. In particular cross-domain depth estimation is tested, 3D object detection accuracy is tested and view synthesis is tested.
- The results are good both quantitatively and qualitatively. The supplementary video showing the fully reconstructed scene and cars is quite impressive (however, it would be nice to see more examples).
- The method shows great generalization capability. It is entirely trained on synthetic images (Parallel domain and VKITTI2) and is able to generalize well to real KITTI and DDAD. 

**Weaknesses**
- The approach works only for a single class of object. For each new class, a database of object models is required that have been placed in a canonical frame of reference. This is ok for objects that don’t vary much in shape or appearance (like cars) but won’t work well for general objects. Of course, in the scenario autonomous / assisted driving this is not too much of an issue. However, it does limit the application to other areas.
- There are not many example scenes shown in the results. There are only two individual frames in the supplementary video and one in the main paper. Since the paper addresses the problem of view synthesis it would be good to see a more diverse range of reconstructions to get a better idea of the quality. 
- There is not indication of the runtime of the method and how stable it is when applied to video prediction (which is how it would probably be applied in the autonomous vehicle / ADAS case).

summary_of_recommendation: This is a good paper in the sense that it presents an interesting method and achieves some solid results. However, the weaknesses I’ve considered in the recommendation are the fact that it will be difficult to adapt to other setups (apart from road scenes), and the paper limited examples showing how the approach works.

issues: I would like the authors to address the following in the rebuttal (or final revision):
- How accurate are the different stages of the pipeline? I.e. how does the accuracy of the PnP compare to the final optimized result?
- How is the scale factor determined at testing time, is there a fixed scale for each of the 27 database objects?
- It would be good to provide more qualitative examples to show the performance of the approach across different scenes. When does the method fail? And when does it work really well?
- How would the approach work with vehicles of other shapes like buses, vans or bicycles?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presented a new single view scene reconstruction method that can decompose the scene into the background and multiple objects (cars in the shown demo) and recover their full geometry and appearance. It also involves an optimisation step to refine the geometry, appearance and poses of the detected instances based on NOCS-SDF and RGB-LF fitting with a surfel-based differentiable rendering method. The quantitative evaluations show SOTA performance in terms of depth evaluation and 3D detection. It furthers shows that it can achieve scene manipulation and generation to enable novel view synthesis. 
strengths_and_weaknesses: This paper presents a novel single-view holistic scene reconstruction/composition method. The strength is that it can recover the appearance by adding an extra latent embedding for LF. It also contains an optimisation capability using a surfel-based differentiable rendering approach, which makes the method applicable to unseen environments. 

In terms of the weakness, there are a few points.
1. Comparison: In the related work, the authors perhaps missed some recent work in the single-view scene reconstruction that can also recover the full object geometry. Such as "Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image" (CVPR 2020), "Holistic 3D Scene Understanding from a Single Image with Implicit Representation" (CVPR 2021), and some others. The authors might need to add these work in the related work part and show how their work is positioned against those work. The mentioned works can also recover the room layout and generate multi-class instances. From my perspective, the strength of this presented paper is that it can also recover the full appearance of the detected instances. However, this point is not evaluated or demonstrated in the experiment part. This innovation needs a thorough evaluation. 

2. Explanation of method: Section 3.2 needs a more clear explanation of the losses used for optimisation, for example, z_{sdf}, RGB loss, L_{symm}, L_{{NOCS_2D}}, L_{LF_{2D}}, L_{mask}, pose optimisation, normal distances. The equations for those losses and how they are summed under different circunsmaces are needed to be able to reproduce the results. 

3. Evaluations: Some qualitative results are needed to demonstrate the numbers in the view synthesis. Besides, as the pose estimation is an important step in the pipeline, it is expected to see some evaluations in the camera and object pose estimation. Also, the object and scene 3D reconstruction should also be evaluated. Since the aim is to recover the full geometry, it is expected also to evaluate the unseen part, not just the depth evaluation for the seen surface. 

summary_of_recommendation: The presented paper shows some new ideas and novel results. However, it requires extra evaluations to prove its innovation and also needs more clear explanation in its method part. This makes me on the borderline (tends to weak accept). The final recommendation will depend on if the mentioned concerns can be solved.
issues: In addition to some issues mentioned in the weakness part:
* L150: Fig.5a is not the figure referred does not include object prior database.
review_title: null

Meta-review: All reviewers received this paper positively. Both presentation and results were reported as convincing, however, the ratings were borderline (1x weak reject, 2x weak accept). The main criticism relates to the lack of qualitative results (all reviewers), evaluation shortcomings (pv1C: ablation study, JgCa: runtime and stability analysis, jaEn: accuracy of camera and pose estimation) and presentation (pv1C+jaEn: section 3.2). Given the authors revise the manuscript accordingly, I’d be happy to accept the paper. 
Decision: Accept (Poster)