Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a way to perform panoptic 3d scene reconstruction from a single image. Depth and instance segmentation from an rgb image help create a 3d volume through back-projection which is subsequently refined to produce semantics, instances and 3d reconstruction, which effectively translates to a panoptic representation. 


main_review: Strong Points
- The authors are able to perform the hard task of panoptic reconstruction from a single image
- The method works better than SSCNet and Mesh R-CNN

Weak Points
- The evaluation of the method is severely lacking. As the authors produce 3d reconstruction and semantic segmentation in addition to instance ids, there should be proper evaluation for both these outputs against state of the art. 
- I understand that no other method does panoptic 3d reconstruction. However, the authors should 'downgrade' panoptic segmentation and compare with other semantic segmentation methods as opposed to 'upgrading' an existing method to produce panoptic labels. This will avoid bias and validate the usefulness the proposed approach at a more modular level. 
- Several relevant approaches have not been discussed and compared against. For e.g., CoReNet by Popov et. al. should be compared for the task of 3d reconstruction and 3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior by Chen et.al. is a better and more recent baseline compared to SSCNet. 

limitations_and_societal_impact: The authors discuss limitations and societal impact of the work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper defines and presents a method for the novel task of panoptic 3D scene reconstruction, i.e. jointly predicting complete 3D scene geometry, 3D semantic and 3D instance labels. The method consists of using backbone 2D models for depth and instance segmentation and lifting them into 3D before processing with a SG-NN [5] like model to complete and refine the initial volume into instance, semantic and occupancy/TSDF volumes. The authors propose an "instance propagation" method to associate 2D instance masks with 3D instances which is shown to improve over a post hoc "instance clustering" in a 3D semantic only volume. Experiments on a synthetic dataset (3D Front) and Matterport3D show improvement over prior works , some of which tackle a subset of the problem (e.g. only scene structure and semantics). 

The primary contribution of this paper is the definition of this new holistic panoptic 3D scene reconstruction task and a new method and evaluation metric for the same with improved performance over prior works which tackle these individual tasks in isolation.
main_review: *Strengths*
- 3D scene structure, 3D semantic and 3D instance segmentation are critical to a holistic understanding of the scene and have been seeing a lot of progress in recent years. Similar to the 2D analog of panoptic segmentation, 3D panoptic reconstruction is the next logical step in this evolution and this paper helps define it with metrics and a method for it. This is quite an important contribution in my opinion.

- The paper is well written and the illustrations make it easier to understand the paper (e.g. Fig 3 helps explain the instance propagation idea well). The visual results are quite impressive.

- The comparisons against prior work (Table 1 and 2) have been carefully done with proper explanation of the alternative used when an existing system wasn't directly applicable in this setting (e.g. input depth for SSCNet, Total3D requiring canonical poses for objects).

*Weaknesses*
- One concern I have with this paper is with technical novelty. At the surface it seems like a simple combination of existing depth + instance segmentation algorithms combined with prior work for scene completion (SG-NN) with extension to semantic and instance label volumes. The instance propagation idea is slightly novel but perhaps not enough. However, defining the 3D panoptic scene reconstruction task and proposing a metric for it could be considered a significant enough contribution.

*Clarifications*
  - L120 - How are multiple feature map from different spatial resolution from the ResNet concatented into $\theta$?
  - Why was the choice to randomly select a pixel for a voxel with multiple backprojections made? Why would you not use standard bilinear sampling for the same?
  - In Table 2, Mesh R-CNN seems to show significantly better performance on "Things" over this method. What is the primary reason for this?


limitations_and_societal_impact: I believe the authors have addressed some limitations in Sec 6.5 of the paper. I would appreciate responses to the questions mentioned in the clarifications section above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors define a new problem called "panoptic 3D scene reconstruction" - a joint task of 3D geometry prediction, semantic segmentation, and instance segmentation, from a single RGB image. To address this new task, a two-stage method is proposed. In the first stage, 2D CNN is used to predict 2D features, a depth image, and 2D instance segmentations. In the second stage, these 2D features and instance segmentation results are back-projected into 3D, and then passed into a 3D CNN for the final 3D predictions. This proposed pipeline is demonstrated on the synthetic 3D-front dataset, as well as the Matterport3D real-world dataset, and significantly outperforms prior related work.
main_review: Originality

The newly defined "panoptic 3D scene reconstruction" problem is a natural extension of 2D panoptic segmentation. The components used in the proposed pipeline, including reusing 2D features, uplifting 2D features to 3D via backprojection, multi-task joint training, and 3D CNNs, are all well studied in the community. 

Quality 

Based on the evaluation metrics, the proposed method significantly outperforms related prior work in terms of geometry, semantic segmentation, and instance segmentation. The example predictions also seem to align better with the ground truth. However, the experimental section is relatively simple, and I do have questions about the proposed method and some suggestions regarding ablation studies. 

1. Based on Section 4.1, it is my understanding that the 2D network and the 3D network are trained sequentially and separately, rather than jointly. Could you elaborate on the reasoning behind this design choice? Does joint training lead to sub-optimal results?
2. The current evalutions are all based on 3D volumetric representations, but not 2D. When the 3D predictions are rendered back into the image space, how is the quality of depth, semantic mask, and instance segmentation? Does the 3D network improve the quality, or decrease it?
3. According to Section 4.1, the real-world model is pretrained on synthetic data and then fine-tuned on Matterport3D. Is this the same training procedure for the prior work for a fair comparison? Is there additional effort to make sure that the prior method (e.g. Mesh R-CNN [13]) is trained properly on these datasets and is indeed achieving their optimal performance?
4. The comparison against SSCNet [31] seems a bit strange, given that the method expects a single (good) depth image as input. A more fair experiment would be to use ground truth depth in both SSCNet and the proposed method.
5. After reading the paper, it is still unclear to me why there has been a significant improvement in the metrics compared to prior work - is it because of joint task learning providing additional supervision and task-shared feature representations, or is the proposed network architecture far better than prior related work? Please elaborate and consider adding ablation studies to support your arguments.  
6. On Line 127, viewing directions are also used as input - does this help improve quality?

Clarity 

The paper is straightforward and easy to follow.

Significance

I believe panoptic 3D scene reconstruction, and more generally multi-task prediction in 3D, is an important topic in the computer vision / machine learning community. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an approach for panoptic 3D reconstruction that combines, in an end-to-end trainable manner: (i) 2D depth from rgb, (ii) mask rcnn-style instance segmentation and (iii) a final refinement stage to produce the overall final result.
main_review: Positive:
* While I am somewhat uninformed about the state of the art, this does appear to be the first paper to propose an end to end trained panoptic 3D reconstruction (/ stuff + instance segmentation) solution. 
* The paper is well written and easy to understand.

Negative / Questions:
* The key components of the approach are not entirely novel or special, with the various bits of the system being used before in previous tasks (eg the standard-ish depth generation and 2D instance segmentation, traditional-ish lifting, sparse + dense Unet as used before in shape completion).
* The overall structure is also somewhat predictable (except for, maybe, the sparse + dense unet component).
* Literature review seems rather brief, especially in the single view 3D reconstruction section, eg CoReNet and Points2Objects (and related) seem to be missing, though would be quite relevant.
* A version of Figure 1 + 2 from the supplementary should be in the main paper, since it aids quite a bit in understanding the proposed method.
* A benchmark / ablation of sparse vs dense 3D CNNs at the start of the network could be added.
* Given the bottoms-up nature of the approach, a baseline of eg backprojected / lifted 2D panoptic segmentation would seem like a natural approach to investigate / compare against. Is this included in the paper?

Overall, while I do think the solution is rather simple (and reads a bit like a concatenation of previous papers), I am for acceptance because of the novelty of the task, the good results and the quality of the writing. 
limitations_and_societal_impact: -
review_title: null

Meta-review: The paper proposes a method to learn panoptic 3D scene reconstruction from a single image. Depth and instance segmentation from an RGB image instantiate a 3D feature volume through back-projection which is subsequently refined to produce semantics, instance segmentations and 3D reconstruction. All reviewers acknowledge the good empirical performance of the method and the high  relevance of the problem tackled for the NeurIPS audience. 
Concerns were raised regarding comparisons of the paper to previous methods that did not perform jointly the tasks of segmentation and 3D reconstruction, which the rebuttal addressed. Authors are encouraged to include all additional experiments in the final version.

Decision: Accept (Poster)