Panoptic 3D Scene Reconstruction From a Single RGB Image
Authors: Manuel Dahnert
Ji Hou
Matthias Niessner
Angela Dai
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3d reconstruction
3d from a single image
3D scene understanding

Abstract: Richly segmented 3D scene reconstructions are an integral basis for many high-level scene understanding tasks, such as for robotics, motion planning, or augmented reality. 
Existing works in 3D perception from a single RGB image tend to focus on geometric reconstruction only, or geometric reconstruction with semantic  segmentation or instance segmentation.
Inspired by 2D panoptic segmentation, we propose to unify the tasks of geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation into the task of panoptic 3D scene reconstruction -- from a single RGB image, predicting the complete geometric reconstruction of the scene in the camera frustum of the image, along with semantic and instance segmentations.
We propose a new approach for holistic 3D scene understanding from a single RGB image which learns to lift and propagate 2D features from an input image to a 3D volumetric scene representation.
Our panoptic 3D reconstruction metric evaluates both geometric reconstruction quality as well as panoptic segmentation.
Our experiments demonstrate that our approach for panoptic 3D scene reconstruction outperforms alternative approaches for this task.

Panoptic 3D Scene Reconstruction From a Single RGB Image
Manuel Dahnert
Hou
Matthias Nießner
Angela Dai
Technical University of Munich
Abstract
Understanding 3D scenes from single image is fundamental to wide variety tasks, such as for robotics, motion planning, or augmented reality. Existing works in 3D perception from a single RGB image tend focus on geometric reconstruction only, or geometric reconstruction with semantic segmentation or instance segmentation. Inspired the tasks of geometric reconstruction. 3D semantic segmentation, and 3D instance segmentation into the task of panoptic 3D scene reconstruction from single RGB image, predicting the complete geometric reconstruction of the scene in the camera frustum of the image, along with semantic and instance segmentations. We thus propose new approach for holistic 3D scene understanding from single RGB image which learns lift and propagate 2D features from an input image 3D volumetric scene representation. We demonstrate that this holistic view of joint scene reconstruction, semantic, and instance segmentation is beneficial over treating the tasks independently, thus outperforming alternative approaches.
2D panoptic segmentation, propose unify
Introduction
3D scene understanding from single RGB image is fundamental to many downstream applications, such for robotics, motion planning, or augmented reality. From a photograph of scene, one can infer the underlying geometric structures and identify object and structural semantics, even from such a partial, single view observation. Such an understanding is also fundamental towards enabling higher-level interactions with environments, as estimating unseen geometry and semantics without having to observe all parts of an object or scene is paramount for tasks such as robotic interactions, augmented reality, content creation.
Recently, significant advances have been made towards complete geometric reconstruction from an RGB image input [10) (37), as well as reconstruction of object instances detected in the image however, 3D semantic reconstruction and instance reconstruction have largely been considered independently. We thus aim to unify the tasks of geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation from a single image, and inspired by the 2D panoptic segmentation task [27]. we call this task panoptic 3D scene reconsiruction. Inspired by the unification of the 2D semantic segmentation and instance segmentation tasks to panoptic segmentation we similarly adopt joint semantic segmentation of "stuff" and "things." where "stuff" elements do not have any distinct instances (c.g., structural elements such as walls and floors) and "things" denote semantic objects with instance ids defining the distinct object instances. Thus. for panoptic 3D reconstruction, we aim to predict the surface geometry of the scene from an image, including any missing regions from occlusion, and for each point the predicted surface geometry. it must be assigned a semantic label and an instance id. To evaluate a predicted panoptic reconstruction, we then evaluate both the segmentation quality and recognition quality of the 3D object instance geometry and geometry corresponding to structural semantic labels.
15
28 32
227.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Input RGB Image
Panoptic Reconstruction
Geometry
Instances
Semantics
Figure 1: We propose the task of Panoptic 3D Scene Reconstruction from single RGB image. From an RGB image input, we simultaneously predict geometric reconstruction, semantic labels and object instances, and combine these to form panoptic reconstruction output.
We believe that this task not only brings together geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation, but also introduces new algorithmic challenges, and hope that the panoptic 3D scene reconstruction task leads to additional insights in holistic 3D perception.
To address the panoptic 3D scene reconstruction task, propose new method jointly reconstruct and segment the observed 3D scene from a RGB image. We first extract features from the image with 2D convolutional backbone which predicts both depth estimates and 2D instance segmentation. We use the depth estimates to guide the feature lifting to 3D by projeprroe the learned features into 3D volume of the camera frustum, and develop instance propagation approach effectively leverage instance predictions in 2D as seeds for propagation to 3D geometric instances, providing effective object recognition. In contrast prior approaches for 3D reconstruction from 2D images, we develop propagation-bas based approach for object recognition, explicitly bringing 2D predictions 3D for refinement rather than direct 3D prediction on coarse resolution than the 2D signal. We train the object recognition jointly with reconstruction loss on the predicted scene geometry as well as semantic "stuff" labels, resulting in a holistic 3D semantic scene estimate for RGB image.
In summary, our contributions are:
We introduce the task of panoptic 3D scene reconstruction, which aims for holistic 3D scene understanding by jointly predicting scene geometry, semantic labels, and instance ids.
We propose a new approach for panoptic 3D scene reconstruction by learning to lift 2D features 3D while propagating learned instance information 3D object understanding for a robust joint prediction of scene geometry, semantics, and object recognition
Our method significantly outperforms alternative approaches which treat reconstruction and semanties separately and/or infer only subset of the outputs.
2 Related Work
Panoptic Segmentation. Kirillov et al. proposed the task of panoptic segmentation for 2D images, establishing a unified, holistic scene understanding task for image understanding and panoptic quality metric for evaluation. The task of panoptic segmentation of single color image is to assign each pixel in the image with an instance label and a semantic label. The semantic label set is composed of "stuff" and "things," where "stuff" labels do not consider instance ids as all pixels belonging to the same "stuff" class belong to the same instance, and "things" labels consider instance ids to distinguish different object instances. This has achieved notable success in inspiring further holistic analysis of image understanding, and we believe such holistic analysis also plays important role in 3D perception. We aim to bridge together geometric scene reconstruction 337
10
3D semantic segmentation 7 35), and 3D instance segmentation 440 along with a panoptic 3D scene reconstruction evaluation to capture these characteristics. Narita et al. [3I] proposed an approach for joint semantic and instance segmentation 3D reconstructed scenes, but rely on given 3D scene information without inferring geometry. To the best of our knowledge, work is the first to bring the panoptic reconstruction task into 3D perception from RGB images.
20
42
12
26
41
29
'Code be found at https: /ttp://gttttbbtt http:///..comm/mho//aaeeeeeeeeee http://tth....ooooooooooottttttt

---

3D Semantic Scene Completion and Semantic Instance Completion. Several recent works have explored the task of 3D semantic scene completion, in which both complete scene geometry and semantic segmentation are predicted from partial observations of a scene. From a single RGB-I frame, SSCNet (38 predicts the geometric scene occupancy and semantic class labels in volumetric grid. [3] follows up on it and proposes to guide the semantic scene completion with coarse 3D) sketches of the scene, ScanComplete [8] presented an autoregressive approach for large-scale sean completion and semantic segmentation. Dahnert et al. [6] proposed to retrieve complete objects of partially scanned objects from a shape database by constructing a joint embedding space of scan and CAD objects. Hou et al. [21] and Nie et al. [33] then focused on joint scan completion with instance segmentation of the predicted complete geometry. These approaches have shown the synergies in joint geometric completion along with semantic segmentation instance segmentation, and we aim to unify all these tasks together for a holistic approach to 3D perception from a single RGB image input.
Single-Vier View 3D Reconstruction. Estimating object or scene geometry from only single RGB image view has been long-studied, with recent advances deep learning showing significant potential in the reconstruction of shapes from single views 4 16 39 30]. As intermediate representation
Huang et al. extend 2D bounding boxes to 3D via joint estimation [23] or perspective constraints [25). Huang et al. (24] proposes layout to be optimized to fit a single RGB input image. Shin et al. [37 and Denninger et al. [10] recently proposed methods to estimate the scene geometry within the camera frustum of an image view. Recently, Mesh-RCNN [15] pioneered an approach to extend single object reconstruction to RGB images with multiple objects by detecting and reconstructing the object geometry of each detected object in the image, along with their relative orientations. Kuo et al. (28) similarly predicted the object geometry for detected objects in an RGB image, but by CAD model retrieval rather than generative mesh model. CoReNet (34) omitted explicit object detection and instead extracted 2D) information via physically-based ray-traced skip connection between the image and the 3D volume. The approach reconstructs the shape and semantic class of multiple objects from single RGB image directly in 3D volumetric grid. Nic
grammar, which allows to retrieve CAD models and estimate scene
[32] then proposed an approach detect object poses, their geometry, and cuboid room layouts. Similar to Nie et al., we aim to more comprehensively understand both object and scene structures and semantics, but remove any lifting 2D features and instance information to 3D for prediction of both "stuff" and "things," and additionally propose new holistic evaluation metric to understand the full panoptic 3D scene reconstruction quality.
assumptions, instead
3 Method Overview
In the panoptic 3D scene reconstruction task, single RGB image is taken input, from which the scene geometry within the camera frustum is predicted, along with semantic and instance label for each surface location the predicted geometry. We introduce strong 2D features from the input RGB image. The features are further projected and propagated to 3D, in which the final geometry, semantic, and instance predictions are predicted.
method for this task, extracting
From an input RGB image, we leverage a state-of-ther the-tht-rt 2D instance segmentation network as a 2D backbone to extract features, with proxy losses for 2D instance segmentation and depth estimation. The predicted depth is then used to estimate a coarse 3D representation by back projeccinn the learned 2D features and predictions into sparse volumetric representation. This coarse 3D estimate is then encoded by 3D sparse convolutions, and a coarse-to-fine approach is used to predict high-resolution output sparse volumetric representation, with each voxel containing distance value representing the surface geometry as a distance field, a semantic label, and an instance id. In particular, we exploit the 2D instance predictions by directly propagating them to 3D to be refined, which provides a robust semantic 3D prediction.
4 Hybrid 2D-3D Panoptic Reconstruction
Learning 2D Priors. An input RGB image 7 is first encoded by a 2D backbone O; realized with a R8 [T8] architecture. To learn effective priors for the panoptic 3D scene reconstruction task, we employ proxy losses on (I) for both 2D instance segmentation and depth estimation. We
3

---

Input
2D
2D-3D
Panoptic Reconstruction
Depth
Genmetry
Buckprojoction
Instances
Features
Semantics
RGB Image
Instance
Instances
Propagution
Figure 2: Method overview: from single RGB image, extract 2D image features along with depth estimates and 2D instance mask predictions. We back- both learned features as well instance predictions to a 3D volume in camera space, thus enabling instance propagation to 3D. The initial 3D estimates are then spatially refined with a sparse 3D generative convolutional network while jointly predicting the scene geometry, semantic labels, and object instance ids within the camera frustum of the image view, including occluded regions. detailed architecture overview can be found in the supplemental material.
follow Mask R- R-CNN for the instance segmentation, predicting 2D object bounding boxes, class category labels, and instance masks from (I). For depth estimation, we use the depth estimation decoder of [22], which predicts depth from multiscale features extracted by θ, using losses on the logarithm of the depth errors, gradients, and normals. We then extract features from the output of each ResNet block in O as well as the decoder outputs, which are then concatenated to 0 to be projected to 3D.
Lifting 2D Features to 3D. We then lift the 2D features to 3D using the known camera intrinsics of the image and the predicted depth estimates. The 2D features 0, depth estimates, and predicted 2I) back-projected into 256 3D volumetric grid using the estimated depth, and encoded as a truncated signed distance field (TSDF) along the view direction, where positive values in front of the predicted surface, zero encodes the predicted surface, and negative values unseen regions. We use a truncation of t = 3 voxels, and thus use a sparse 3D representation.
instance segmentation
To spatially correlate learned 2D features to directly inform the 3D panoptic reconstruction, the image features 0 as well as instance segmentation mask logits corresponding to each depth estimate copied along the view direction within the truncation, resulting in sparse 3D feature volume F (Fa, Fi) as the concatenation the computed distance field Fa. projected image features Fy, and projected instance masks F.. Multiple pixels falling into the same voxel bin randomly sampled. The 1-D instance segmentation mask logits for each object of a maximum of detected objects arbitrarily associated with the channels of N + 1-channel volume, with the extra channel encoding 1 -hht encoding for voxels with no instance mask associations. This enables instance propagation from detected 2D object instances final refined 3D object "things," explicitly leveraging strong 2D priors which result in more robust panoptic reconstruction than relying only on distinction of objects from much coarser 3D representation than the information from the high-resolution 2D image.
Sparse Generative Panoptic 3D Scene Reconstruction. The panoptic 3D scene reconstruction is then predicted from the feature volume = (Fa, Fy, F.) with a sparse generative approach. Each of (Fa. Fy, Fi) are first separately encoded with sparse 3D convolutions to transform the different features characteristics into a common space. The resulting features are then concatenated as F and encoded together with a sparse 3D encoder using common sparse encoder O,, realized with UNet-style architecture [36].
Since the panoptic reconstruction should both refine the estimated depth as well predict new geometry for unobserved regions, θ, follows the same geometric structure as its input F*, spatially downsampling by factor of 4, while the panoptic reconstruction is predicted in coarse-to-fine fashion, starting with a coarse prediction from (F') using dense 3D convolutions to enable generation of new geometric structures, similar to the sparse generative approach of SG-NN O for geometric scene completion. The initial dense prediction is then decoded to sparse panoptic predictions each spatial hierarchy level, with separate prediction heads for geometric occupancy, semantic labels, and

---

instance ids, which are then used to inform the next hierarchy level for refined predictions. Similar to [9]. we also employ dense and sparse skip connections between the encoder and decoder. At the last hierarchy level, we additionally predicted refined surface geometry as a sparse distance field, along with semantic label and instance ids per voxel.
The final panoptic 3D scene reconstruction is then obtained by extracting surface geometry from the isosurface of the predicted distance field. Each location with distance the surface less than т, is assigned the semantic label from the 3D semantic segmentation head if that label corresponds with a "stuff" label, and otherwise assigned the semantic label and instance id of the predicted instance mask at the location. Empirically, we found the precedence for "stuff" labels to perform slightly more robustly than the reverse, and provide the nearest semantic and instance labels for any locations that may have not had any semantic or instance predictions.
Losses.
We train our panoptic 3D scene reconstruction with the following loss:
L = waaa + wiCi : Σ (w + waC' + w.Cb))
(1)
where La denotes the 2D depth estimation loss, L: the 2D instance segmentation, and at each hierarchy level h, a loss for geometry Ly semantic label L. and instance id C. Note that the loss is not computed for any regions of the 3D predictions that lie outside of the image view frustum.
For 2D depth estimation loss La, we follow the losses used by 22 using a log-encoded & loss on the predicted depth values along with cosine normal consistency loss and gradient loss to regularize the depth estimation. The 2D instance segmentation loss L, follows the standard Mask R-CNN [9] procedure, with losses on the regressed 2D boxes, class category labels, and binary masks.
Our panoptic 3D reconstruction is trained to predict geometry, semantic labels, and instance ids for sparse voxels within the truncation region. For the geometric loss Cy. each hierarchy level is trained to predict geometry occupancy using binary cross entropy loss, and the final hierarchy level also employs an l loss on predicted distance values determining the surface geometry:
C = BCE pred O²)) + L^== Spedd - S
where O denote the occupancy predictions and ground truth at the hierarchy level, and for the final hierarchy level h 0, S denotes the distance field representing the surface geometry. The occupancy predictions at each level are used to generate the sparse 3D structure used for the sparse 3D convolutions in the following hierarchy level.
Semantic labels are trained with a class hlass-weighted cross entropy loss C. for each hierarchy level h. 3D instance ids are predicted as binary masks corresponding to the detected objects encoded in the channels of F,. That is, each detected object informs an instance mask refinement and propagation from the initial projected 2D mask logits. The 21) object masks are correlated with ground truth 21) masks when the loU > 0.5. The objects are then arbitrarily ordered to enable a cross entropy loss for the 3D predictions, with the correlated ground truth masks assigned the arbitrary ordering. A visualization of our instance propagation loss is shown in Figure 3 This provides strong prior on 3D objects from the 2D predictions, from which only a refinement must be learned in 3D, resulting in more robust and accurate object recognition.
4.1 Training
We train our approach for panoptic 3D scene reconstruction on single RTX 2080 Ti. We first jointly pretrain the 2D encoder, depth estimation and 2D instance predictions with an ADAM optimizer using batch size of 8 and learning rate lc-4 for 500k iterations (+o 5 days). The learning rate is decreased by a factor of 10 after 250k and 350k iterations. We then train the 3D sparse generative panoptic reconstruction in coarse-to-fine fashion with a batch size of 1, with the hierarchy levels trained for 10k, 5k and 5k each iterations before the next hierarchy level is added to the training. The full hierarchy is then trained for another 300k iterations.
For training on real-world data, we use model pre-trained on synthetic data, and fine-tune the 2D predictions for 100k iterations with batch size of 8 and learning rate of of le-4 that is decayed after 50k iterations, and fine-tune the 3D hierarchy for 5k iterations for each hierarchy level and additional 185k iterations for the final predictions.
5

---

2D Masks
3D) Instances
One- Hot Instance Encoding
Randomized Assignment
Figure 3: Instance Propagation from 2D to 3D. We directly propagate instance predictions from 2D to 3D, enabling 3D refinement from high-resolution 2D signal which results more robust 3D object recognition. During training, each predicted 2D instance mask is matched with a ground truth mask if their loU> 0.5, with matching segments assigned arbitrary instance ids. The ids assigned to predicted 2D masks are then used to assign the mask predictions to the channels of an N + 1-channel 3D volume, where denotes the maximum number of detected objects. and assigned a random instance id which is one-hot encoded and propagated into 3D. This feature volume is then used for refinement in 3D, with the refined output mask logits per channel representing 3D instance masks.
For both datasets, the per-class weights for the dlas-wwigghtdd semantic loss are compute by the inverse trgetraafformed appearance frequency in the data. Semantic freespace is weighted by 0.001.
5
Data
To train and evaluate the task of panoptic 3D scene reconstruction, we consider both synthetic and real-world datasets with dense semantic and instance annotations.
3D-Front [4] is a synthetic 3D dataset of indoor scenes, containing 6,801 mid-size apartments with 18,797 rooms populated by 3D shapes from the 3D-Future [13] dataset. We 11 class categories, of which have instance-level segmentation, and wall and floor as structural "stuff." Excluding empty scenes with no objects, we use train/val/test split of 4,389/489/1 2202 scenes.
To generate image inputs for the panoptic 3D reconstruction task, we render 320 x 240 image views of the synthetic 3D rooms. We randomly sample rooms which contain at least one object, and from these rooms, we sample random 10 camera locations per room at 0.75m height above the floor. Views discarded if they do not contain at least one object in the central region of the image, objects lie within the 1-7 meter range, or the camera falls within or above an object. We additionally discard views where any rendered 3D objects comprise 200 pixels of the image. We use BlenderProc [II] to generate photo-realistic 2D images along with depth, semantic and instance information. Textures for structural elements were recently added 3D-Front but not considered here. This results in 96,252/1 1,20//2,, train/val/tess images.
To generate 3D ground truth, we use SDFGen [ to generate 3D volume at 3em resolution from the same view and cull anything outside of the view frustum. Semantic labels and instance ids stored for each surface voxel and are inherently consistent with the 2D image views, which were rendered from the 3D data. Note that we only consider the geometry of individual rooms, as predicting geometry semanties behind walls is highly ambiguous.
Matterport3D contains reconstructed RGB-D seans of real-world indoor environments, consist- ing of 90 buildings reconstructed from RGB-D frames. We use 12 class categories, the same 9 thing classes as with 3D-Front, and wall, floor, and ceiling as stuff classes. use the official train/val/test split of 61/1 1/18 scenes.
We use the originally captured RGB frames from the downward and forward looking camera angle as input, and use BlenderProc to render the corresponding 2D semantic, instance, region and depth information for each frame. Image views are discarded from training and evaluation if they contain less than 30% valid depth pixels and the overlap between the visible geometry and complete geometry least 20%. 3D supervision generated by volumetric fusion [5] at 3em resolution the depth frames which observe the region within the camera frustum of the input image. In order to avoid fusing ambiguous geometry, e.g. geometry behind walls, we associate each camera location with a Matterport3D region by checking for the intersection between the camera position and the region bounding box. If the camera intersects multiple bounding boxes, the one with the largest overlap with the camera frustum is selected. During training and evaluating we mask depth pixels which belong

---

to different region as the camera as these are not present in the ground truth data. This results in 34,737/4.898/8.631 train/val/test images.
6 Evaluation and Results
6.1 Panoptic Reconstruction Metric
Similar to the 2D panoptic segmentation quality [27]. we evaluate panoptic reconstruction quality (PRQ) as an average measure across the class categories, with PRO for category c:
Σ(SJ)TTP Io)
PRO
+ + + 0.5|
where TP, FP, and FN denote true positives, false positives, and false negatives for class respectively, and predicted segments are matched with ground truth via a voxelized loU overlap of 2 25%. A segment is considered the geometry belonging to the same object instance id for things categories, and all surface belonging to a particular stuff category. Note that both predicted and ground truth geometry outside of the camera frustum is not considered in the evaluation, and predicted and ground truth segments are matched by a greedy search for the maximum loU overlap. All non-matched ground truth segments are considered as false negatives, and all non-matched predicted segments as false positives. The PRQ for a class c can be seen as the product of reconstructed segmentation quality (RSQ) and reconstructed recognition quality (RRQ) for the class c:
PRQ = RSQ x RRQ = (..)) ( x TP + 0.5 FP + FN5
TP
TPP
We evaluate panoptic reconstruction by voxelizing predicted panoptic surface meshes resolution of 3em for synthetic data and 6cm for real data. This captures the accuracy of predicted geometric structures well as semantic classes and instance segmentation. We additionally evaluate these metrics for only things classes and only stuff classes.
6.2 Comparison to State of the Art
We compare our approach to state-of-the-an alternatives for the task of panoptic 3D scene reconstruc- tion: SSCNet [3]] which predicts geometric occupancy and semantic segmentation of a scene from an RGB-D image: Mesh R-CNN [5]. which predicts object instance geometries from an RGB image, and Manl3Dnndersttndii from an RGB image. Since SSCNet takes as input an RGB-D image, we provide our predicted depth maps as input along with the RGB image, and obtain additional object instance ids from the semantic labels by applying instance clustering (described in Sec. 6..) Mesh R-CNN builds from Mask R- R-CNN [9] to predict object geometry for each detected object, so we evaluate only things classes performance. Total3D similarly predicts object geometry for detected objects in RGB images, with an additional output head to predict the room layout cuboid. We convert Total3D room layout predictions wall (vertical) and floor (horizontal) predictions for evaluation; note that since Total3D requires known canonical poses for objects the scenes, which are not provided by the Matterport3DD dataset, do not compare with Total3D Matterport3D. All evaluated methods have been trained on our generated panoptic data for the respective validation dataset.
[32. which predicts object instance geometry and cuboid room layouts
comparison to these baselines on synthetic 3D-Front [14] data. Our holistic approach, effectively leveraging depth and 2D object recognition for 3D propagation, results significant improvement in panoptic reconstruction quality, in comparison with state-of-the-art alternatives that tend to predict stuff and things separately, with little to no shared information. As shown in the qualitative comparison in Figure our approach more accurately generates both global structures and object locations as well as local geometric structure, in both observed and occluded regions in the camera frustum.
We also compare with state of the art on real-worke image data from Matterport3D real-world imagery of this data contains much more complexity in scene geometry, object appearance, and arrangement, along with inherent noise sensor capture, making the panoptic reconstruction task significantly more challenging. Even under this challenging scenario, our holistic approach to panoptic reconstruction results in more effective geometric and semantic understanding than previous approaches which tackled the component tasks independently.
Table The
7

---

RGB Input
SSCNet
Mesh R-CNN
Total3D
Ours
Groundtruth
Figure 4: Qualitative comparison of panoptic 3D scene reconstruction on synthetic 3D-Front [4]
images, in comparison with SSCNet (38) followed by instance clustering, Mesh R-CNN [15] and Total3D [32. Our propagation- based joint panoptic reconstruction effectively captures global structures as well as local object arrangements. Different colors denote different instances.
Table 1: Quantitative evaluation of Panoptic Reconstruction Quality on 3D-Front
14]
PRQ
RSQ
RRQ
PRQ
RSQ
RRQ
PRQ
RSQ
RRQ
Things
Stuff
SSCNet [38 IC
11.50
32.90
33.00
8.03
32.07
24.69
26.95
36.75
70.25
Mesh R- R-CNN 155
20.90
38.00
53.20
Total3D 332
15.08
36.63
40.15
13.77
34.88
38.89
20.94
44.49
45.85
Ours w/o IP
20.65
53.87
29.62
8.48
48.30
15.07
75,40
78.95
95.10
Ours w/o hier.
44.05
55.31
70.54
37.34
50.12
65.33
74.20
78.65
93.95
Ours
46.77
57.35
73.13
40.52
52.52
68.43
74.90
79.10
94.25
Table 2: Quantitative evaluation of Panoptic Reconstruction Quality on Matterport3D 2
PRQ
RSQ
RRQ
PRQ
RSQ
RRQ
PRQ
RSQ
RRQ
Things
Stuff
SSCNet 38 IC
0.49
21.68
1.50
0.19
22.75
0.59
1.43
20.43
4.43
Mesh R- R-CNN [5
6.29
31.12
15,60
Ours
7.01
28.57
17.65
6.34
26.06
16.06
10.78
40.03
26.77

---

RGB Input
Mesh R-CNN
Ours
Groundtruth
Figure 5:
Qualitative comparison of panoptic 3D scene reconstruction from real-world Matter-
port3D 2 images, comparing the object reconstructions of Mesh R-CNN [I5] with our panoptic reconstruction of the scene. Different colors denote different object instances.
6.3 Instance Clustering
Instance clustering a commonly used technique for 3D instance segmentation, used in many state- of-the-art methods 26 12 17 We integrate instance clustering to sate-of-theeane comparisons with methods which produce only semantic segmentation, as well as to compare our instance propagation with 3D instance clustering approach.
We apply instance clustering semantic segmentation predictions, generating instance predictions in a bottom-up fashion. From the semantic labels, surface locations sharing the same semantic label and lying close together in euclidean space are grouped into instances. A greedy breadth-first search is performed to iteratively find clusters of surface geometry belonging to the same semantic label, where 3D location is included in the cluster if it within 2cm of the nearest location in the cluster. A surface location that does not fulfill this requirement then becomes seed for new instance cluster. Final output clusters then form instance predictions, where the semantic given by the semantic label of the locations in the cluster, and the confidence of the instance is the average of the semantic probability scores of the locations in the cluster.
6.4 Ablation Studies
Does instance propagation help? In Table T we analyze our instance propagation from 2D, in comparison with 3D instance prediction approach (Ours w/o IP), instead using instance clustering as described in Sec, (6.3 Since our instance propagation can directly leverage high-resolution 2D signal to be refined to 3D objects rather than direct prediction coarser estimated geometry, instance propagation results significantly improved panoptic reconstruction performance.
What is the effect of the progressive coarse-to-fine prediction? coarse-to-fine hierarchical predictions in Table in comparison to an alternative without the use of progressively generated intermediate hicrarchy predictions (Ours w/o hier). The coarse-to-fine strategy helps to localize predictions carly, resulting in a performance improvement.
additionally analyze the our
What is the effect of providing ground truth depth information? We investigate the perfor- mance of our approach and baseline methods when we provide ground truth depth information. For our method and SSCNet 38 we train and evaluate with ground truth depth. For MeshR [15 we feed the ground truth z-location of each object. Here, we also train and compare with Sketch-Aware SSCNet [3]. The results can be found in Table

---

Table 3: Oracle ablation with providing ground truth depth for SSCNet, Sketch and our approach.
For MeshRCNN we supply the ground truth z location of each object.
PRQ
PRQ
PRQ
all
Things
Stuff
SSCNet [38] IC w/ GT depth
15.88
12.36
31.75
Sketch [3] IC w/ GT depth
26.12
26.23
25.60
Mesh R-CNN IS w/ GT z
35.76
Ours w/ GT depth
48.08
41.67
76.95
6.5 Limitations
The task of panoptic scene reconstruction sets forth a evaluation of simultaneously accurate geometric reconstruction, semantic labeling, and object recognition. Our propagation- based approach from 2D estimates to 3D refinement shows effective results on synthetic environments, but can sometimes struggle to accurately place things and stuff for real-world observations where there significantly more noise and clutter, which can result in shifts of predicted structures. We believe that stronger priors on object sizes and relations are a promising avenue, as well as further integration of the joint prediction of stuff and things from 2D to 3D.
7 Conclusion
We introduced the task of panoptic scene reconstruction from single RGB image, to predict holistic understanding of the image encompassing geometric reconstruction, 3D semantic segmenta- tion, and 3D instance segmentation. Our new method for panoptic reconstruction effectively leverages lifting both learned 2D features well as predicted depth and 2D instances to 3D, significantly outperforming state-of-the-arr alternatives to this task. We hope that the panoptic reconstruction task will help to drive forward research towards more comprehensive holistic scene understanding, of which structural and geometric understanding play a fundamental part.
Broader Impact
Our work proposes new, holistic scene understanding task of panoptic 3D scene reconstruction from a single RGB image, which lies at the basis of many visual perception tasks, such as autonomous navigation, mixed reality, content creation. As our method is heavily data-driven, we must be aware of the use of data and any new data captures to ensure consent and privacy considerations, as well as possible unintended biases from data captured in limited set of locations.
Acknowledgements
This project is funded by the Bavarian State Ministry of Science and the Arts and coordinated by the Bavarian Research Institute for Digital Transformation (bidt), the TUM Institute of Advanced Studies (TUM-IAS), the ERC Starting Grant Scan2CAD (804724), and the German Research Foundation (DFG) Grant Making Machine Learning on Static and Dynamic 3D Data Practical.
References
[1] C. Batty. SDFGen. hattps://githe httn:///.tttps/gggggi...o///rrrn https/hhww...ooooro.ootttaaattap
[2] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner. M. Savva, S. Song, A. Zeng and Y. Zhang Matterport3D: Learning from RGB-D data in indoor environments. International Conference on 3D Vision (3DV), 2017.
[3] X. Chen, K.-Y. Lin, C. Qian, G. Zeng, and H. Li. 3d sketch-aware semantic seene completion via semi- supervised structure prior. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2020.
[4] C. B. Choy, D. Xu, Gwak, K. Chen, and Savarese, 3d-r2n2: A unified approach for single and multi- view 3d object reconstruction. In European Conference on Computer Vision, pages 628 644. Springer, 2016.
10

---

[5] B. Curless and M. Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd anmual conference Computer graphics and interactive techniques, pages 303-312. ACM, 1996.
[6] M. Dahnert, A. Dai, J. Guibas, and M. Nießner. Joint embedding of 3d scan and cad objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8749-8758, 2019.
[7] A. Dai and M. Nießner. 3dmv: Joint 3d-multi- miulti-view prediction
semantic scene segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[8] A. Dai, D. Ritchie, M. Bokeloh, S. Reed, J. Sturm, and M. Nießner. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. In Proc. Computer Vision and Pottern Recognition (CVPR), IEEE, 2018.
[9] A. Dai, C. Diller, and M. Nießner. Sg-nm: Sparse generative neural networks for self-supervised scene the IEEE/CVF Conference Computer Vision and Pottern completion of rgb-d scans. In Proceedings the Recognittin, pages 849-858, 2020.
[10] M. Denninger and R. Triebel. 3d scene reconstruction from a single viewport. In European Conference on Computer Vision, pages 51 67. Springer, 2020.
[11] M. Denninger, M. Sundermeyer, D. Winkelbauer, Y. Zidan, D. Olefir, M. Elbadrawy, A. Lodhi, and 2019.
H. Katam. Blenderproc. arXiv preprint urXiv:1911.0199
[12] F. Engelmann, M. Bokeloh, A. Fathi, B. Leibe, and M. Nießner. 3D-MPA: Multi-Propont Aggregation for 3D Semantic Instance Segmentation. In CVPR, 2020.
[13] H. Fu, R. Jia. L. Gao, M. Gong, B. Zhao, S. Maybank, and D. Tao. 3d-future: 3d furniture shape with texture. arXiv preprint arXiv:2009.0%6: 2020.
[14] H. Fu, B. Cai, L. Gao, L. Zhang, J. W. C. Li, Z. Xun, C. Sun, R. Jia, B. Zhao, and H. Zhang. 3d-front: 3d furnished rooms with layouts and semantics. arXiv preprint arXiv:201 1.00011100110 1.09127, 2021.
[15] G. Gkioxari, J. Malik, and J. Johnson. Mesh r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9785-9795, 2019.
[16] T. Groueix, M. Fisher, G. Kim, B. C. Russell, and M. Aubry. A papier-måché approach learning 3d surface generation. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 216 216-224, 2018.
[17] I. Han, T. Zheng, L.. Xu, and I.. Fang. OccuSeg: Occupancy-awan 3D instance segmentation. In CVPR, 2020.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[19] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. In ICCV, 2017.
[20] Hou, A. Dai, and M. NieBner. 3D-SIS: Semantic Instance Segmentation of RGB-D Scans. In CVPR,
2019.
[21] J. Hou, A. Dai, and M. Nießner. RevealNet: Seeing Behind Objects in RGB-D Scans. In CVPR, 2020.
[22] Hu, M. Ozay, Zhang, and T. Okatani. Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), 2019.
[23] S. Huang, S. Qi. Y. Xiao, Y. Zhu, Y. N. Wu, and S.-C. Zhu. Cooperative holistic scene understanding: Unifying object, layout, and camera pose estimation. arXiv preprint arXiv: 13049, 2018.
[24] S. Huang, S. Qi, Y. Zhu, Y. Xiao, Y. Xu, and S.-C. Zhu. Holistic 3d scene parsing and reconstruction from a single rgb image. In Proceedings of the European conference computer vision (ECCV), pages 187-203, 2018.
[25] S. Huang, Y. Chen, T. Yuan, S. Qi, Y. Zhu, and S.-C. Zhu, Perspectivenet: 3d object detection from a single rgb image via perspective points. arXiv preprint 1912.0771 2019.
[26] L.. Jiang, H. Zhao, S. Shi, S. Liu, C.-W. Fu, and Jia. PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. In CVPR, 2020.
[27] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Dollár, Panoptic segmentation. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9404-9413, 2019.
[28] Kuo, A. Angelova, T.-Y. Lin, and A. Dai. Mask2cad: 3d shape prediction by learning to segment and retrieve. In European Conference on Computer Vision (ECCV), volume 1, page 3. Springer, 2020.
[29] J. Lahoud, B. Ghanem, M. Pollefeys, and M. R. Oswald. 3d instance segmentation via multi- task metric learning. In ICCV, 2019.
[30] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460 4470, 2019.

---

[31] G. Narita, I. Seno, T. Ishikawa, and Y. Kaji. Panopticfusion: Online volumetric semantic mapping at the level of stuff and things. arXiv preprint arXiv:1903.01177, 2019.
[32] Y. Nie, X. Han, S. Guo, Y. Zheng, J. Chang, and J. Zhang. TotalBdunderstanddnn: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 55 64, 2020.
[33] Y. Nie, J. Hou, X. Han, and M. Nießner. RiD-Net: Point Scene Understanding by Semantic Instance Reconstruction. In Proceedings of Computer Vision and Pattern Recognition (CVPR), JEEE, 2021.
[34] S. Popov, Bauszat, and V. Ferrari. Corenet: Coherent 3d scene reconstruction from single rgb image. In European Conference on Computer Vision, pages 366-383. Springer, 2020.
[35] C. R. Qi. H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3D classification and segmentation. CVPR, 2017.
[36] O. Ronneberger, P. Fischer, and T. Brox, U-Net: Convolutional networks for biomedical image segmenta- tion. In MICCAI, 2015.
[37] D. Shin, Ren, E. Sudderth, C. C. Fowlkes. scene reconstruction with multi-layer depth and epipolar transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 2172-2182, 2019.
2019.
[38] S. Song, Yu, A. Zeng. A. X. Chang, M. Savva, and T. Funkhouser. Semantic scene completion from single depth image. Proceedings of 30th IEEE Conference on Computer Vision and Puttern Recognition, 2017.
[39] N. Wang, Y. Zhang, Li, Y. Fu, W. Liu, and Y.G.. Jiang. Pixel2mesh: Generating mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52 67, 2018.
[40] Wang, R. Yu, Q. Huang, and Neumann. Sgpn: Similarity group proposal network point cloud instance segmentation. CVPR, 2018.
[41] B. Yang, J. Wang, R. Clark, Q Hu, S. Wang, A. Markham, and N. Trigoni. Learning object bounding boxes for 3D instance segmentation on point clouds. In NeurlPS, 2019.
[42] L. Yi, W. Zhao, H. Wang, M. Sung, and L. Guibas. GSPN: Generative shape proposal network for 3D instance segmentation in point cloud. In CVPR, 2019.
12


Reviews:
[{"review_id": "yBV1NiInO5u", "reviewer": "Reviewer_iPdb", "summary": "The authors propose a way to perform panoptic 3d scene reconstruction from a single image. Depth and instance segmentation from an rgb image help create a 3d volume through back-projection which is subsequently refined to produce semantics, instances and 3d reconstruction, which effectively translates to a panoptic representation. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong Points\n- The authors are able to perform the hard task of panoptic reconstruction from a single image\n- The method works better than SSCNet and Mesh R-CNN\n\nWeak Points\n- The evaluation of the method is severely lacking. As the authors produce 3d reconstruction and semantic segmentation in addition to instance ids, there should be proper evaluation for both these outputs against state of the art. \n- I understand that no other method does panoptic 3d reconstruction. However, the authors should 'downgrade' panoptic segmentation and compare with other semantic segmentation methods as opposed to 'upgrading' an existing method to produce panoptic labels. This will avoid bias and validate the usefulness the proposed approach at a more modular level. \n- Several relevant approaches have not been discussed and compared against. For e.g., CoReNet by Popov et. al. should be compared for the task of 3d reconstruction and 3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior by Chen et.al. is a better and more recent baseline compared to SSCNet. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss limitations and societal impact of the work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "ufhf7sMUAfd", "reviewer": "Reviewer_8qDd", "summary": "This paper defines and presents a method for the novel task of panoptic 3D scene reconstruction, i.e. jointly predicting complete 3D scene geometry, 3D semantic and 3D instance labels. The method consists of using backbone 2D models for depth and instance segmentation and lifting them into 3D before processing with a SG-NN [5] like model to complete and refine the initial volume into instance, semantic and occupancy/TSDF volumes. The authors propose an \"instance propagation\" method to associate 2D instance masks with 3D instances which is shown to improve over a post hoc \"instance clustering\" in a 3D semantic only volume. Experiments on a synthetic dataset (3D Front) and Matterport3D show improvement over prior works , some of which tackle a subset of the problem (e.g. only scene structure and semantics). \n\nThe primary contribution of this paper is the definition of this new holistic panoptic 3D scene reconstruction task and a new method and evaluation metric for the same with improved performance over prior works which tackle these individual tasks in isolation.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n- 3D scene structure, 3D semantic and 3D instance segmentation are critical to a holistic understanding of the scene and have been seeing a lot of progress in recent years. Similar to the 2D analog of panoptic segmentation, 3D panoptic reconstruction is the next logical step in this evolution and this paper helps define it with metrics and a method for it. This is quite an important contribution in my opinion.\n\n- The paper is well written and the illustrations make it easier to understand the paper (e.g. Fig 3 helps explain the instance propagation idea well). The visual results are quite impressive.\n\n- The comparisons against prior work (Table 1 and 2) have been carefully done with proper explanation of the alternative used when an existing system wasn't directly applicable in this setting (e.g. input depth for SSCNet, Total3D requiring canonical poses for objects).\n\n*Weaknesses*\n- One concern I have with this paper is with technical novelty. At the surface it seems like a simple combination of existing depth + instance segmentation algorithms combined with prior work for scene completion (SG-NN) with extension to semantic and instance label volumes. The instance propagation idea is slightly novel but perhaps not enough. However, defining the 3D panoptic scene reconstruction task and proposing a metric for it could be considered a significant enough contribution.\n\n*Clarifications*\n  - L120 - How are multiple feature map from different spatial resolution from the ResNet concatented into $\\theta$?\n  - Why was the choice to randomly select a pixel for a voxel with multiple backprojections made? Why would you not use standard bilinear sampling for the same?\n  - In Table 2, Mesh R-CNN seems to show significantly better performance on \"Things\" over this method. What is the primary reason for this?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe the authors have addressed some limitations in Sec 6.5 of the paper. I would appreciate responses to the questions mentioned in the clarifications section above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "NwvpQ_2jht", "reviewer": "Reviewer_9SvP", "summary": "The authors define a new problem called \"panoptic 3D scene reconstruction\" - a joint task of 3D geometry prediction, semantic segmentation, and instance segmentation, from a single RGB image. To address this new task, a two-stage method is proposed. In the first stage, 2D CNN is used to predict 2D features, a depth image, and 2D instance segmentations. In the second stage, these 2D features and instance segmentation results are back-projected into 3D, and then passed into a 3D CNN for the final 3D predictions. This proposed pipeline is demonstrated on the synthetic 3D-front dataset, as well as the Matterport3D real-world dataset, and significantly outperforms prior related work.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n\nThe newly defined \"panoptic 3D scene reconstruction\" problem is a natural extension of 2D panoptic segmentation. The components used in the proposed pipeline, including reusing 2D features, uplifting 2D features to 3D via backprojection, multi-task joint training, and 3D CNNs, are all well studied in the community. \n\nQuality \n\nBased on the evaluation metrics, the proposed method significantly outperforms related prior work in terms of geometry, semantic segmentation, and instance segmentation. The example predictions also seem to align better with the ground truth. However, the experimental section is relatively simple, and I do have questions about the proposed method and some suggestions regarding ablation studies. \n\n1. Based on Section 4.1, it is my understanding that the 2D network and the 3D network are trained sequentially and separately, rather than jointly. Could you elaborate on the reasoning behind this design choice? Does joint training lead to sub-optimal results?\n2. The current evalutions are all based on 3D volumetric representations, but not 2D. When the 3D predictions are rendered back into the image space, how is the quality of depth, semantic mask, and instance segmentation? Does the 3D network improve the quality, or decrease it?\n3. According to Section 4.1, the real-world model is pretrained on synthetic data and then fine-tuned on Matterport3D. Is this the same training procedure for the prior work for a fair comparison? Is there additional effort to make sure that the prior method (e.g. Mesh R-CNN [13]) is trained properly on these datasets and is indeed achieving their optimal performance?\n4. The comparison against SSCNet [31] seems a bit strange, given that the method expects a single (good) depth image as input. A more fair experiment would be to use ground truth depth in both SSCNet and the proposed method.\n5. After reading the paper, it is still unclear to me why there has been a significant improvement in the metrics compared to prior work - is it because of joint task learning providing additional supervision and task-shared feature representations, or is the proposed network architecture far better than prior related work? Please elaborate and consider adding ablation studies to support your arguments.  \n6. On Line 127, viewing directions are also used as input - does this help improve quality?\n\nClarity \n\nThe paper is straightforward and easy to follow.\n\nSignificance\n\nI believe panoptic 3D scene reconstruction, and more generally multi-task prediction in 3D, is an important topic in the computer vision / machine learning community. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "BIHs1EFuWpS", "reviewer": "Reviewer_faBK", "summary": "The paper proposes an approach for panoptic 3D reconstruction that combines, in an end-to-end trainable manner: (i) 2D depth from rgb, (ii) mask rcnn-style instance segmentation and (iii) a final refinement stage to produce the overall final result.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive:\n* While I am somewhat uninformed about the state of the art, this does appear to be the first paper to propose an end to end trained panoptic 3D reconstruction (/ stuff + instance segmentation) solution. \n* The paper is well written and easy to understand.\n\nNegative / Questions:\n* The key components of the approach are not entirely novel or special, with the various bits of the system being used before in previous tasks (eg the standard-ish depth generation and 2D instance segmentation, traditional-ish lifting, sparse + dense Unet as used before in shape completion).\n* The overall structure is also somewhat predictable (except for, maybe, the sparse + dense unet component).\n* Literature review seems rather brief, especially in the single view 3D reconstruction section, eg CoReNet and Points2Objects (and related) seem to be missing, though would be quite relevant.\n* A version of Figure 1 + 2 from the supplementary should be in the main paper, since it aids quite a bit in understanding the proposed method.\n* A benchmark / ablation of sparse vs dense 3D CNNs at the start of the network could be added.\n* Given the bottoms-up nature of the approach, a baseline of eg backprojected / lifted 2D panoptic segmentation would seem like a natural approach to investigate / compare against. Is this included in the paper?\n\nOverall, while I do think the solution is rather simple (and reads a bit like a concatenation of previous papers), I am for acceptance because of the novelty of the task, the good results and the quality of the writing. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to learn panoptic 3D scene reconstruction from a single image. Depth and instance segmentation from an RGB image instantiate a 3D feature volume through back-projection which is subsequently refined to produce semantics, instance segmentations and 3D reconstruction. All reviewers acknowledge the good empirical performance of the method and the high  relevance of the problem tackled for the NeurIPS audience. 
Concerns were raised regarding comparisons of the paper to previous methods that did not perform jointly the tasks of segmentation and 3D reconstruction, which the rebuttal addressed. Authors are encouraged to include all additional experiments in the final version.

Decision: Accept (Poster)