Panoptic 3D Scene Reconstruction From a Single RGB Image

1 Introduction . 3D scene understanding from a single RGB image is fundamental to many downstream applications , such as for robotics , motion planning , or augmented reality . From a photograph of a scene , one can infer the underlying geometric structures and identify object and structural semantics , even from such a partial , single view observation . Such an understanding is also fundamental towards enabling higher-level interactions with environments , as estimating unseen geometry and semantics without having to observe all parts of an object or scene is paramount for tasks such as robotic interactions , augmented reality , or content creation . Recently , significant advances have been made towards complete geometric reconstruction from an RGB image input [ 10 , 37 ] , as well as reconstruction of object instances detected in the image [ 15 , 28 , 32 ] ; however , 3D semantic reconstruction and instance reconstruction have largely been considered independently . We thus aim to unify the tasks of geometric reconstruction , 3D semantic segmentation , and 3D instance segmentation from a single image , and inspired by the 2D panoptic segmentation task [ 27 ] , we call this task panoptic 3D scene reconstruction . Inspired by the unification of the 2D semantic segmentation and instance segmentation tasks to panoptic segmentation in [ 27 ] , we similarly adopt a joint semantic segmentation of “ stuff ” and “ things , ” where “ stuff ” elements do not have any distinct instances ( e.g. , structural elements such as walls and floors ) and “ things ” denote semantic objects with instance ids defining the distinct object instances . Thus , for panoptic 3D reconstruction , we aim to predict the surface geometry of the scene from an image , including any missing regions from occlusion , and for each point on the predicted surface geometry , it must be assigned a semantic label and an instance id . To evaluate a predicted panoptic reconstruction , we then evaluate both the segmentation quality and recognition quality of the 3D object instance geometry and geometry corresponding to structural semantic labels . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We believe that this task not only brings together geometric reconstruction , 3D semantic segmentation , and 3D instance segmentation , but also introduces new algorithmic challenges , and hope that the panoptic 3D scene reconstruction task leads to additional insights in holistic 3D perception . To address the panoptic 3D scene reconstruction task , we propose a new method to jointly reconstruct and segment the observed 3D scene from a RGB image . We first extract features from the image with a 2D convolutional backbone which predicts both depth estimates and 2D instance segmentation . We use the depth estimates to guide the feature lifting to 3D by back-projecting the learned features into a 3D volume of the camera frustum , and develop an instance propagation approach to effectively leverage instance predictions in 2D as seeds for propagation to 3D geometric instances , providing effective object recognition . In contrast to prior approaches for 3D reconstruction from 2D images , we develop a propagation-based approach for object recognition , explicitly bringing 2D predictions to 3D for a refinement rather than direct 3D prediction on a coarse resolution than the 2D signal . We train the object recognition jointly with a reconstruction loss on the predicted scene geometry as well as semantic “ stuff ” labels , resulting in a holistic 3D semantic scene estimate for an RGB image . In summary , our contributions are : • We introduce the task of panoptic 3D scene reconstruction , which aims for holistic 3D scene understanding by jointly predicting scene geometry , semantic labels , and instance ids . • We propose a new approach for panoptic 3D scene reconstruction by learning to lift 2D features to 3D while propagating learned instance information to 3D object understanding for a robust joint prediction of scene geometry , semantics , and object recognition1 . • Our method significantly outperforms alternative approaches which treat reconstruction and semantics separately and/or infer only a subset of the outputs . 2 Related Work . Panoptic Segmentation . Kirillov et al . [ 27 ] proposed the task of panoptic segmentation for 2D images , establishing a unified , holistic scene understanding task for image understanding and a panoptic quality metric for evaluation . The task of panoptic segmentation of a single color image is to assign each pixel in the image with an instance label and a semantic label . The semantic label set is composed of “ stuff ” and “ things , ” where “ stuff ” labels do not consider instance ids as all pixels belonging to the same “ stuff ” class belong to the same instance , and “ things ” labels consider instance ids to distinguish different object instances . This has achieved notable success in inspiring further holistic analysis of image understanding , and we believe such a holistic analysis also plays an important role in 3D perception . We aim to bridge together geometric scene reconstruction [ 37 , 10 ] , 3D semantic segmentation [ 7 , 35 ] , and 3D instance segmentation [ 40 , 20 , 42 , 12 , 26 , 17 , 41 , 29 ] , along with a panoptic 3D scene reconstruction evaluation to capture these characteristics . Narita et al . [ 31 ] proposed an approach for joint semantic and instance segmentation on 3D reconstructed scenes , but rely on given 3D scene information without inferring geometry . To the best of our knowledge , our work is the first to bring the panoptic reconstruction task into 3D perception from RGB images . 1Code can be found at https : //github.com/xheon/panoptic-reconstruction . 3D Semantic Scene Completion and Semantic Instance Completion . Several recent works have explored the task of 3D semantic scene completion , in which both complete scene geometry and semantic segmentation are predicted from partial observations of a scene . From a single RGB-D frame , SSCNet [ 38 ] predicts the geometric scene occupancy and semantic class labels in a volumetric grid . [ 3 ] follows up on it and proposes to guide the semantic scene completion with coarse 3D sketches of the scene . ScanComplete [ 8 ] presented an autoregressive approach for large-scale scan completion and semantic segmentation . Dahnert et al . [ 6 ] proposed to retrieve complete objects of partially scanned objects from a shape database by constructing a joint embedding space of scan and CAD objects . Hou et al . [ 21 ] and Nie et al . [ 33 ] then focused on joint scan completion with instance segmentation of the predicted complete geometry . These approaches have shown the synergies in joint geometric completion along with semantic segmentation or instance segmentation , and we aim to unify all these tasks together for a holistic approach to 3D perception from a single RGB image input . Single-View 3D Reconstruction . Estimating object or scene geometry from only a single RGB image view has been long-studied , with recent advances in deep learning showing significant potential in the reconstruction of shapes from single views [ 4 , 16 , 39 , 30 ] . As intermediate representation Huang et al . extend 2D bounding boxes to 3D via joint estimation [ 23 ] or perspective constraints [ 25 ] . Huang et al . [ 24 ] proposes scene grammar , which allows to retrieve CAD models and estimate scene layout to be optimized to fit a single RGB input image . Shin et al . [ 37 ] and Denninger et al . [ 10 ] recently proposed methods to estimate the scene geometry within the camera frustum of an image view . Recently , Mesh-RCNN [ 15 ] pioneered an approach to extend single object reconstruction to RGB images with multiple objects by detecting and reconstructing the object geometry of each detected object in the image , along with their relative orientations . Kuo et al . [ 28 ] similarly predicted the object geometry for detected objects in an RGB image , but by CAD model retrieval rather than a generative mesh model . CoReNet [ 34 ] omitted explicit object detection and instead extracted 2D information via physically-based ray-traced skip connection between the image and the 3D volume . The approach reconstructs the shape and semantic class of multiple objects from a single RGB image directly in a 3D volumetric grid . Nie et al . [ 32 ] then proposed an approach to detect object poses , their geometry , and cuboid room layouts . Similar to Nie et al. , we aim to more comprehensively understand both object and scene structures and semantics , but remove any cuboid-structure assumptions , instead lifting 2D features and instance information to 3D for prediction of both “ stuff ” and “ things , ” and additionally propose a new holistic evaluation metric to understand the full panoptic 3D scene reconstruction quality . 3 Method Overview . In the panoptic 3D scene reconstruction task , a single RGB image is taken as input , from which the scene geometry within the camera frustum is predicted , along with semantic and instance label for each surface location on the predicted geometry . We introduce a new method for this task , extracting strong 2D features from the input RGB image . The features are further projected and propagated to 3D , in which the final geometry , semantic , and instance predictions are predicted . From an input RGB image , we leverage a state-of-the-art 2D instance segmentation network as a 2D backbone to extract features , with proxy losses for 2D instance segmentation and depth estimation . The predicted depth is then used to estimate a coarse 3D representation by back-projecting the learned 2D features and predictions into a sparse volumetric representation . This coarse 3D estimate is then encoded by 3D sparse convolutions , and a coarse-to-fine approach is used to predict high-resolution output as a sparse volumetric representation , with each voxel containing a distance value representing the surface geometry as a distance field , a semantic label , and an instance id . In particular , we exploit the 2D instance predictions by directly propagating them to 3D to be refined , which provides a robust semantic 3D prediction . 4 Hybrid 2D-3D Panoptic Reconstruction . Learning 2D Priors . An input RGB image I is first encoded by a 2D backbone Θi , realized with a ResNet-18 [ 18 ] architecture . To learn effective priors for the panoptic 3D scene reconstruction task , we employ proxy losses on Θi ( I ) for both 2D instance segmentation and depth estimation . We follow Mask R-CNN [ 19 ] for the instance segmentation , predicting 2D object bounding boxes , class category labels , and instance masks from Θi ( I ) . For depth estimation , we use the depth estimation decoder of [ 22 ] , which predicts depth from multiscale features extracted by Θi , using losses on the logarithm of the depth errors , gradients , and normals . We then extract features from the output of each ResNet block in Θi as well as the decoder outputs , which are then concatenated to θ to be projected to 3D . Lifting 2D Features to 3D . We then lift the 2D features to 3D using the known camera intrinsics of the image and the predicted depth estimates . The 2D features θ , depth estimates , and predicted 2D instance segmentation are back-projected into a 2563 3D volumetric grid using the estimated depth , and encoded as a truncated signed distance field ( TSDF ) along the view direction , where positive values are in front of the predicted surface , zero encodes the predicted surface , and negative values unseen regions . We use a truncation of τ = 3 voxels , and thus use a sparse 3D representation . To spatially correlate learned 2D features to directly inform the 3D panoptic reconstruction , the image features θ as well as instance segmentation mask logits corresponding to each depth estimate copied along the view direction within the truncation , resulting in a sparse 3D feature volume F = ( Fd , Ff , Fi ) as the concatenation of the computed distance field Fd , projected image features Ff , and projected instance masks Fi . Multiple pixels falling into the same voxel bin are randomly sampled . The 1-D instance segmentation mask logits for each object of a maximum of N detected objects are arbitrarily associated with the channels of an N + 1-channel volume , with the extra channel encoding a 1-hot encoding for voxels with no instance mask associations . This enables instance propagation from detected 2D object instances to final refined 3D object “ things , ” explicitly leveraging strong 2D priors which result in more robust panoptic reconstruction than relying only on distinction of objects from a much coarser 3D representation than the information from the high-resolution 2D image . Sparse Generative Panoptic 3D Scene Reconstruction . The panoptic 3D scene reconstruction is then predicted from the feature volume F = ( Fd , Ff , Fi ) with a sparse generative approach . Each of ( Fd , Ff , Fi ) are first separately encoded with sparse 3D convolutions to transform the different features characteristics into a common space . The resulting features are then concatenated as F ′ and encoded together with a sparse 3D encoder using a common sparse encoder Θs , realized with a UNet-style architecture [ 36 ] . Since the panoptic reconstruction should both refine the estimated depth as well as predict new geometry for unobserved regions , Θs follows the same geometric structure as its input F ′ , spatially downsampling by a factor of 4 , while the panoptic reconstruction is predicted in coarse-to-fine fashion , starting with a coarse prediction from ΘS ( F ′ ) using dense 3D convolutions to enable generation of new geometric structures , similar to the sparse generative approach of SG-NN [ 9 ] for geometric scene completion . The initial dense prediction is then decoded to sparse panoptic predictions at each spatial hierarchy level , with separate prediction heads for geometric occupancy , semantic labels , and instance ids , which are then used to inform the next hierarchy level for refined predictions . Similar to [ 9 ] , we also employ dense and sparse skip connections between the encoder and decoder . At the last hierarchy level , we additionally predicted refined surface geometry as a sparse distance field , along with semantic label and instance ids per voxel . The final panoptic 3D scene reconstruction is then obtained by extracting surface geometry from the isosurface of the predicted distance field . Each location with distance to the surface less than τs is assigned the semantic label from the 3D semantic segmentation head if that label corresponds with a “ stuff ” label , and otherwise assigned the semantic label and instance id of the predicted instance mask at the location . Empirically , we found the precedence for “ stuff ” labels to perform slightly more robustly than the reverse , and provide the nearest semantic and instance labels for any locations that may have not had any semantic or instance predictions . Losses . We train our panoptic 3D scene reconstruction with the following loss : L = wdLd + wiLi + ∑ h ( wgLhg + wsLhs + woLho ) , ( 1 ) where Ld denotes the 2D depth estimation loss , Li the 2D instance segmentation , and at each hierarchy level h , a loss for geometry Lhg , semantic label Lhs , and instance id Lho . Note that the loss is not computed for any regions of the 3D predictions that lie outside of the image view frustum . For 2D depth estimation loss Ld , we follow the losses used by [ 22 ] , using a log-encoded ` 1 loss on the predicted depth values along with a cosine normal consistency loss and a gradient loss to regularize the depth estimation . The 2D instance segmentation loss Li follows the standard Mask R-CNN [ 19 ] procedure , with losses on the regressed 2D boxes , class category labels , and binary masks . Our panoptic 3D reconstruction is trained to predict geometry , semantic labels , and instance ids for sparse voxels within the truncation region . For the geometric loss Lhg , each hierarchy level is trained to predict geometry occupancy using a binary cross entropy loss , and the final hierarchy level also employs an ` 1 loss on predicted distance values determining the surface geometry : Lhg = BCE ( Ohpred , Ohgt ) + 1h==0 ‖Spred − Sgt‖1 , where O denote the occupancy predictions and ground truth at the hierarchy level , and for the final hierarchy level h = 0 , S denotes the distance field representing the surface geometry . The occupancy predictions at each level are used to generate the sparse 3D structure used for the sparse 3D convolutions in the following hierarchy level . Semantic labels are trained with a class-weighted cross entropy loss Lhs for each hierarchy level h. 3D instance ids are predicted as binary masks corresponding to the detected objects encoded in the channels of Fi . That is , each detected object informs an instance mask refinement and propagation from the initial projected 2D mask logits . The 2D object masks are correlated with ground truth 2D masks when the IoU > 0.5 . The objects are then arbitrarily ordered to enable a cross entropy loss for the 3D predictions , with the correlated ground truth masks assigned to the arbitrary ordering . A visualization of our instance propagation loss is shown in Figure 3 . This provides a strong prior on 3D objects from the 2D predictions , from which only a refinement must be learned in 3D , resulting in more robust and accurate object recognition .

Summary: The authors define a new problem called "panoptic 3D scene reconstruction" - a joint task of 3D geometry prediction, semantic segmentation, and instance segmentation, from a single RGB image. To address this new task, a two-stage method is proposed. In the first stage, 2D CNN is used to predict 2D features, a depth image, and 2D instance segmentations. In the second stage, these 2D features and instance segmentation results are back-projected into 3D, and then passed into a 3D CNN for the final 3D predictions. This proposed pipeline is demonstrated on the synthetic 3D-front dataset, as well as the Matterport3D real-world dataset, and significantly outperforms prior related work.