Panoptic 3D Scene Reconstruction From a Single RGB Image

1 Introduction . 3D scene understanding from a single RGB image is fundamental to many downstream applications , such as for robotics , motion planning , or augmented reality . From a photograph of a scene , one can infer the underlying geometric structures and identify object and structural semantics , even from such a partial , single view observation . Such an understanding is also fundamental towards enabling higher-level interactions with environments , as estimating unseen geometry and semantics without having to observe all parts of an object or scene is paramount for tasks such as robotic interactions , augmented reality , or content creation . Recently , significant advances have been made towards complete geometric reconstruction from an RGB image input [ 10 , 37 ] , as well as reconstruction of object instances detected in the image [ 15 , 28 , 32 ] ; however , 3D semantic reconstruction and instance reconstruction have largely been considered independently . We thus aim to unify the tasks of geometric reconstruction , 3D semantic segmentation , and 3D instance segmentation from a single image , and inspired by the 2D panoptic segmentation task [ 27 ] , we call this task panoptic 3D scene reconstruction . Inspired by the unification of the 2D semantic segmentation and instance segmentation tasks to panoptic segmentation in [ 27 ] , we similarly adopt a joint semantic segmentation of “ stuff ” and “ things , ” where “ stuff ” elements do not have any distinct instances ( e.g. , structural elements such as walls and floors ) and “ things ” denote semantic objects with instance ids defining the distinct object instances . Thus , for panoptic 3D reconstruction , we aim to predict the surface geometry of the scene from an image , including any missing regions from occlusion , and for each point on the predicted surface geometry , it must be assigned a semantic label and an instance id . To evaluate a predicted panoptic reconstruction , we then evaluate both the segmentation quality and recognition quality of the 3D object instance geometry and geometry corresponding to structural semantic labels . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We believe that this task not only brings together geometric reconstruction , 3D semantic segmentation , and 3D instance segmentation , but also introduces new algorithmic challenges , and hope that the panoptic 3D scene reconstruction task leads to additional insights in holistic 3D perception . To address the panoptic 3D scene reconstruction task , we propose a new method to jointly reconstruct and segment the observed 3D scene from a RGB image . We first extract features from the image with a 2D convolutional backbone which predicts both depth estimates and 2D instance segmentation . We use the depth estimates to guide the feature lifting to 3D by back-projecting the learned features into a 3D volume of the camera frustum , and develop an instance propagation approach to effectively leverage instance predictions in 2D as seeds for propagation to 3D geometric instances , providing effective object recognition . In contrast to prior approaches for 3D reconstruction from 2D images , we develop a propagation-based approach for object recognition , explicitly bringing 2D predictions to 3D for a refinement rather than direct 3D prediction on a coarse resolution than the 2D signal . We train the object recognition jointly with a reconstruction loss on the predicted scene geometry as well as semantic “ stuff ” labels , resulting in a holistic 3D semantic scene estimate for an RGB image . In summary , our contributions are : • We introduce the task of panoptic 3D scene reconstruction , which aims for holistic 3D scene understanding by jointly predicting scene geometry , semantic labels , and instance ids . • We propose a new approach for panoptic 3D scene reconstruction by learning to lift 2D features to 3D while propagating learned instance information to 3D object understanding for a robust joint prediction of scene geometry , semantics , and object recognition1 . • Our method significantly outperforms alternative approaches which treat reconstruction and semantics separately and/or infer only a subset of the outputs . 2 Related Work . Panoptic Segmentation . Kirillov et al . [ 27 ] proposed the task of panoptic segmentation for 2D images , establishing a unified , holistic scene understanding task for image understanding and a panoptic quality metric for evaluation . The task of panoptic segmentation of a single color image is to assign each pixel in the image with an instance label and a semantic label . The semantic label set is composed of “ stuff ” and “ things , ” where “ stuff ” labels do not consider instance ids as all pixels belonging to the same “ stuff ” class belong to the same instance , and “ things ” labels consider instance ids to distinguish different object instances . This has achieved notable success in inspiring further holistic analysis of image understanding , and we believe such a holistic analysis also plays an important role in 3D perception . We aim to bridge together geometric scene reconstruction [ 37 , 10 ] , 3D semantic segmentation [ 7 , 35 ] , and 3D instance segmentation [ 40 , 20 , 42 , 12 , 26 , 17 , 41 , 29 ] , along with a panoptic 3D scene reconstruction evaluation to capture these characteristics . Narita et al . [ 31 ] proposed an approach for joint semantic and instance segmentation on 3D reconstructed scenes , but rely on given 3D scene information without inferring geometry . To the best of our knowledge , our work is the first to bring the panoptic reconstruction task into 3D perception from RGB images . 1Code can be found at https : //github.com/xheon/panoptic-reconstruction . 3D Semantic Scene Completion and Semantic Instance Completion . Several recent works have explored the task of 3D semantic scene completion , in which both complete scene geometry and semantic segmentation are predicted from partial observations of a scene . From a single RGB-D frame , SSCNet [ 38 ] predicts the geometric scene occupancy and semantic class labels in a volumetric grid . [ 3 ] follows up on it and proposes to guide the semantic scene completion with coarse 3D sketches of the scene . ScanComplete [ 8 ] presented an autoregressive approach for large-scale scan completion and semantic segmentation . Dahnert et al . [ 6 ] proposed to retrieve complete objects of partially scanned objects from a shape database by constructing a joint embedding space of scan and CAD objects . Hou et al . [ 21 ] and Nie et al . [ 33 ] then focused on joint scan completion with instance segmentation of the predicted complete geometry . These approaches have shown the synergies in joint geometric completion along with semantic segmentation or instance segmentation , and we aim to unify all these tasks together for a holistic approach to 3D perception from a single RGB image input . Single-View 3D Reconstruction . Estimating object or scene geometry from only a single RGB image view has been long-studied , with recent advances in deep learning showing significant potential in the reconstruction of shapes from single views [ 4 , 16 , 39 , 30 ] . As intermediate representation Huang et al . extend 2D bounding boxes to 3D via joint estimation [ 23 ] or perspective constraints [ 25 ] . Huang et al . [ 24 ] proposes scene grammar , which allows to retrieve CAD models and estimate scene layout to be optimized to fit a single RGB input image . Shin et al . [ 37 ] and Denninger et al . [ 10 ] recently proposed methods to estimate the scene geometry within the camera frustum of an image view . Recently , Mesh-RCNN [ 15 ] pioneered an approach to extend single object reconstruction to RGB images with multiple objects by detecting and reconstructing the object geometry of each detected object in the image , along with their relative orientations . Kuo et al . [ 28 ] similarly predicted the object geometry for detected objects in an RGB image , but by CAD model retrieval rather than a generative mesh model . CoReNet [ 34 ] omitted explicit object detection and instead extracted 2D information via physically-based ray-traced skip connection between the image and the 3D volume . The approach reconstructs the shape and semantic class of multiple objects from a single RGB image directly in a 3D volumetric grid . Nie et al . [ 32 ] then proposed an approach to detect object poses , their geometry , and cuboid room layouts . Similar to Nie et al. , we aim to more comprehensively understand both object and scene structures and semantics , but remove any cuboid-structure assumptions , instead lifting 2D features and instance information to 3D for prediction of both “ stuff ” and “ things , ” and additionally propose a new holistic evaluation metric to understand the full panoptic 3D scene reconstruction quality . 3 Method Overview . In the panoptic 3D scene reconstruction task , a single RGB image is taken as input , from which the scene geometry within the camera frustum is predicted , along with semantic and instance label for each surface location on the predicted geometry . We introduce a new method for this task , extracting strong 2D features from the input RGB image . The features are further projected and propagated to 3D , in which the final geometry , semantic , and instance predictions are predicted . From an input RGB image , we leverage a state-of-the-art 2D instance segmentation network as a 2D backbone to extract features , with proxy losses for 2D instance segmentation and depth estimation . The predicted depth is then used to estimate a coarse 3D representation by back-projecting the learned 2D features and predictions into a sparse volumetric representation . This coarse 3D estimate is then encoded by 3D sparse convolutions , and a coarse-to-fine approach is used to predict high-resolution output as a sparse volumetric representation , with each voxel containing a distance value representing the surface geometry as a distance field , a semantic label , and an instance id . In particular , we exploit the 2D instance predictions by directly propagating them to 3D to be refined , which provides a robust semantic 3D prediction . 4 Hybrid 2D-3D Panoptic Reconstruction . Learning 2D Priors . An input RGB image I is first encoded by a 2D backbone Θi , realized with a ResNet-18 [ 18 ] architecture . To learn effective priors for the panoptic 3D scene reconstruction task , we employ proxy losses on Θi ( I ) for both 2D instance segmentation and depth estimation . We follow Mask R-CNN [ 19 ] for the instance segmentation , predicting 2D object bounding boxes , class category labels , and instance masks from Θi ( I ) . For depth estimation , we use the depth estimation decoder of [ 22 ] , which predicts depth from multiscale features extracted by Θi , using losses on the logarithm of the depth errors , gradients , and normals . We then extract features from the output of each ResNet block in Θi as well as the decoder outputs , which are then concatenated to θ to be projected to 3D . Lifting 2D Features to 3D . We then lift the 2D features to 3D using the known camera intrinsics of the image and the predicted depth estimates . The 2D features θ , depth estimates , and predicted 2D instance segmentation are back-projected into a 2563 3D volumetric grid using the estimated depth , and encoded as a truncated signed distance field ( TSDF ) along the view direction , where positive values are in front of the predicted surface , zero encodes the predicted surface , and negative values unseen regions . We use a truncation of τ = 3 voxels , and thus use a sparse 3D representation . To spatially correlate learned 2D features to directly inform the 3D panoptic reconstruction , the image features θ as well as instance segmentation mask logits corresponding to each depth estimate copied along the view direction within the truncation , resulting in a sparse 3D feature volume F = ( Fd , Ff , Fi ) as the concatenation of the computed distance field Fd , projected image features Ff , and projected instance masks Fi . Multiple pixels falling into the same voxel bin are randomly sampled . The 1-D instance segmentation mask logits for each object of a maximum of N detected objects are arbitrarily associated with the channels of an N + 1-channel volume , with the extra channel encoding a 1-hot encoding for voxels with no instance mask associations . This enables instance propagation from detected 2D object instances to final refined 3D object “ things , ” explicitly leveraging strong 2D priors which result in more robust panoptic reconstruction than relying only on distinction of objects from a much coarser 3D representation than the information from the high-resolution 2D image . Sparse Generative Panoptic 3D Scene Reconstruction . The panoptic 3D scene reconstruction is then predicted from the feature volume F = ( Fd , Ff , Fi ) with a sparse generative approach . Each of ( Fd , Ff , Fi ) are first separately encoded with sparse 3D convolutions to transform the different features characteristics into a common space . The resulting features are then concatenated as F ′ and encoded together with a sparse 3D encoder using a common sparse encoder Θs , realized with a UNet-style architecture [ 36 ] . Since the panoptic reconstruction should both refine the estimated depth as well as predict new geometry for unobserved regions , Θs follows the same geometric structure as its input F ′ , spatially downsampling by a factor of 4 , while the panoptic reconstruction is predicted in coarse-to-fine fashion , starting with a coarse prediction from ΘS ( F ′ ) using dense 3D convolutions to enable generation of new geometric structures , similar to the sparse generative approach of SG-NN [ 9 ] for geometric scene completion . The initial dense prediction is then decoded to sparse panoptic predictions at each spatial hierarchy level , with separate prediction heads for geometric occupancy , semantic labels , and instance ids , which are then used to inform the next hierarchy level for refined predictions . Similar to [ 9 ] , we also employ dense and sparse skip connections between the encoder and decoder . At the last hierarchy level , we additionally predicted refined surface geometry as a sparse distance field , along with semantic label and instance ids per voxel . The final panoptic 3D scene reconstruction is then obtained by extracting surface geometry from the isosurface of the predicted distance field . Each location with distance to the surface less than τs is assigned the semantic label from the 3D semantic segmentation head if that label corresponds with a “ stuff ” label , and otherwise assigned the semantic label and instance id of the predicted instance mask at the location . Empirically , we found the precedence for “ stuff ” labels to perform slightly more robustly than the reverse , and provide the nearest semantic and instance labels for any locations that may have not had any semantic or instance predictions . Losses . We train our panoptic 3D scene reconstruction with the following loss : L = wdLd + wiLi + ∑ h ( wgLhg + wsLhs + woLho ) , ( 1 ) where Ld denotes the 2D depth estimation loss , Li the 2D instance segmentation , and at each hierarchy level h , a loss for geometry Lhg , semantic label Lhs , and instance id Lho . Note that the loss is not computed for any regions of the 3D predictions that lie outside of the image view frustum . For 2D depth estimation loss Ld , we follow the losses used by [ 22 ] , using a log-encoded ` 1 loss on the predicted depth values along with a cosine normal consistency loss and a gradient loss to regularize the depth estimation . The 2D instance segmentation loss Li follows the standard Mask R-CNN [ 19 ] procedure , with losses on the regressed 2D boxes , class category labels , and binary masks . Our panoptic 3D reconstruction is trained to predict geometry , semantic labels , and instance ids for sparse voxels within the truncation region . For the geometric loss Lhg , each hierarchy level is trained to predict geometry occupancy using a binary cross entropy loss , and the final hierarchy level also employs an ` 1 loss on predicted distance values determining the surface geometry : Lhg = BCE ( Ohpred , Ohgt ) + 1h==0 ‖Spred − Sgt‖1 , where O denote the occupancy predictions and ground truth at the hierarchy level , and for the final hierarchy level h = 0 , S denotes the distance field representing the surface geometry . The occupancy predictions at each level are used to generate the sparse 3D structure used for the sparse 3D convolutions in the following hierarchy level . Semantic labels are trained with a class-weighted cross entropy loss Lhs for each hierarchy level h. 3D instance ids are predicted as binary masks corresponding to the detected objects encoded in the channels of Fi . That is , each detected object informs an instance mask refinement and propagation from the initial projected 2D mask logits . The 2D object masks are correlated with ground truth 2D masks when the IoU > 0.5 . The objects are then arbitrarily ordered to enable a cross entropy loss for the 3D predictions , with the correlated ground truth masks assigned to the arbitrary ordering . A visualization of our instance propagation loss is shown in Figure 3 . This provides a strong prior on 3D objects from the 2D predictions , from which only a refinement must be learned in 3D , resulting in more robust and accurate object recognition .

Summary: This paper defines and presents a method for the novel task of panoptic 3D scene reconstruction, i.e. jointly predicting complete 3D scene geometry, 3D semantic and 3D instance labels. The method consists of using backbone 2D models for depth and instance segmentation and lifting them into 3D before processing with a SG-NN [5] like model to complete and refine the initial volume into instance, semantic and occupancy/TSDF volumes. The authors propose an "instance propagation" method to associate 2D instance masks with 3D instances which is shown to improve over a post hoc "instance clustering" in a 3D semantic only volume. Experiments on a synthetic dataset (3D Front) and Matterport3D show improvement over prior works , some of which tackle a subset of the problem (e.g. only scene structure and semantics).   The primary contribution of this paper is the definition of this new holistic panoptic 3D scene reconstruction task and a new method and evaluation metric for the same with improved performance over prior works which tackle these individual tasks in isolation.