Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image

Inferring 3D locations and shapes of multiple objects from a single 2D image is a long-standing objective of computer vision . Most of the existing works either predict one of these 3D properties or focus on solving both for a single object . One fundamental challenge lies in how to learn an effective representation of the image that is well-suited for 3D detection and reconstruction . In this work , we propose to learn a regular grid of 3D voxel features from the input image which is aligned with 3D scene space via a 3D feature lifting operator . Based on the 3D voxel features , our novel CenterNet-3D detection head formulates the 3D detection as keypoint detection in the 3D space . Moreover , we devise an efficient coarse-to-fine reconstruction module , including coarse-level voxelization and a novel local PCASDF shape representation , which enables fine detail reconstruction and one order of magnitude faster inference than prior methods . With complementary supervision from both 3D detection and reconstruction , one enables the 3D voxel features to be geometry and context preserving , benefiting both tasks . The effectiveness of our approach is demonstrated through 3D detection and reconstruction in single object and multiple object scenarios . Code is available at http : //cvlab.cse . msu.edu/project-mdr.html . 1 Introduction . As a fundamental computer vision task , instance-level 3D scene understanding from a single image has drawn substantial attention from researchers due to its importance in applications such as robotics [ 1 , 2 ] , AR/VR [ 3 ] and autonomous driving [ 4–6 ] . The important 3D properties include 3D bounding box ( pose , size , location ) and 3D shape of object instances . In this work , we aim to design a framework to infer all these 3D properties of multiple objects from a single 2D image . In recent years , various monocular methods are proposed to predict either 3D boxes [ 7–12 ] or 3D shapes [ 13–17 ] . However , only a few studies [ 18–23 ] consider both 3D detection and reconstruction for a total 3D scene understanding . The complexity of real-world scenarios and diverse category variations make it challenging to fully reconstruct the scene context ( both semantics and geometry ) at the instance level from a single image . Moreover , those methods primarily assign 3D semantic labels to pixels . Yet , such a 2D representation with depth ambiguity is insufficient for 3D geometry and context reasoning . It is thus crucial to develop an effective representation of the image that is relevant to 3D geometry and spatial information for performing accurate 3D detection and reconstruction . In light of this , attempts like grid-based representation have been made for tasks such as rendering [ 24 ] , detection [ 25 , 26 ] , or reconstruction [ 27 ] . OFT [ 25 ] proposes to sample and transform image features into a BEV grid representation , which enables holistic reasoning of the 3D scene configuration . CaDDN [ 26 ] extends the BEV grid representation with a categorical depth prior , leading to higher 3D detection accuracies . DeepVoxels [ 24 ] and UCLID-Net [ 27 ] build voxel features by back-projecting 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 2D features to 3D space for respective rendering or single object reconstruction purposes . Inspired by this line of works , we propose a novel voxel-based 3D detection and reconstruction framework for predicting 3D bounding boxes and surfaces of multiple objects from a single image ( see Fig . 1 ) . Specifically , we first divide a 3D scene space into a regular grid of voxels . For each voxel , we assign 3D features by sampling from the image plane via a 2D-to-3D feature lifting operator and the known camera projection matrix . As multiple voxels can be projected to the same position , this leads to similar features along the camera ray and increased difficulty for downstream tasks . To remedy this , we use a positional encoding strategy to make our voxel features position-aware and more discriminative . Based on the intermediate voxel features , we carefully devise our detection and reconstruction modules . For detection , we introduce a novel CenterNet-3D detector head . Instead of formulating the 3D detection as 2D keypoint detection problem as conventional CenterNet-based methods [ 28 , 29 ] , each object is directly represented by its 3D keypoint . Predicting a class-specific 3D heatmap can show probabilities of 3D object centers in the pre-defined voxel space , leading to improved 3D center accuracy . For reconstruction , we propose a multi-level shape representation with two components : coarse-level occupancy representation and fine-level local PCA-SDF representation . The coarse-level voxel grid represents the whole 3D scene with continuous occupancy values . At a fine level , we represent the occupied voxels with a PCA-based signed distance function ( SDF ) by assuming that the local shapes of different voxels are similar either within an object instance , or across different objects . In summary , the contributions of this work include : We propose a novel voxel-based 3D detection and reconstruction framework , which infers the 3D locations and 3D surfaces for multiple object instances with only a 2D image as input . We present a novel CenterNet-3D detector , where each object is represented by its center point in a partitioned 3D grid space . CenterNet-3D avoids estimating depth directly from image features , leading to increased detection performance . We propose a novel local PCA-SDF shape representation , which provides finer reconstruction and order of magnitude faster inference than SOTA local implicit function methods like DeepLS [ 30 ] . We demonstrate the superiority of our method in multiple object 3D reconstruction and detection , as well as 3D shape representation . We assemble a 3D detection and reconstruction benchmark with 18 , 000 real images , annotated with 3D models and bounding boxes of 19 object categories . 2 Related Work . 3D Scene Understanding and Single Object Reconstruction . Tremendous efforts have been devoted to instance-level 3D scene understanding [ 7 , 9 , 18 , 28 , 31–37 ] over the last decade . However , most of these approaches estimate object orientation [ 34 , 35 , 38 ] or 3D bounding boxes [ 7–9 , 39–45 ] . Since describing objects with boxes only offers a coarse information of 3D objects in images , the usage of 3D models as shape priors can complement and enrich 3D scene understanding . Yet , scene understanding at the instance level remains challenging due to the large number of objects with various categories . With the substantial growth in the number of publicly available 3D models , datasets such as ShapeNet [ 46 ] have allowed neural networks to train on the 3D shape reconstruction task from single images [ 47–52 ] . To further leverage real-world images in 3D modeling , as Liu et al . [ 53 ] propose a semi-supervised learning framework for generic objects . However , most of these methods estimate 3D shapes in the object-centric coordinate system , which differs from the shape prediction of multiple instances at scene-level 3D reconstruction – the focus of our work . Multiple Object 3D Reconstruction A common characteristic amongst aforementioned single object 3D reconstruction approaches is that they usually treat objects as isolated geometries without considering the scene context , such as object locations , and instance-to-instance interactions . Recently , there is progress in multiple object 3D reconstruction . 3D-RCNN [ 18 ] exploits the idea of using inverse graphics to map image regions to the 3D shape and pose of object instances . The shape is represented by a simple linear subspace which limits its application for objects with large intra-class variability . Mesh R-CNN [ 21 ] augments Mask R-CNN [ 54 ] with a mesh predictions branch that estimates a 3D mesh for each detected in an image . Total3DUnderstanding [ 19 ] presents a framework that predicts room layout , 3D object bounding boxes , and meshes for all objects in an image based on the known 2D bounding boxes . However , these three methods first detect objects in the 2D image , and then independently produce their 3D shapes with single object reconstruction modules . This could be problematic when 3D boxes of objects intersect , such as a chair is pushed under a table . Recently , CoReNet [ 22 ] performs multiple object reconstructions in a fixed 1283 voxel grid without recovering 3D position information in the world space . Points2Objects [ 23 ] combines a 3D object detector and shape retrieval to detect and reconstruct 3D objects from an image . However , it suffers from two limitations : 1 ) Its CenterNet-based 3D detector reasons 3D boxes directly in the 2D image domain , which is inherently challenging due to the lack of reliable depth cue . 2 ) Retrieval-based methods depend on the size and diversity of the pre-defined CAD model pool . Moreover , [ 22 , 23 ] train on synthetic renderings , which limits their applicability to real-world scenarios . Instead of relying on 2D feature for 3D detection or reconstruction , we propose to learn a geometry and context preserving voxel feature representation , which is well suited for 3D detection and reconstruction . Moreover , we validate our method on real-world images from 19 object categories . Local Shape Priors Many neural architectures are proposed to model 3D objects via geometric representations , e.g. , point clouds [ 55 ] , meshes [ 16 , 56 ] , voxels [ 13 , 57 ] , or implicit functions [ 15 , 58–60 ] . Recently , neural implicit functions have demonstrated their effectiveness by encoding geometry in latent vectors and network weights , which parameterize surfaces through level-sets . Instead of an object-level representation , some follow-up works learn patch-level or primitive-level representations of surfaces , e.g. , PatchNet [ 61 ] , CvxNet [ 62 ] , BSP-Net [ 63 ] . To further leverage local geometric priors , another line of works learn implicit geometry on sparse regular [ 64,65 ] or 3D voxel grids [ 30 , 66 ] . A latent code of each voxel is responsible for representing implicit geometry in a small neighborhood , enabling fine-grained reconstruction . However , these methods often suffer from inefficient inference as each point needs a forward pass through of the implicit function network . Instead , we build a local PCA-SDF shape representation , which represents each local shape as a linear combination of implicit volumetric prototypes , leading to finer details and order of magnitude faster inference than prior works . Similar eigenanalysis of SDF has been applied for either global shape representation [ 67 ] or geometry compression [ 68 , 69 ] . However , none of them develop their algorithms from our perspective of local shape priors , which is motivated from the assumption that local shapes at the voxel level share similarities . 3 Methodology . We illustrate the overall architecture of our method in Fig . 2 , which consists of three key modules : i ) 3D voxel feature learning ; ii ) CenterNet-3D detector ; and iii ) Coarse-to-fine 3D reconstruction . 3.1 3D Voxel Feature Learning . Our network learns to produce a compact 3D voxel feature representation of the image with complementary supervision from both 3D detection and reconstruction , which enables rich 3D context and geometric information and allows the two tasks to benefit each other . We first define a 3D grid V ∈ RX×Y×Z by partitioning of the scene space into voxels Vi with a voxel size of r. The 3D grid size , Xr × Y r × Zr , is set based on the minimum volume of the scenes in the width , height and length dimensions , that can encompass all annotated instances in the dataset . Feature Extraction . We utilize a convolutional feature extractor to generate a hierarchy of multiscale 2D feature maps . Specifically , the input to the feature extraction network is a RGB image I ∈ RWI×HI×3 , where WI and HI are the image size . The convolutions are followed by down scaling the input , creating growing receptive fields and resulting in D-channel multi-scale 2D feature maps F ∈ RWF×HF×D . WF and HF are the width and height of feature F. Lifting 2D Features to 3D . The lifting layer back-projects 2D feature maps F into 3D voxel space , resulting in initial 3D voxel features G. Formally , the objective of the lifting operator is to populate the 3D voxel features G ( x , y , z ) with the projected 2D features F { ( u , v ) } , where { · } denotes bilinear interpolation on the 2D feature maps . We assume a full perspective camera model . Any voxel center ( x , y , z ) can be projected to image plane via a camera projection matrix P ∈ R3×4 : [ u · d , v · d , d ] T = P [ x , y , z , 1 ] T . Here u and v are the 2D position of the projection and d is its depth from the camera . The resulting voxel features G ∈ RX×Y×Z×D provide a scene representation that is free from the effects of perspective projection . 3D Voxel Features Aggregation . The lifting mechanism we use is similar to the one in [ 24 , 25 , 27 ] , which has a major weakness that all voxels along a camera ray will receive the same 2D feature . This feature smearing issue increases the difficulties of 3D detection and reconstruction . To mitigate this issue , we propose to employ the positional encoding ( PE ) [ 70 ] strategy that adds 3D voxel center position to the voxel features : RX×Y×Z×D PE−−→ RX×Y×Z× ( D+3 ) , which helps the voxel features to be more discriminative and position-embedded . We further utilize a 3D convolutional hourglass ( U-Net ) network [ 71 ] , comprised of a series of down- and upsampling convolutions with skip connections , to integrate both local and global information . The final voxel features are thus G : RX×Y×Z× ( D+3 ) U−Net−−−−−→ RX×Y×Z× ( D+3 ) . This voxel features serve as the cornerstone for the downstream tasks of 3D detection and reconstruction . 3.2 Monocular CenterNet-3D Detector Conventional CenterNet-based [ 28 ] monocular 3D detection methods such as Points2Objects [ 23 ] formulate 3D detection as a projected 2D keypoint detection problem . In contrast , we propose a novel CenterNet-3D detection head , where each object is directly represented by its 3D keypoint . Then 3D properties such as object size and orientation can be intuitively inferred from the 3D voxel features at the center location by a regression branch . Compared to [ 23 ] , CenterNet-3D avoids estimating depth values of 3D boxes directly from 2D features , leading to improved detection accuracy . 3D Keypoint Branch . The 3D keypoint branch takes the voxel features G as input and predicts a 3D heatmap Y ∈ RX×Y×Z×C ( see Fig . 2 ) , where C is the number of object categories . The values of each voxel in Y indicates how likely the 3D centroid of a certain object category exists at the voxel center . By computing the local maxima and filtering via a threshold , we obtain a preliminary estimation of the 3D centroids , denoted as c̃3d = [ xc , yc , zc ] T . To remedy the discretization error of voxels , the regression branch additionally predicts a local offset to c̃3d , which is discussed next . Regression Branch . The regression branch predicts the essential properties to construct a 3D bounding box for each voxel of the 3D heatmap . We parameterize a 3D box as the prior works [ 8,19 ] and set up the world system located at the camera center with its vertical ( y- ) axis perpendicular to the floor and its forward ( z- ) axis toward the camera , such as the pitch and roll angles could be included in the camera pose . Specifically the 3D box is encoded as a 8-tuple τ = [ δxc , δyc , δzc , δh , δw , δl , sin θ , cos θ ] . Here ∆c3d = [ δxc , δyc , δzc ] T denotes the 3D center offset compensating voxel discretization . [ l , h , w ] T = [ l̄ · eδl , h̄ · eδh , w̄ · eδw ] T represents the object size , where [ l̄ , h̄ , w̄ ] T is a pre-calculated category-wise average box size , [ δh , δw , δl ] represents the corresponding transformations . θ denotes the rotation angle around y-axis . Here the network estimates the vectorial representation of rotation angle θ [ 72 ] . The output size of the regression branch is thus X × Y × Z × 8 . Given the outputs of keypoint and regression branches , the 3D bounding box B ∈ R3×8 can be restored as 8 corners : B = Rθ [ ±l/2 ±h/2 ±w/2 ] + c3d , c3d = c̃3d + ∆c3d , ( 1 ) where Rθ ∈ R3×3 is the rotation matrix .

Summary: This paper presents a joint framework for detecting multiple 3D objects from input images while also reconstructing the 3D shapes. The proposed method improves upon previous work [20,21] for joint 3D object detection and reconstruction mostly at the system/architectural level. The authors also propose a new way of compactly representing 3D shapes with signed distance functions (SDF) using PCA that can be naturally incorporated into the framework. Experiments are benchmarked with synthetic multi-object ShapeNet renderings, Pix3D, and a new dataset with annotations for multi-object detection & reconstruction.