Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image
Authors: Feng Liu
Xiaoming Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D voxel features
Monocular 3D detection and reconstruction
Multiple objects
Local PCA-SDF

Abstract: Inferring 3D locations and shapes of multiple objects from a single 2D image is a long-standing objective of computer vision. Most of the existing works either predict one of these 3D properties or focus on solving both for a single object. One fundamental challenge lies in how to learn an effective representation of the image that is well-suited for 3D detection and reconstruction. In this work, we propose to learn a regular grid of 3D voxel features from the input image which is aligned with 3D scene space via a 3D feature lifting operator. Based on the 3D voxel features, our novel CenterNet-3D detection head formulates the 3D detection as keypoint detection in the 3D space. Moreover, we devise an efficient coarse-to-fine reconstruction module, including coarse-level voxelization and a novel local PCA-SDF shape representation, which enables fine detail reconstruction and two orders of magnitude faster inference than prior methods. With complementary supervision from both 3D detection and reconstruction, one enables the 3D voxel features to be geometry and context preserving, benefiting both tasks. The effectiveness of our approach is demonstrated through 3D detection and reconstruction on single-object and multiple-object scenarios. 

Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image
Feng Liu Xiaoming Liu Department of Computer Science and Engineering Michigan State University, East Lansing MI 48824 {1iufeng6, liuxm] @msu. edu
Abstract
Inferring 3D locations and shapes of multiple objects from single 2D image is a long-standing objective computer vision. Most of the existing works either predict one of these 3D properties or focus solving both for single object. One fundamental challenge lies in how to learn an effective representation of the image that is well-suited for 3D detection and reconstruction. In this work, propose to learn a regular grid of 3D voxel features from the input image which aligned with 3D scene space via 3D feature lifting operator. Based the 3D voxel features, our novel CenterNet-3DD detection head formulates the 3D detection as keypoint detection in the 3D space. Moreover, devise an efficient coarse-to-fine reconstruction module, including coarse-level voxelization and a novel local PCA- SDF shape representation, which enables fine detail reconstruction and one order of magnitude faster inference than prior methods. With complementary supervision from both 3D detection and reconstruction, one enables the 3D voxel features be geometry and context preserving, benefiting both tasks. The effectiveness of our approach is demonstrated through 3D detection and reconstruction in single object and multiple object scenarios. Code is available at http: //tp:///c.ll.//.. cse. msu. edu/ www/projeceeeeceee wwwppcttttaaoeeeooooop mdd html.
1 Introduction
As a fundamental computer vision task, instance-level 3D scene understanding from single image has drawn substantial attention from researchers due importance in applications such as robotics [1, AR/VR [3] and autonomous driving [4-6]. The important 3D properties include 3D bounding box (pose, size, location) and 3D shape of object instances. In this work, aim to design framework to infer all these 3D properties of multiple objects from single 2D image.
In recent years, various monocular methods proposed to predict either 3D boxes [7-12] 3D shapes [13-17]. However, only a few studies [18-23] consider both 3D) detection and reconstruction for total 3D scene understanding. The complexity of real-world scenarios and diverse category variations make it challenging to fully reconstruct the scene context (both semantics and geometry) at the instance level from single image. Moreover, those methods primarily assign 3D semantic labels to pixels. Yet, such a 2D representation with depth ambiguity is insufficient for 3D geometry and context reasoning. is thus crucial develop an effective representation of the image that relevant to 3D geometry and spatial information for performing accurate 3D detection and reconstruction.
In light of this, attempts like grid-based representation have been made for tasks such as rendering [24]] detection [25,26], or reconstruction [27]. OFT [25] proposes to sample and transform image features into BEV grid representation, which enables holistic reasoning of the 3D scene configuration. CaDDN [26] extends the BEV grid representation with a categorical depth prior, leading to higher 3D detection accuracies. DeepVoxeh: [24] and UCLID-Net [27] build voxel features by back- projeetijg
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Given single image as input, our proposed approach jointly predicts 3D object bounding boxes and surfaces.
Figure 1:
2D features to 3D space for respective rendering or single object reconstruction purposes. Inspired by this line of works, we propose novel voxel-based 3D detection and reconstruction framework for predicting 3D bounding boxes and surfaces of multiple objects from a single image (see Fig. 1).
Specifically, first divide 3D scene space into regular grid of voxels. For each voxel, assign 3D features by sampling from the image plane via a 2D-to-3D feature lifting operator and be projected the same position, this
the known camera projection matrix. As multiple voxels leads to similar features along the camera ray and increased difficulty for downstream tasks. To remedy this, use positional encoding strategy to make our voxel features position-aware and more discriminative. Based on the intermediate voxel features, we carefully devise our detection and reconstruction modules. For detection, introduce novel CenterNet-3D detector head. Instead of formulating the 3D detection as 2D keypoint detection problem as conventional CenterNet based methods [28, 29], each object directly represented by its 3D keypoint. Predicting class-specific 3D heatmap can show probabilities of 3D object centers in the pre-defined voxel space, leading to improved 3D center accuracy. For reconstruction, we propose a multi-level shape representation with two components: coarse-level occupancy representation and fine-level local PCA-SDF representation. The coarse-level voxel grid represents the whole 3D scene with continuous occupancy values. fine level, we represent the occupied voxels with a PCA-based signed distance function (SDF) by assuming that the local shapes of different voxels are similar either within an object instance, or across different objects.
In summary, the contributions of this work include:
We propose novel voxel-based 3D detection and reconstruction framework, which infers the 3D locations and 3D surfaces for multiple object instances with only 2D image input.
We present novel CenterNet-3D detector, where each object represented by its center point in partitioned 3D grid space. CenterNet-3D avoids estimating depth directly from image features, leading to increased detection performance.
We propose novel local PCA-SDF shape representation, which provides finer reconstruction and order of magnitude faster inference than SOTA local implicit function methods like DeepLS [30].
We demonstrate the superiority of our method in multiple object 3D reconstruction and detection, well 3D shape representation. We assemble 3D detection and reconstruction benchmark with 18, 000 real images, annotated with 3D models and bounding boxes of object categories.
2 Related Work
3D Scene Understanding and Single Object Reconstruction. Tremendous efforts have been devoted to instance-level 3D scene understanding [7,9, 9, 18, 24,28,31-37 over the last decade. However, most of these approaches estimate object orientation 33,,35,38) 38] or 3D bounding boxes [7-9,39-45) Since describing objects with boxes only offers a coarse information of 3D objects in images, the usage of 3D models as shape priors can complement and enrich 3D scene understanding. Yet, scene understanding at the instance level remains challenging due to the large number of objects with various categories. With the substantial growth in the number of publicly available 3D models, datasets such as ShapeNet [46] have allowed neural networks to train on the 3D shape reconstruction task from single images [47-52]. To further leverage real-world images in 3D modeling, as Liu et al. [53] propose semi-supervised learning framework for generic objects. However, most of these methods estimate 3D shapes in the object-centric coordinate system, which differs from the shape prediction of multiple instances at scene-level 3D reconstruction the focus of our work.

---

Multiple Object 3D Reconstruction A common characteristic amongst aforementioned single object 3D reconstruction approaches is that they usually treat objects as isolated geometries without considering the scene context, such as object locations, and instance-to-instanee interactions. Recently, there is progress in multiple object 3I) reconstruction. 3D-RCNN [18] exploits the idea of using inverse graphics to map image regions to the 3D shape and pose of object instances. The shape is represented by a simple linear subspace which limits its application for objects with large intra-class variability. Mesh R-CNN [21] augments Mask R-CNN [54] with mesh predictions branch that estimates a 3D mesh for each detected in an image. Total3Dnnderstanding [19] presents a framework that predicts room layout, 3D object bounding boxes, and meshes for all objects in image based on the known 2D bounding boxes. However, these three methods first detect objects in the 2D image, and then independently produce their 3D shapes with single object reconstruction modules. This could be problematic when 3D boxes of objects intersect, such as a chair is pushed under a table.
Recently, CoReNet [22] performs multiple object reconstructions in a fixed 128 voxel grid without recovering 3D position information in the world space. Points2Objects [23] combines a 3D object detector and shape retrieval to detect and reconstruct 3D objects from an image. However, it suffers from two limitations: 1) Its CenterNet-basee 3D detector reasons 3D boxes directly in the 2D image domain, which is inherently challenging due to the lack of reliable depth cue. 2) Retrieval-based methods depend on the size and diversity of the pre-defined CAD model pool. Moreover, [22, [22,23] train on synthetic renderings. which limits their applicability relying on 2D feature for 3D detection or reconstruction, we propose to learn geometry and context preserving voxel feature representation, which is well suited for 3D detection and reconstruction. Moreover, we validate our method on real-world images from 19 object categories.
real-world scenarios. Instead of
Local Shape Priors Many neural architectures are proposed to model 3D objects via geometric representations, e.g., point clouds [55], meshes [16,56], voxels [13,57], or implicit functions [15, 58-60J. Recently, neural implicit functions have demonstrated their effectiveness by encoding geometry in latent vectors and network weights, which parameterize surfaces through level-sets. Instead of an object-level representation, some follow-up works learn patch-level primitive-level representations of surfaces, e.g., PatchNet [61], CvxNet [62], BSP-Net [63]. To further leverage local geometric priors, another line of works learn implicit geometry sparse regular [64,65] or 3D voxel grids [30,66] 66]. A latent code of each voxel is responsible for representing implicit geometry in a small neighborhood, enabling fine-grained reconstruction. However, these methods often suffer from inefficient inference as each point needs a forward pass through of the implicit function network. Instead, we build a local PCÁ-SDF shape representation, which represents each local shape linear combination of implicit volumetric prototypes, leading to finer details and order of magnitude faster inference than prior works. Similar eigenanalysis of SDF has been applied for either global shape representation [67] or geometry compression [68, 69]. However, none of them develop their algorithms from our perspective of local shape priors, which motivated from the assumption that local shapes at the voxel level share similarities.
3 Methodology
We illustrate the overall architecture of our method Fig. 2, which consists three key modules: i) 3D voxel feature learning; ii) CenterNet-3D detector; and iii) Coarse-to-fine 3D reconstruction.
3.1
3D Voxel Feature Learning
Our network learss to produce compact 3D voxel feature representation of the image with com- plementary supervision from both 3D detection and reconstruction, which enables rich 3D context and geometric information and allows the two tasks to benefit each other. We first define a 3D grid V X X2 by partitioning of the scene space into voxels V, with a voxel size of r. The 3D grid size, Xr x Yr x Zr, is set based on the minimum volume of the scenes in the width, height and length dimensions, that can encompass all annotated instances in the dataset.
Feature Extraction. We utilize a convolutional feature extractor to generate a hierarchy of multi- scale 2D feature maps. Specifically, the input to the feature extraction network is a RGB image I IR. x where W, and H, are the image size. The convolutions are followed by down scaling the input, creating growing receptive fields and resulting in D-channel multi-scale 2D feature maps F e I W x WF and Hp are the width and height of feature F.
3

---

Imuge Ie B
Feature Maps F ReysHeeD
Voxel Features Ge R
Delection
Revonsbuction
2D-to-3 Feature Lifting
Feature Backbone
CenterNet-3D
Coarse-to-Fing 3D
Detector
Reconstruction
Classification
Coarsc-level Voxelizatinn
3D U-Net
|
3D Heatmap
3D Lifting
Positional Encuding
Camera
3D Box
Calibration
Regression
Local PCA-SDF Representation
Samplerr
Comertletutio Openter
Selection Operntor
Σ Lineur Combitution
Local PCA SDF Rusee
Figure Overview of our approach. The proposed joint framework is composed of three key modules: 3D voxel feature learning (consists of feature backbone and 2D-to-3D feature lifting), CenterNet-3D detector, and coarse-to-fine 3D reconstruction. 2D feature maps are first generated from input image I, which are back-projected into voxel features G using a known camera projection matrix P. The voxel features serve for our novel 3D object detection and reconstruction.
Lifting 2D Features to 3D. The lifting layer back-projects 2D feature maps F into 3D voxel space, resulting in initial 3D voxel features G. Formally, the objective of the lifting operator is to populate the 3D voxel features G(I, 2) with the projected 2D features F( v)}, where {-} denotes bilinear interpolation on the 2D feature maps. We assume full perspective camera model. Any voxel center (x. y, 2) can be projected to image plane via camera projection matrix P E 33x4, [u d, u.dd d, dj" = P x. y, 2, 1)². Here u and v are the 2D position of the projection and d is its depth from the camera. The resulting voxel features G X provide free from the effects of perspective projection.
representation that
3D Voxel Features Aggregation. The lifting mechanism we use is similar to the one in [24, 25, 25,27], which has major weakness that all voxels along a camera ray will receive same 2D feature. This feature smcaring issue increases the difficulties of 3D detection and reconstruction. To mitigate this issue, we propose to employ the positional encoding (PE) [70] strategy that adds 3D voxel
center position to the voxel features: IxXXXx2xD PE XXXY (D+33))) which helps the voxel features to be more discriminative and position- position-embeddee embedded. We further utilize a 3D convolutional hourglass (U-Net) network [71], comprised of series of down- and upsampling convolutions with skip connections, to integrate both local and global information. The final voxel features are thus
G : xxxxx2x(D+3) U-N xXxxxxx(D+x)) This voxel features serve as the cornerstone for the downstream tasks of 3D detection and reconstruction.
3.2 Monocular CenterNet-3D Detector
Conventional CenterNet-based [28] monocular 3D detection methods such as Points2Objects [23] formulate 3D detection as projected 2D keypoint detection problem. In contrast, we propose a novel CenterNet-3D detection head, where each object is directly represented by 3D keypoint. Then 3D properties such as object size and orientation can be intuitively inferred from the 3D voxel features at the center location by regression branch. Compared [23], CenterNet-3D avoids estimating depth values of 3D boxes directly from 2D features, leading improved detection accuracy.

---

Voncel
Vovel
SDF
x(z
(2,
Local PCA-SDE
(b)
Figure 3: 2D examples of (a) DeepSDF [59], (b) DeepLS [30], and (c) our local PCA-SDF shape representation. DeepSDF describes the surfaces with global shape codes. The SDF function in DeepLS outputs scalar value conditional on the local latent code and local coordinate However, its inference is computationally expensive since it requires forward pass through f for every Our shape representation consists of coarse-level voxelization and fine-level local PCA-SDF. The coarse-level voxelization holistically represents the whole 3D surface with binary values. To further represent fine-level surfaces, we propose novel local PCA-SDF model, representing any occupied voxel as linear combination of regular SDF function bases, which enables a more efficient and accurate representation than DeepL..
3D Keypoint Branch. The 3D keypoint branch takes the voxel features G input and predicts 3D heatmap y E IR* x2xC (see Fig. 2), where C' is the number of object categories. The values of each voxel in y indicates how likely the 3D centroid of a certain object category exists at the voxel center. By computing the local maxima and filtering via threshold, we obtain a preliminary estimation of the 3D centroids, denoted as C3d = [Te, ye: že] To remedy the discretization error of voxels, the regression branch additionally predicts local offset to Čअd, which is discussed next.
Regression Branch. The regression branch predicts the essential properties to construct a 3D bound- ing box for each voxel of the 3D heatmap. We parameterize a 3D box as the prior works [8, 19] and set the floor and its forward (z-) axis toward the camera, such as the pitch and roll angles could be included in the camera pose. Specifically the 3D box encoded
up the world system located the camera center with its vertical (y-) axis perpendicular
8-tuple 7 [d._ Ey 8z Ôh, dw. 6, sin 0, cos 0). Here AC3d 16 oyer ō=,]² denotes the 3D center offset compensating voxel discretization. [t, h, w|" [T-e0t h. eh e represents the object size, where [7, h, w7 is pre-caleulated category-wise average box size, [6n, ow. 6,] represents the corresponding transformations. 0 denotes the rotation angle around y-axis. Here the network estimates the vectorial representation of rotation angle 0 [72]. The output size of the regression branch is thus X x Y x Z x 8. Given the outputs of keypoint and regression branches, the 3D bounding box B E R can be restored corners:
2
B Re
hh/2
+ C3d:
C3d
Cad + Acad.
2
where Rs E 1g3x3 is the rotation matrix.
3.3 Coarse-to-Fine 3D Reconstruction
Our reconstruction module is based on a coarse-to-fine shape representation, which consists of two components: coarse-level voxelization and fine-level local PCA-SDE.
Coarse-Level Voxelization. Based on the extracted 3D voxel features G. we first estimate a coarse-level voxelization V by specific branch. The coarsc-level voxelization holistically represents the whole 3D surface with binary occupancy values, where the unoccupied voxels cover "air" the scene, and occupied voxels can be either fully occupied ones inside the object, voxels intersecting with the object's 's surface. For the occupied voxels of both types, we further reconstruct a fine-level local shape via the local PCA-SDE.
Local PCA-SDF Shape Representation. Recent works such as DeepSDF [59] aims to learn global implicit functions to represent shapes (see Fig. 3(a)). However, representing the entire objects with single latent code often results in loss of details, which limits application for scene-level object reconstruction. DeepLS [30] represents 3D surfaces by set of independent latent codes on a regular
5

---

grid (see Fig. 3(b)). Each latent code 2,, concatenated with any point location x, can be decoded into a SDF value s; by the learned implicit network f: s; = fz). However, this shape representation has two limitations. i) Inference is inefficient (in the order of seconds) since every point of a test voxel (e.g. 256 points) required to be sent to f for SDF calculation, making unsuitable for real-time applications. ii) Our key observation is that local voxels, either within an object instance or across different categories, share similar local shapes, e.g., voxels across a table's surface all have planar shapes. However, DeepLS treats voxels independent training samples, without fully leveraging such local shape priors in training. To address these issues, we propose a novel local PCA-SDF shape representation, which represents each voxel shape a linear combination of a set of implicit volumetric prototypes, leading to significantly finer reconstruction and 10x faster inference speed than DeepLS (see Tab. 4).
Formally, as shown in Fig. 3(c), for each occupied voxel V. we define a regular lattice q E R4xxxkx3 and compute their SDFs s E [g toward the surface. By collecting SDFs of Ns occupied weapply Principal Component Analysis (PCA) to find lg (lg << k")) local shape bases, SB E pxxxxkxh, As such, given the learned SB and the latent code 2, any local shape S; of the underlying surface can be implicitly represented by S; = SgZ, The latent code z, for S; can be generated by the corresponding voxel feature G; via mapping network, implemented with two fully-commectee layers. By combining the contributions of all the occupied voxels, we infer a global iso-surface from the SDF field. Similar to DeepLS [30], we apply a 1.5 times receptive field strategy to mitigate the inconsistent surface predictions at the voxel boundaries (Fig. 3())). Accordingly, during inference, we could apply average pooling to combine the SDF values for the boundary area. It is worth mentioning that our local PCA-SDF also allows reconstruction at resolutions higher than the one used during training by simply applying trilinear interpolation on the learned local shape bases SB.
voxels from the training surfaces, we apply
MLP(G.). MLP is
3.4 Loss Functions and Implementation Details
The training data of one example consists of RGB image I, ground-truth 3D bounding boxes B of objects, coarse-level voxelization V", and a set of regular SDF pairs {(idx,,s)) 8,)) sampled from the surface. Here, each set of SDFs s, R idx, is the idx,-th voxel of the holistic grid. During training, we jointly optimize the parameters of 2D feature extraction network, 3D U-Net, detection and reconstruction modules by minimizing three losses: 3D keypoint classification loss Lats, regression loss Lreg, and 3D reconstruction loss Lyenons i.e.,
L = + freg + Lreeon-
Loss Functions. We generate the target heatmaps y by splatting the ground truth 3D center points using a Gaussian kernel (please refer to Supp for details). two Gaussians of the same class overlap, we take the element- wise maximum. The 3D keypoint branch is trained with penalty-reduced focal loss [28,73 73] in a point-wise manner on the 3D heatmap.
(1 -1-yes --yyz>>> (log((iryz)))
if y
Viyze =
Σ sysc -2 ) ()ayze) log(1 y Mryze)
Cd.= N
otherwise
where N is the number of objects per image, μ = 2 and σ 4 are
of the focal loss.
We define the 3D bounding box regression loss as the L, distance between the predicted transform B and the ground truth B²: freg x IB 1=1 The reconstruction loss consists of cross-entropy classification loss En for coarse-level voxelization and fine-level SDF regression loss.
K
Crecom Lo +
Σ
||a2 ss
z, = MMP((Gixx))
(4)
Implementation Detail. We use ResNet-34 network as our 2D feature extractor. We extract features immediately before the final three downsampling layers, resulting in a set of feature maps at 1/8, 1/16, 1/32 scales of the input resolution. We then resize them to the original size of the input kernels are used to map these
images via bilinear interpolation. Convolutional layers with feature maps to common channel size of 64. The set of feature maps is summarized as F before

---

Table 1:
Multiple object reconstruction comparison. We report per-class and mean IoU over all classes, and class-agnostic global loU on 128 voxel grid.
ShapeNet-triplets
ShapeNet-pairs
Method
bottle
bowl
chair
mug
sofa
table
global
global
CoReNet (22)
61.8
36.2
30.1
48.0
52.9
34.8
43.9
49.8
43.1
52.7
Points2Ohjects [23]
63.5
30.2
18.9
41.5
44.5
19.8
36.4
44.7
Proposed
63.3
38.5
31.8
51.7
54.3
36.1
52.3
46.7
55.1
Imput
CoReNet
Proposed
Figure 4: Qualitative results on ShapeNet-triplets dataset. We compare to CoReNet [22] in two different viewpoints. Our model more accurately reconstructs details and hallucinates occluded parts.
passing to 3D lifting layers. For the main experiments, we train our models with batch size of 8 on a GTX 1080Ti GPU for 200 epochs. The learning rate is set 10-1 and drops at and 100 epochs by a factor of 10. For more details, please refer to Sec. 4 or Supp.
Experiments
4.1 Multiple Object Detection and Reconstruction on ShapeNet-pairs and -triplets
Following the experimental setting of CoReNet [22] and Points2Objects [23], we evaluate multiple object detection and reconstruction on ShapeNet-puiry and ShupeNet-triplets datasets [22]. These datasets contain 256 x 256 photorealistic renderings of either pairs or triplets of ShapeNet [46] objects placed on a ground plane with random scale, rotation, and camera viewpoint. The Shape:Net- pairs has several pairs of object classes: bed-pillow, bottle-bowl. nottle-mug, chair-table, display-lamp guitar-piano, motorcycle-co and sofa-table, which contains 365, 600 images on trainval and 91, 200 on test. The ShapeNetttripletss is with bottle-bowl-mug and chair-sofa-table which includes 91, 400 on trainval and 22, 000 on test.
Datasets.
Experimental Settings. In this experiment, we set a voxel grid of size XxYx = 25 = 0.1), which is sufficient enclose all objects in the datasets. We randomly select 200 surfaces from the training set generate Ng Ry 200, 000 occupied voxels. In this experiment, we set & 11, -SaapsNet-aiirs.. We compare with SOTA methods for multiple object reconstruction: CoReNet [22] and Points2Objects [23]. As CoReNet doesn't perform 3D detection, we only compare with Points2Objects on detection. Following [23], we use mean average precision (mAP) the detection metric with 3D box intersection-over-und over-unionnnnn union (IoU) thresholds 0.25 and 0.5. Following [22,23], the metric for reconstruction IoU on a 128 voxel grid.
IB = 64, C (ShapeNet-triplets) or C
Results. We first report the 3D detection results. Our method achieves higher detection accuracy than Points2Objects [23]: 51.5% vs. 48.6% (threshold@0.5) and 80.3% vs. 77.2% (threshold@00.2 25)., which demonstrates that voxel features perform better than image-based features for monocular 3D detection. For reconstruction, we report mean over the per-class IoU, as well as the global IoU of all object instances within a scene, which does not concern predicted class labels. As compared in Tab. 1, our method significantly outperforms two SOTA baselines on both datasets. On ShapeNet- triplets, method achieves relative 5.0% global IoUs gains while 4.8% mean IoUs gains, which
7

---

Input Image
Points2Ohjeets
Prepesed
Percestoge zsplained variation
100
Transtation
41.r
-UFFF
Offr
0.
80
BUASN
01227
00231
0/0215
00899
60
Rotalion
0136
011244
40
16m
0.32m
2 0.306
01211
0.64m
20
5
00MB
00211
01220
0.256
(a)
(b)
(c)
Figure 5: (a) Qualitative comparison on Pix3D. Our reconstructions closely match objects' genuine shape, e.g., the table legs and chair arms. (b) Explained variation of our PCA-SDF representation with three voxel sizes r. (c) Reconstruction errors (Chamfer Distance-L2) of PCA-SDF w.r.t. translated
and rotated 3D shapes.
Input
CoReNet
Proposed
Ground-truth
Qualitative results on real images from ScanNet-MDR. Our reconstructions closely match the objects than CoReNet [22]. Moreover, our method performs better reconstruction of the truncated objects.
Figure 6:
indicates that model performs well reconstructing the overall shapes of objects. Qualitative results of detection and reconstruction are shown in Fig. 4.
4.2 Single Object Reconstruction on Pix3D
We further compare CoReNet [22] and Points2Objects [23] on the real image database, Pix3D [74] in the same protocol (splits Sj and S2) as in [21]. In this experiment, we train our model with the same experimental setting and the same pre-computed PCA-SDF bases as in Sec. 4.1. On average IoU over all 9 object classes, we achieve 38.6% us. 34.1% (CoReNet) us. 33.3% (Points2Objects) on Si and 28.6% us. 26.3% (CoReNet) vs. 23.6% 3.66% (Points2OOjects) on S2. The results demonstrate that our approach improves over baselines on real images. Qualitative results are shown in Fig. 5(a).
4.3 Multiple Object Detection and Reconstruction on ScanNet-MDR
Dataset. Since there is no benchmark providing both 3D CAD models and 3D bounding boxes for multiple objects within single real image, assemble dataset with 18, 000 real images from the

---

Table 2:
Comparisons of 3D object detec- tion and reconstruction on ScanNet-MDR dataset. [Key: =CenterNee, 2 DCCeneerNet..D), D=DeepLS, Local PCA-SDF)
Table 3: Effect of the voxel size and latent code size Lg in detection and reconstruction on ScanNet-MDD dataset (mAP//oU), and inference time per image.
Method
Detection
Recon,
Evaluation
!
mAP (@0.15)
IoU
ln
16
32
61
Time
(ma)
CoReNet [22)
35.2
17.5/
17.7/
18.1/
Proposed-1
36.5
0.64m
44.2
Proposed-2
20.7
30.7
31.1
31.4
Proposed-3
/
19.5
36.9
0.32m
19.1/
18.8/
18.6/
55.6
Proposed-4
/
20.0
35.4
34.1
34.8
35.3
Proposed-w/o
21.2
37.2
21.9/
22.2/
22.8/
Proposed
22.8
38.2
0.16m
37.1
37.6
38,2
78.1
ScanNet [75], termed ScanNet Monocular Detection and Reconstruction (ScanNet-MDR) dataset. For each object in an image, its CAD model is produced by [76]. We then generate the corresponding 3D bounding box label and camera calibration matrix. Unlike the ShapeNet-pairs or ShapeNet-triplett datasets, all the 3D objects in ScanNet-MDR are at absolute scale. Additionally, this dataset contains greater diversity including 19 object categories (bag, basket, bathtub, bed, bench, bookshelf, cabinet, chair, display, file, lamp, microwave, piano, printer, sofa, stove, table, trash and washer). We split the data into 80% for training and 20% for testing.
Experimental Settings. In this dataset, we use voxel grid of size 10.28 x 3.2 x 6.4m (X = 64, Y = 20, Z 40, r = 0.16m), a minimum to encompass all annotated 3D objects in the dataset. The PCA-SDF is pre-computed with Ng = 560, 000 occupied voxels from 200 training surfaces. We set k 17, C 19 and [g 64. For comparison, we train CoReNet [22] using the released code on our training data. We use mAP with 3D box IoU threshold of 0.15 as detection metric [19], and global 3D IoU on 128 voxel grid as reconstruction metric.
Results and Ablation Studies. We report detection and reconstruction results on the testing set. As shown in Tab. 2, our method significantly improves over CoReNet [22] on 3D reconstruction and advances the ablated versions on both detection and reconstruction. Qualitative results are shown in Fig. 6. CoReNet, as an image-to-vore reconstruction network without special design for feature smearing issue, cannot handle truncated objects in the input image.
Joint Framework us. Separate Modules. Moreover, Tab. shows the ablation results of our models without detection (Proposed-1) or reconstruction (Proposed-2) modules, where one can conclude that joint framework in this work performs better than solving either task exclusively.
CenterNet-3D and PCA-SDF. To further validate the effectiveness of the proposed CenterNet-3D detector over the conventional CenterNet, we train model (Proposed-3) by combining the recon- struction module with the conventional CenterNet, which formulates 3D detection as problem of 2D keypoint detection directly from the pixel-based image features. As compared in Tab. 2, our model outperforms Proposed-3 both detection and reconstruction. To compare PCA-SDF with DeepLS [30] in the joint detection and reconstruction framework, we train model (Proposed-4) by using DeepLS representation as our fine-level reconstruction module. Tab. 2 shows that both detection and reconstruction performances are worse than ours.
Effect on Voxel Size r Latent Code Size lp. The validity of local shape pair, expressed by PCA-SDF in our work, depends on the voxel size. For instance, we show the percentage of explained variation for three voxel sizes in Fig. 5(b). As larger voxel sizes are used, the first few bases could explain less variation, due to the diminished local shape similarity among larger voxels, Le.. weakened local shape prior. This is also validated by the ablation voxel size r and latent code size lp in Tab. 3, where larger voxel sizes lead to lower detection and reconstruction accuracies. On the other hand, larger latent code size results in better representation power (Fig. 5(b) but not necessarily reconstruction, since it imposes more challenging task for the network to predict higher-dim code.
Effect on Positional Encoding. To investigate the effect of positional encoding operator on 3D detection and reconstruction. we retrain model without the positional encoding (Proposed-w/o PF). As compared in Tab. 2, both detection and reconstruction accuracies are worse than ours, which indicates that the positional encoding indeed enhances the voxel feature representation.

---

Table 4: Comparison of reconstructing 3D shapes from ShapeNet test set, evaluated by Chamfer Distance- I.2 (multiplied by 103). PCA-SDE achieves higher accuracy and efficiency than Deepl. even with fewer decoder and representation parameters. Decoder para. refer to the decoder network parameters for DeepSDF or DeepLS,, and PCA bases for our PCA-SDE, [Key: Best, Second Best]
#Decoder Pura. (M)
#Represent. Paru. (K)
Inference
Method
Chair
Plane
Table
Lamp
Sofa
Mean
Unseen
Time
DeepSDF [59]
0:204
0.143
0.553
0.832
0.132
0.372
1.8
0.3
6.9626
Deepl.8 S [30] (g=125)
0.030
0.018
0.032
0.078
0.044
0.040
0.05
4096
0.8081
PCA-SDF (pg 32)
0.031
0.016
0.033
0.035
0.032
0.029
0.111
0.02
1049
0.0126
PCA-SDF (g=64)
0.027
0.012
0.030
0.027
0.030
0.025
0.059
0.05
2097
0.0129
PCA-SDF (25)
0.026
0.010
0.029
0.016
0.029
0.022
0.028
0.09
4096
0.0132
Computation Time. Tab. validates our inference time per image with different voxel sizes GTX
1080Ti GPU. Since & does not affect the runtime much, we show the average time across three lg.
4.4 3D Shape Representation Power of PCA-SDF
While PCA-SDF has demonstrated its advantage in 2D to 3D reconstruction, this is rooted from its ability in representing 3D shapes. To quantify its 3D shape representation power, we design the following experiment and compare with DeepSDF [59] and Deepl.S [30], without involving 2D image inputs. Following the setting of [30], we utilize 1,000 ShapeNet shapes (200 each from categories) to compute our PCA-SDF bases. Fach 3D shape is split by x2x32x32 grids (r ) During training. both DeepSDF and DeepLS optimize the latent codes and decoder weights through backpropagation, to best represent the training shapes. In inference, decoder weights are fixed, and the optimal latent code is estimated given a testing shape. In contrast, we compute the latent codes whose multiplication with PCA-SDF bases can best approximate the ground-trum -trth SDF.
We evaluate 3D shape reconstruction accuracy on various categories. As shown in Tab. 4, PCA-SDF has lower reconstruction error than DeepLS, even with smaller number of representation parameters. Moreover, our inference is 10x more efficient (infer at 256 resolution) which meets the real-time requirement for downstream tasks.
To study whether the PCA-based representation is sensitive to tiny geometry perturbation, we apply minimal translation (10..r, #0.05r) and rotation (+t4", +22) to testing surfaces of the 5 categories and evaluate surface reconstruction error on these data, while no data augmentation was applied to the training data of shape bases computing. As shown in Fig. 5(c), the error is stable in very small range, which illustrates that PCA-SDF is robust to translation and rotation variations.
Generalization to Unseen Category. In order to investigate the generalization of PCA-SDF, we design an experiment compute PCA-SDF from single category (200 shapes), and reconstruct 3D shapes from the other four unseen categories. We repeat the training/testing 5 times across 5 categories. As reported in Tab. 4, PCA-SDF trained on unseen categories achieves comparable performance (0.028 vs. 0.022) with the one trained on seen categories when [B = 125, which indicates that local 3D shapes the voxel level are indeed similar each other, even across different categories.
5 Conclusion
We present voxel-based 3D detection and reconstruction framework for predicting 3D bounding boxes and shapes of multiple objects from a single image. Specifically, we first learn a regular grid of 3D voxel features for the input images. Based on the voxel features, we devise a novel CenterNet-3D detector to detect and regress 3D bounding boxes in the 3D space. With a coarse-level voxelization and fine-level local PCA-SDF representation, our reconstruction module provides highly efficient and accurate reconstructions. The comprehensive experiments show the superiority of the proposed method 3D detection and reconstruction, as well as shape representation power. The same as CoReNet and Points2Objects, one limitation of our approach is that it requires the camera calibration matrix as input which might limit its application to real images. Therefore, one future direction is to invest the necessity of this requirement and/or integrate with auto-calibration methods.
10

---

References
[1] Joris Guerry, Alexandre Boulch, Bertrand Le Saux, Julien Moras, Aurélien Plyer, and David Filliat. Snapnet-r: Consistent 3D multi-view semantic labeling for robotics. In ICCV, 2017.
[2] Keisuke Tateno, Federico Tombari, Iro Laina, and Nassir Navab. CNN-SLAM: Real-time dense monocular slam with learned depth prediction. In CVPR, 2017.
[3] Lei Han, Tian Zheng, Yinheng Zhu, Lan Xu, and Lu Fang. Live semantic 3D perception for immersive augmented reality. TVCG, 2020.
[4] Aseem Behl, Omid Hosseini Jafari, Siva Karthik Mustikovela, Hassan Abu Alhaija, Carsten Rother, and Andreas Geiger. Bounding boxes, segmentations and object coordinates: How important is recognition for 3D scene flow estimation in autonomous driving scenarios? In
ICCV, 2017.
[5] Yiping Chen, Jingkang Wang, Jonathan Li, Cewu Lu, Zhipeng Luo, Han Xue, and Cheng Wang. LiDAR-video driving dataset: Learning driving policies effectively. In CVPR, 2018.
[6] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the
KITTI vision benchmark suite. In CVPR, 2012.
[7] Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular 3D object detection for autonomous driving, In CVPR, 2016.
[8] Siyuan Huang, Siyuan Qi, Yinxue Xiao, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Coop- erative holistic scene understanding: Unifying 3D object, layout, and camera pose estimation. In NeurlPS, 2018.
[9] Garrick Brazil and Xiaoming Liu. M3D-RPN: Monocular 3D region proposal network for object detection. In ICCV, 2019.
[10] Yongjian Chen, Lei Tai, Kai Sun, and Mingyang Li. Monopair: Monocular 3D object detection using pairwise spatial relationships. In CVPR, 2020.
[11] Garrick Brazil, Gerard Pons-Moll, Xiaoming Liu, and Bernt Schiele. Kinematic 3d object detection in monocular video. In CVPR, 2020.
[12] Abhinav Kumar, Garrick Brazil, and Xiaoming Liu, Groomed-nms: Grouped mathematically differentiable nms for monocular 3D object detection. In CVPR, 2021.
[13] Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, Bill Freeman, and Josh Tenenbaum. Marrett: 3D shape reconstruction via 2.5l sketches. In NeurIPS, 2017.
[14] Jun- Yan Zhu, Zhoutong Zhang. Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenen- baum, and Bill Freeman. Visual object networks: Image generation with disentangled 3D representations. In NewrIPS, 2018.
[15] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019.
[16] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. Atlasnet: A papier-mâché approach to learning 3D surface generation. In CVPR, 2018.
[17] Chao Wen, Yinda Zhang, Zhuwen Li, and Yanwei Fu. PixeI2Mesh++: multi-view 3D mesh generation via deformation. In ICCV, 2019.
[18] Abhijit Kundu, Yin Li, and James M Rehg. 3D-RCNN: Instance-level 3D object reconstruction via render-and-compare. CVPR, 2018
[19] Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang. Total3DUnderstandding Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. CVPR, 2020.
[20] Martin Runz, Kejie Li, Meng Tang, Lingni Ma, Chen Kong, Tanner Schmidt, Ian Reid, Lourdes Agapito, Julian Straub, Steven Lovegrove, et al. FroDO: From detections to 3D objects. In CVPR, 2020.
[21] Justin Johnson Georgia Gkioxari, Jitendra Malik. Mesh R-CNN. In ICCV, 2019.
[22] Stefan Popov, Pablo Bauszat, and Vitorio Ferrari. CoReNet: Coherent 3D scene reconstruction
from single RGB image. In 202.

---

[23] Francis Engelmann, Konstantinos Rematas, Bastian Leibe, and Vittorio Ferrari. From points to multi-object 3D reconstruction. In CVPR, 2021.
[24] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Voxels:: Learning persistent 3D feature embeddings. In CVPR, 2019.
[25] Thomas Roddick, Alex Kendall, and Roberto Cipolla. Orthographic feature transform for monocular 3D object detection. In BMVC, 2018.
[26] Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander. Categorical depth distribution network for monocular 3D object detection. In CVPR, 2021.
[27] Benoit Guillard, Edoardo Remelli, and Pascal Fua. UCLID-Net: Single view reconstruction in object space. In NewrlPS, 2020.
[28] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects points. arXiv preprint ar04v 07850,, 2019.
[29] Francis Engelmann, Jörg Stückler, and Bastian Leibe. Joint object pose estimation and shape reconstruction in urban street scenes using 3D shape priors. In GCPR, 2016.
[30] Rohan Chabra, Jan E Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In ECCV, 2020.
[31] Muhammad Zeeshan Zia, Michael Stark, and Konrad Schindler. Are cars just 3D boxes?-jointly estimating the 3d shape of multiple objects. In CVPR, 2014.
[32] Florian Chabot, Mohamed Chaouch, Jaonary Rabarisoa, Céline Teuliere, and Thierry Chateau. Deep MANTA: A coarse-to-fine many-task network for joint 2D and 3D vehicle analysis from monocular image. In CVPR, 2017.
[33] Chi Li, M Zeeshan Zia, Quoc-Huy Tran, Xiang Yu, Gregory D Hager, and Manmohan Chan- draker. Deep supervision with shape concepts for ocelusion-aware 3D object parsing. In CVPR, 2017.
[34] Hao Su, Charles R Qi. Yangyan Li, and Leonidas J Guibas. Render for CNN: Viewpoint estimation in images using cnns trained with rendered 3D model views. In ICCV, 2015.
[35] Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints. In CVPR, 2015.
[36] Yu Xiang, Wongun Choi, Yuanqing Lin, and Silvio Savarese. Data-driven 3D voxel patterns for object category recognition. In CVPR, 2015.
[37] M Zeeshan Zia, Michael Stark, and Konrad Schindler. Towards scene understanding with detailed 3D object representations. IJCV, 112(():188-20) 2015.
[38] Patrick Poirson, Phil Ammirato, Cheng Yang Fu, Wei Liu, Jana Kosecka, and Alexander C Berg. Fast single shot detection and pose estimation. In 3DV, 2016.
[39] Yousef Atoum, Joseph Roth, Michael Bliss, Wende Zhang, and Xiaoming Liu. Monocular video-based trailer coupler detection using multiplexer convolutional neural network. In ICCV, 2017.
[40] Xiaozhi Chen, Kaustav Kundu, Yukun Zhu, Andrew G Berneshawi, Huimin Ma. Sanja Fidler, and Raquel Urtasun. 3D object proposals for accurate object class detection. In NewrIPS, 2015.
[41] Bin Xu and Zhenzhong Chen. Multi-level fusion based 3D object detection from monocular images. In CVPR, 2018.
[42] Andrea Simonelli, Samuel Rota Bulo, Lorenzo Porzi, Manuel López-Antequera, and Peter Kontschieder. Disentangling monocular 3D object detection. In ICCV, 2019.
[43] Lijie Liu, Jiwen Lu, Chunjing Xu, Qi Tian, and Jie Zhou. Deep fitting degree scoring network for monocular 3D object detection. In CVPR, 2019.
[44] Wongun Choi, Wei Chao, Caroline Pantofaru, and Silvio Savarese. Understanding indoor scenes using 3D geometric phrases. In CVPR, 2013.
[45] Siyuan Huang, Siyuan Qi, Yixin Zhu, Yinxue Xiao, Yuanlu Xu, and Song-Chun Zhu. Holistic
3D scene parsing and reconstruction from a single RGB image. In ECCV, 2018.
[46] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo
Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An information-rie 3D model repository. arXiv preprint arXiv:1512.03012, 2015.
12

---

[47] Zi-Hang Jiang, Qianyi Wu, Keyu Chen, and Juyong Zhang. Disentangled representation learning for 3D face shape. In CVPR, 2019.
[48] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael Black. Generating 3D faces using convolutional mesh autoencoders. In ECCV, 2018.
[49] Feng Liu, Luan Tran, and Xiaoming Liu. 3D face modeling from diverse raw scan data. In ICCV, 2019.
[50] Timur Bagautdinov, Chenglei Wu, Jason Saragih, Pascal Fua, and Yaser Sheikh, Modeling facial geometry using compositional VAEs. In CVPR, 2018. 8.
[51] Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner. Shape completion using 3D- encoder-predictor CNNs and shape synthesis. In CVPR, 2017.
[52] David Stutz. and Andreas Geiger. Learning 3D shape completion from laser scan data with weak supervision. In CVPR, 2018.
[53] Feng Liu, Luan Tran, and Xiaoming Liu. Fully understanding generic objects: Modeling, segmentation, and reconstruction. In CVPR, 2021.
[54] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017.
[55] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3D classification and segmentation. In CVPR, 2017.
[56] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3D mesh models from single RGB images. In ECCV, 2018.
[57] Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3D-R2N2: A unified approach for single and multi-view 3D object reconstruction. In ECCV, 2016.
[58] Lars Mescheder, Michael Oechsle. Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. In CVPR, 2019.
[59] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
[60] Feng Liu and Xiaoming Liu, Learning implicit functions for topology-varying dense 3D shape correspondence. In NeurlPS, 2020.
[61] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. ICCV, 2019.
[62] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. CvxNet: Learnable convex decomposition. In CVPR, 2020.
[63] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. BSP-Net: Generating compact meshes via binary space partitioning. In CVPR, 2020.
[64] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3D scenes. In CVPR, 2020.
[65] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In CVPR, 2021.
[66] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NewrlPS, 2020.
[67] Mateusz Michalkiewiez, Eugene Belilovsky, Mahsa Baktashmotlagh, and Anders Eriksson. A simple and scalable shape representation for 3d reconstruction. In BMVC, 2020.
[68] Daniel Ricao Canelhas, Erik Schaffernicht, Todor Stoyanov, Achim J Lilienthal, and Andrew J Davison. Compressed voxel-based mapping using unsupervised learning. Robotics, 2017.
[69] Danhang Tang, Mingsong Dou, Peter Lincoln, Philip Davidson, Kaiwen Guo, Jonathan Taylor, Sean Fanello, Cem Keskin, Adarsh Kowdle, Sofien Bouaziz, et al. Real-time compression and streaming of 4D performances. TOG, 2018
[70] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
13

---

[71] Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In MICCAI, 2016.
[72] Zechen Liu, Zizhang Wu, and Roland Tóth. Smoke: Single-stage monocular 3d object detection via keypoint estimation. In CVPRW, 2020.
[73] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[74] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman. Pix3D; Dataset and methods for single-image 3D shape modeling. In CVPR, 2018.
[75] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3D reconstructions of indoor scenes. In CVPR, 2017.
[76] Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X Chang, and Matthias Nießner. Scan2CAD: Learning CAD model alignment in RGB-D scans. In CVPR, 2019.
14


Reviews:
[{"review_id": "srcF-IotwfD", "reviewer": "Reviewer_7tQr", "summary": "The authors propose a method for 3d detection and reconstruction from a single image. The main novelty of the paper in my opinion is the PCA_SDF shape representation which improves inference time and helps capture details in the reconstruction. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong Points\n- The paper improves the SOTA in 3d reconstruction from a single image\n- The PCA_SDF improves details while being computationally effiicient.\n\nWeak Points\n- The paper does 3d detection using a 3d centernet. It is unclear what is the objective of performing this detection. There is a large body of work on monocular 3d object detection that this paper completely ignores in related work and in experiments. Is the 3d detection a way to regularize the network? Does the network benefit from multi-task effects? \n- The overall novelty of the approach is marginal. Each module is isolation has been well studied. In that case, the main novelty is piecing these individual elements together which is more engineering that research. The fact that the authors do both detection and reconstruction dilutes the contributions of the paper. The authors should focus on reconstruction and dive into details. \n- The PCA_SDF borrows ideas from EIgenSDF [63] and Jiang et. al. [60]. The authors need to do more ablation studies by swapping out their PCA-SDF with eigen-sdf/ local implicit grids and be more thorough about the benefit of the pca-sdf. A lot of work on sdf has emerged since deepsdf and it is important to validate this work against the latest and greatest. \n\nPost Rebuttal: Based on the response of the authors, and having evaluated the other reviews, I am updating my rating for this paper. Although some of the other reviewers mirrored my concerns of the paper, I agree that the improvement in results with simple innovations is of broad appeal to the research community working on 3D reconstruction. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss this. Perhaps, a sentence or two can be added beyond stating that there are no obvious negative impacts. The non-obvious ones are the ones to think and discuss. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "XtJoBMkwCqo", "reviewer": "Reviewer_NjXM", "summary": "This paper presents a joint framework for detecting multiple 3D objects from input images while also reconstructing the 3D shapes. The proposed method improves upon previous work [20,21] for joint 3D object detection and reconstruction mostly at the system/architectural level. The authors also propose a new way of compactly representing 3D shapes with signed distance functions (SDF) using PCA that can be naturally incorporated into the framework. Experiments are benchmarked with synthetic multi-object ShapeNet renderings, Pix3D, and a new dataset with annotations for multi-object detection & reconstruction.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ A nice overview of related literature is provided. \n+ The method description is clear, and it is easy to follow and understand. The overview diagram is also helpful in understanding the entire framework.\n+ The proposed PCA-SDF representation is particularly interesting, as high-quality SDF shapes can be broken down into a collection of SDF parts that can be compactly represented using a pretrained dictionary, which is generalizable to out-of-category shapes and greatly speeds up inference time.\n+ Experimental results show that the proposed system outperforms the baseline methods on different benchmarks.\n\nWeaknesses:\n- The introduction doesn't give much context on where this work currently stands in the problem scope of 3D object detection and reconstruction. Why is joint 3D detection/reconstruction important? It is also unclear what the current major problems are for this specific task, and the intro is written rather much like another related work section.\n- In the related work section, it is unclear how two-stage pipelines [17,19,51] would be inferior and why one would not prefer these methods over the proposed (L87-88). Personally, I do not see how decoupling detection and object reconstruction would be undesirable. It is also unjustified why depth estimation in [21] is undesirable and why the proposed CenterNet-3D can improve detection accuracy (L92-93,150-151). The authors should also discuss the difference between this work and [A], a very related prior work on voxel-based 3D detection/reconstruction.\n- \"CenterNet\" seems to be an important part of the pipeline (detection) and the term has been repeated throughout the paper, but it was never explained what CenterNet is. I think the authors should devote at least a paragraph explaining what CenterNet is and how it is useful for the design of the proposed method. The authors should not assume it is a commonly known concept.\n- It is unclear which component in the system is the most crucial to the performance improvement. Is it the use of voxel features, the design of CenterNet-3D, or the use of the new PCA-SDF representation? It is also unclear from the experiments section, as the authors reported results without justifying why there was the improvement. In the current form, I cannot see what the key message or takeaway is from this paper.\n- It is difficult to compare the presented visual results. For one, the ground-truth 3D CAD models are not shown (Fig. 4, 5, 6) so it's hard to tell which method is actually better, despite the results from the proposed method looking slightly more plausible. Ground-truth shapes were shown in the supplementary videos but lacks baseline comparisons, and I think having all of them together is important for evaluation. On a side note, the visualization between CoReNet and the proposed in Fig. 4 are different, which slightly prevents a direct comparison.\n- I appreciate the authors' effort to create a new real-world dataset for more supportive evaluation, but I think there should be at least some visualization of what the new dataset looks like by including some example images and ground-truth CAD models processed from Scan2CAD. The quality of the dataset, and thus the evaluation quality, is currently unclear. Also, why is [21] not compared against on this new dataset? Evaluating the detection mAP but only comparing against [20] (which does not perform detection) is not meaningful. Finally, will the new dataset be released to the public?\n\nI think in general, the authors seem to aim to get better benchmarking results as the ultimate goal, but it is not clear what components really contributes to the system. As many of the components seem orthogonal (e.g. 3D feature lifting, coarse-to-fine voxelization, and PCA-SDF), it would be good to analyze and ablate each individual components in depth, and show that an integrated system with all the improvements would be most desirable.\n\nOther minor problems:\n- Why are $\\delta_h$, $\\delta_w$, $\\delta_l$ residual sizes? This term is misleading as they are actually the logarithm of the sizes.\n- Concatenating (x,y,z) coordinates in convolutional operations is not positional encoding, but rather CoordConv [B]. Positional encoding refers to a deterministic mapping for coordinate-based representations, which is different. On a side note, the original concept for positional encoding should be from Transformers [C] instead of NeRF [66].\n- What does \"might receive the same 2D feature\" mean in L136?\n- What do \"200 surfaces\" mean (L231,264)? What counts as \"a surface\"?\n- The citation for focal loss is wrong -- it should be [D].\n- It is unclear why predicting a higher-dimensional code is more challenging for neural networks (L290).\n\n[A] Tung et al. \"Learning Spatial Common Sense with Geometry-Aware Recurrent Networks.\" CVPR 2019  \n[B] Liu et al. \"An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution.\" NeurIPS 2018  \n[C] Vaswani et al. \"Attention Is All You Need.\" NIPS 2017  \n[D] Lin et al. \"Focal Loss for Dense Object Detection.\" ICCV 2017", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6hr"}, "extra_text": {}}, {"review_id": "I1imXtBjuG", "reviewer": "Reviewer_2Xpx", "summary": "The paper proposes a novel method for 3D detection and reconstruction of multiple objects from single images, that combines 2D feature extraction and 2D-3D lifting with a modified (3D) version of the CentreNet detector and  novel coarse-to-fine reconstruction stage.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive:\n* The paper is well organised, very well written and easy to understand.\n* The approach has several novel components and (i) appears to produce state of the art results while (ii) improving on inference speed over other competing methods.\n* The experimental section is quite extensive and does appear to cover most essential benchmarks and ablations.\n\nNegative / Questions:\n* Proposed 1-4 is somewhat confusing — maybe put more text in the caption. I’d be good to expand the text on lines 275 - 282.\n* What if you had bigger rotation perturbations? At what angled would we expect PCA-SDF to fail?\n* Are there any advantages of PCA-SDF over using PCA over a standard occupancy? \n* Is the importance of the positional encoding benchmarked anywhere?\n* Did you do any ablation on the size of the PCA-SDF local volume? Would smaller volumes require more fewer basis functions and generalize better, or does the size of the volume not matter much?\n* Some typos (eg line 283).\n\nOverall, I am very much in favour of acceptance. The proposed approach is principled and produces very good results, and the paper is very well written.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "-j1kdg5JgN", "reviewer": "Reviewer_gf8G", "summary": "The paper proposes a technique for joint detection and reconstruction of objects from images. The core idea is to express the signed distance function (used for reconstruction) within a set of pre-trained linear bases, and regress the bases coefficients by a deep network. The effectiveness of the solution is evaluated on a number of datasets against recently published work, and shows good performance.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think the technique is solid and easy to understand, the results very competitive against SOTA (with a simpler technique), and the writing simply **excellent** (congrats! the introduction was the *only* section I did not enjoy reading). I am willing to raise the score by one point if my questions are answered properly, and I do not score the paper even higher, simply because I have not personally worked on all of the datasets this work is evaluated on (I would like to see the other reviewer's comments before taking a championing stance).\n\n## Main questions (Q):\n1. my main concern is the overall case of DeepLS vs. PCA-SDF. While without doubt the latter should be more computationally performant, I am skeptical that the accuracy (or latent space) of the former to not be more expressive. Aside from experiments, the paper lacks a theoretical justification of why this should be the case, and I would like to hear the authors' opinion in regards. For example, the recent NGLOD and ACORN papers use *tiny* decoders, and they represent geometry to very high accuracy?\n1. the use of positional encoding in the UNet structure is one of the main differences w.r.t. past works (I am aware of), and I am simply amazed it was not quantitatively ablated? I only saw qualitative comparisons in Suppl/Fig.2. I would expect the method to collapse without this enabled in the multi-object setting?\n1. Why euler angles and not quaternions, expmaps, or other vector representations? euler is especially a challenge as the representation is non-periodic (0==360). How can this not be an issue?\n1. is reconstruction performed within each object's bounding box domain, or at the scene level (I assume the latter)? regardless of the answer, picking one begs you should compare against the other in terms of reconstruction accuracy? Why was this not done?\n1. if an image is captured in the wild, knowing where to place the grid becomes a chicken and egg problem (you could estimate the grid position from detections, but detections are derived from the grid). Shouldn't this limitation be clearly acknowledged?\n1. how finnicky is thresholding the detection maps? And if this becomes noisy, wouldn't this require some form of maximal suppression?\n1. Sec 4.2: You are using the Pix3D splits from MeshRCNN, yet provide no quantitative comparisons to this paper... can you justify?\n1. you mention \"truncated objects\" and that other methods have \"no special design for feature smearing\"... does yours have it? and can you spell out the issue?\n\n## Technical details (T):\n1. I am pretty sure in (1) you want to write everything in homogeneous coordinates.. otherwise the math is actually wrong (e.g. $c_3d$ becomes a scale rather than a translation)\n1. You mention interpolation in L199, but I do not recall seeing any results/ablations in regards? Also note that NSVF and NGLOD both allow for this?\n1. in (3) you employ the L1 norm for projecting the signal to the PCA basis in the loss, but PCA uses the L2 norm in its optimization, have you tried L2? Doesn't L1 result in suboptimal performance due to the mismatch?\n1. L213: the L1 norm of a matrix is the maximum absolute column sum... pretty sure that this is not what you meant to say, are you referring to the Frobenius norm?\n\n## Further comments (C)\n\n1. Title: not sure that this paper is voxel based, as when I think of voxels I do not think of implicit functions... consider changing your title to reflect this? \n1. L32-37 should be moved to the related works. It communicated very little to me as a reader\n1. L112-113: not sure this sentence is true, I think it is more proper to say that this is not used within a deep network for inference task (it was used as a prior for compression)?\n1. L123-126: it would be good to acknowledge past work from which you inherited this design, I recall learning it from  \"learnable triangulation of human pose\" (LTUP), but I am sure there was an earlier paper...\n1. L127-144: same as above, this is almost identical to LTUP, with the exception of positional encoding?\n1. L161: residual with respect to what? it was only mentioned later on in the paper, and you should mention it here\n1. L155: it would be good to note that, compared to past works such as KeypointNet or LTUP this is performed because you need multi-modal instance detection. \n1. L170: voXELization\n1. L188: q is defined and never used (this is also typically called a `meshgrid`)\n1. L189: what does \"toward the surface\" mean?\n1. L209: please add a line to mention why this focal loss is needed?\n1. L215: typo \"without as\"\n1. L303: how? by the classical inner product of the lexicographically-unrolled SDF with the PCA projection matrix? (hint: cite or math, your choice)\n1. L34: please avoid undeclared acronyms at all costs.. BEV? Bird Eye View? maybe...", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not sure, and not sure I should care for a paper of this type.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper initially had mixed reviews (8,8,5,5). After reading the rebuttal and checking the other reviewers' comments, the two negative reviewers were convinced and raised their score to 6. Reviewer 7tQr appreciated the simple innovation this paper offers, which is of broad appeal to the 3D reconstruction community. Reviewer NjXM stated that the rebuttal addressed their concerns adequately. Hence, this paper now has 4 positive scores and it should be accepted.


Decision: Accept (Poster)