3D Surface Reconstruction in the Wild by Deforming Shape Priors from Synthetic Data
Authors: Nicolai Haeni
Jun-Jee Chao
Volkan Isler
Venue: Submitted to ICLR 2023 (2023)
Keywords: 3D reconstruction
pose estimation
shape deformation

Abstract: We present a new method for category-specific 3D reconstruction from a single image. A limitation of current color image-based 3D reconstruction models is that they do not generalize across datasets, due to domain shift. In contrast, we show that one can learn to reconstruct objects across datasets by shape priors learned from synthetic 3D data and a point cloud pose canonicalization method. Given a single depth image at test time, we first place this partial point cloud in a canonical pose. Then, we use a neural deformation field in the canonical coordinate frame to reconstruct the 3D surface of the object. Finally, we jointly optimize object pose and 3D shape to fit the partial depth observation. Our approach achieves state-of-the-art reconstruction performance across several real-world datasets, even when trained without ground truth camera poses (which are required by some of the state-of-the-art methods). We further show that our method generalizes to different input modalities, from dense depth images to sparse and noisy LIDAR scans. 

Under review as a conference paper at ICLR 2023
3D SURFACE RECONSTRUCTION IN THE WILD BY DE- FORMING SHAPE PRIORS FROM SYNTHETIC DATA
Anonymous authors Paper under double-blind review
ABSTRACT
We present new method for category-speciff 3D reconstruction from single image. A limitation of current deep learning color image-based 3D reconstruc- tion models is that they do not generalize across datasets, due to domain shift. In contrast, we show that one can learn to reconstruct objects across datasets through shape priors learned from synthetic 3D data and point cloud pose canonicaliza- tion method. Given a single depth image at test time, we first place this partial point cloud in canonical pose. Then, we a neural deformation field in the canonical coordinate frame to reconstruct the 3D surface of the object. Finally, we jointly optimize object pose and 3D shape to fit the partial depth observa- tion. Our approach achieves state-of-the-art reconstruction performance across several real-world datasets, even when trained without ground truth camera poses (which are required by some of the state-of- tfate-of-the-art methods). We further show that our method generalizes different input modalities, from dense depth images sparse and noisy LIDAR scans.
INTRODUCTION
Reconstructing 3D object surfaces from images is a longstanding problem in the computer vision community, with applications to roboties (Bylow et al. 2013) content creation (Huang et al., 2017]. Every computational approach aimed at 3D reconstruction has to answer the question of which representation is best suited for the underlying 3D structure. An increasingly popular an- swer is to use neural fields (Park et al., 2019, Mescheder et al., 2019) for this task. These neural fields, trained on 3D ground truth data, represent the de-facto gold standard regarding reconstruc- tion quality. However, the reliance on 3D ground truth has, for now, limited these approaches to synthetic data, To remove the reliance on 3D data, the community has shifted to dense (Milden- hall et al. 2020), or sparse (Zhang et al. 2021) multi-view supervision with known camera poses. Similarly, single-view 3D reconstruction methods have also made considerable progress by using neural fields as their shape representation (Lin et al.] [2000] Duggal & Pathak 2022). While these single-view methods can be trained from unconstrained image collections, they have not achieved the high quality of multi-view 3D ground truth supervised models. In this work, we aim answer the question: How can we achieve the reconstruction quality of 3D supervised methods from single view observations in the wild?
With recent advances in generative modeling of synthetic 3D data (Gao et al. 2022 using 3D data for supervision has become practical once again. However, the problem aligning image observations to canonical spaces remains challenging. One way to solve this aligmment problem is learn the camera pose from data (Ye al., 202I). However, learning camera pose prediction from color images is a complex problem, and existing methods do not generalize to new datasets due to domain shifts. Another promising research direction the use of equivariant neural networks. For example, Condor (Sajnani et al.) 2022) and Equi-Pose (Li et al. 1 use equivariant network layers to canonicalize complete and partial point clouds through a self-supervised reconstruction loss.
Given image taken from calibrated camera, instead of using ground truth camera poses dur- ing inference, as other single-view 3D reconstruction methods Lin et al. (2020); Duggal & Pathak 2022), , suggest using single depth image together with pretrained canonicalization network to register the partial point clouds to the canonical coordinate space. However, during our work we found that canonical reconstruction methods are extremely sensitive deviations in the estimated

---

Under review as a conference paper at ICLR 2023
canonical pose (Section 4.5 and pose using only the partial shape (Figure 2). We achieve 3D reconstruction results on synthetic data close to better than the state-of-the art. Furthermore, we show that using depth images as input allows for generalization across various datasets, from dense depth in synthetic and natural images sparse depth inputs from LIDAR scans.
To recover from bad registration results, we jointly fine-tune 3D shape
Diverse 3D reconstructions
Synthetic 3D data
3D shape prion
Figure I: We leverage synthetic 3D data learn shape prior. Using pose registration algorithm, we canonicalize partial point clouds to the canonical coordinate frame to generate diverse 3D recon- structions.
RELATED WORK
3D object reconstruction based on a conditional input, such as images or depth an active research area (Chen et al. 2019 Mescheder et al. 2019 Mildenhall et al. 2020; Sitzmann et al. 2019 Tul-
siani al.) 2017 Häni al. 2020). The defacto gold standard in terms of reconstruction quality uses 3D ground truth data (Park al. 2019 Mescheder et al.] 2019). However, these approaches are largely limited to synthetic data, such as Shapenet (Chang et al. 2015). Reconstruction of real- world shapes has been performed by transferring the learned representation across domains (Duggal iet al. 2022; Bechtold et al.) 202I) or with the use special depth sensors (Newcombe et al. 2011 Choe et al. 2021). However, collecting 3D ground truth data in the real world be difficult. With the development of neural rendering and inverse graphics methods, the requirement for 31) ground truth has been relaxed in favor of dense multi-view supervision (Xu et al.) 2019) Mildenhall et al.
2020 Goel et al. 2022 Zhang et al. 2021 or single view methods that require ground truth camera poses (Lin et 2020; Duggal Pathak 2022). However, not all applications allow for the collec- tion of multi view images, and estimating camera poses from images remains challenging. With the advent of generative models for 3D shapes (Gao et al., 2022), using 3D supervision has become an interesting prospect once more. However, these 3D models are all living in a canonical coordinate frame. Our work shows how we can leverage such canonical 3D data for shape reconstruction the wild.
2.1
3D RECONSTRUCTION FROM SINGLE VIEWS
There have been extensive studies on 3D reconstruction from single view images using various 3D representations, such as voxels (aan al. 2016 Tulsiani et al., 2017 Wu al. 2018 Yang al.
Wu et al. 2018 2017). points (Fan et al., 2017: Yang et al.] 2219)), primitives (Deng et al. Chen al. 2020) or meshes (Kanazawa al. 2018 Goel al. (022). Most of the methods above use explicit representations, which suffer from limited resolution or fixed topology. Neural rendering and neural fields provide an alternative representation to overcome these limitations. Re- cent methods showed how to learn Signed Distance Functions (SDFs) (Xu et al.) 2019 Lin et al.) 2020 Duggal Pathak 2022) or volumetric representations such as occupancy (Ye et al. 2021). which have shown great promise in learning category-speciff 3D reconstructions from unstructured image collections. However, these methods usually require additional information, such as ground truth camera poses, which limits their applicability. In our work, we propose method that does not require ground truth camera poses and leverages widely available synthetic data to learn a category- specific 3D prior model.
2018
2020
2

---

Under review as a conference paper at ICLR 2023
2.2 LEARNING SHAPE RECONSTRUCTION THROUGH DEFORMATION
Learning a generalizable model that maps low-dimensional latent code to 3D surfaces can suffer from low-quality reconstructions. Category-specifie deformable shape priors are useful to improve the quality of the reconstruction (Blanz& & Vetter 1999; Engelmann et al.) 2017 Kanazawa et al.) 2018 Kar et al.) 2013 Loper et al., 2015 Mitchell et al., 2019). These methods generally learn the deformation to initall base shape. More recent work has used neural rendering together with SDFs (Lin et al. 2020 Duggal & Pathak (022) to learn 3D shape priors from image collections and their associated camera poses. Other methods (Deng al. 2021 Zheng et al., 2021) jointly learn the deformation and the template shape in a canonical frame. In this work, we go one step further and show how we can leverage template shape and deformation models for incomplete observations registered to the template coordinate frame.
2.3 POSE REGISTRATION AND 3D SHAPE CANONICALIZATION
Reliance on camera poses is an issue for many real-world datasets but a necessary step for neural rendering or deformation-basee models. Point cloud registration can estimate the object pose di- rectly and has achieved good performance when matching point clouds of the same object; however, these methods are unsuitable for single view pose estimation without a ground truth 3D model (Jiang |021 Wu et al. 2021: Qin et al. 2022). Category-level object pose estimation methods achieve
ct al.
tremendous results, for supervised training mechanisms (Rempe et al. 2020
Novotny et al. 2019
Wang et al.
2019), and using only supersupe (Spezialetti et al. 2020 [Sun et al. 2021
et al. 2021 1 Sajnani et al., 2022 Katzir et al. 2022). For example, Canonical Capsules (Sun et al.
202 learn represent object parts with pose-invariant capsules by training Siamese network in a se self-supervised manner. Although the learned capsules can reconstruct the input point cloud in the learned canonical frame, Canonical Capsules only works on complete point clouds. In contrast, Equi-pose (Li et al. 202T) can canonicalize both complete and partial point clouds. By leveraging an SE(3) equivariant network, Equi-pu pose simultaneously learns to estimate object pose and canon- ical point cloud completion. Our work shows that one can leverage Equi-pose with test time pose refinement to get accurate shape reconstructions canonical space.
2.4 POINTCLOUD COMPLETION
Instead of relying on ground truth camera poses, we use depth images to register the partial 3D point cloud into canonical frame. As
depth images as input, method closely relates to point cloud completion algorithms. Farly work used 3D convolutions to learn shape completion DDii
ett al.]
2017 Huang al. 2020). However, 3D convolutions are costly and operate on canonical
voxel grid.
More recently. PointNet encoders were used for shape completion (Liu et al., 2020
Yuan et al.
2018). Transformers have also been shown to work well on this task (Yu et al. 2021
2022). However, these methods rely on points already in a canonical coordinate frame. Further, these methods do not reconstruct the underlying surface of the object but output limited number of points. In contrast, out method does not rely on canonical input points and reconstructs the underlying object surface with high fidelity.
Yan et al.
3 METHOD
Given a single segmented RGB-D image of an object, our goal is to reconstruct the underlying 3D surface without access to ground truth camera poses. To do so, we first learn a category-specift 3D template the canonical coordinate frame together with an instance specific deformation field by leveraging synthetic 3D data. During test time, rather than directly reconstructing the shape in the camera coordinate frame, we use recent advances in point cloud canonicalization to transform a partial depth scan to the canonical space for surface reconstruction. Next, we describe first how learn the 3D shape prior purely on synthetic data. Then we discuss how we reconstruct the surface of observed depth image by deforming the learned canonical template shape.
3

---

Under review as a conference paper at ICLR 2023
Training DIF-Net
Inference
Input
Deformation
Output
Lifting and Cononicalization
Optimize Pase & Shape
3 = SDF(x)
Deform-Net 12
As
Temphite Pidk
Usper R"
Dat
Secplaste
R
We train a DeF-Net as our 3D representation on purely synthetic data in an auto-decoder fashion. During inference, we lift the depth image into 3D using the known camera intrinsics and estimate initial transformation between the camera frame and the canonical frame. We jointly optimize the object pose and 3D shape to the partial observation. Trainable parameters/network parts marked in red.
Figure 2:
3.1 3D SHIPP PRIOR
Given a set of 3D objects O, our goal is to learn a category-specific 3D shape prior together with a latent space describing the variation in shapes. Instead of directly mapping a low dimensional latent code E R to the 3D shape, we follow recent ad in learning 3D shape priors through deformation of a canonical template (Zheng al. [202]] Deng al.) 2021). We jointly learn our 3D shape prior, represented as neural network, and latent codes z; through the auto-decoder framework presented in Park al. 2019). generate high-quality 3D reconstructions, we use signed distance fields (SDFs). SDF is function that assigns each point 2j E R scalar value 5, R
SDF(x)) 8j.
representing the distance the closest object surface. The sign of sy indicates whether point is in- side (negative) or outside (positive) of the object, and the surface can be extracted as mesh through marching cubes (Lorensen & Cline 1987p. We DIF-Net as our 3D shape prior network (Deng) et al. T02T) and mention the necessary background for completeness. Please see Deng et al. (202T) for additional details. The 3D representation network consists of neural template field and defor- mation field. We use the template field to capture common structures among category of shapes by keeping the weights shared across all instances in the training set. The template field takes a 3D coordinate 2j as input and predicts the signed distance to the closest surface &;
T: 23 E R
$ BER
Note that the template field does not have to represent a valid object but fuses common structures from various objects into single neural field (see Figure 6 for some examples of the template shapes). To deform the template to a specific object instance, we use a deformation field together with structural correction field
D D: RR3 (v. As) E
The vector v deforms a point in the instance space the template space, and the correction factor As modifies the SDF value of point æÿ if it still differs from the ground truth value. The correction factor is beneficial for categories with sigmificant shape variations. For example, for chairs, there exist instances with and without armrests. We use Network (Sitzmann et al. [019 Mitchell jet al. (019)) to condition the deformation field on latent code. With learned template field and deformation field D, the SDF value of point xy can be obtained with
8j T(æ; + v) + As T(x; + Dz(x,)) + Da(x,).
(4)
3.2 TRAINING DIF
During training use the auto-decoder framework (Sizzmann et al. 2019 Park et al. 2019 to jointly learn latent codes 2; and the weights of the DIF network that predicts SDF values v (x). Given collection of shapes with ground truth SDF values the object surface and in free space,
4

---

Under review as a conference paper at ICLR 2023
first apply SDF regression loss from Sitzmann et al. 22200) as
Lady _ (x(x) JE
+ +|+ _ (V (VV,
[[] - +11+ Σ ((v,()))) PER
TES;
(5)
where 8 and n denote the ground truth SDF value and normal, V the spatial gradient of the neural field, Ω is the 3D space in which values are sampled, and 8j is the shape surface. We select an equal number of surface and free space points uniformly at random to compute this loss. The first term in Equ. [ regresses the SDF value; the second term learns consistent normals the shape surface, the third term is the cikonal equation that enforces unit norm or the spatial gradients, and the last term penalizes SDF values close to O which p(s) =
far away from the object surface with >> 1. For more details on this loss, check Sitzmann et al. (219)). We further apply multiple regularization terms help learn smooth deformations and consistent latent space. The first regularization term applies L2 regularization on the embeddings as L. E, ||z:||2 Prior work by Deng et al. (202T) showed that learning a template shape that captures common attributes across a category is improved by enforcing normal consistency across all shapes by regularizing the normals of the template networks with
Lnormal (1- +D())),
(6)
FES,
We further want deformations to be smooth and the optional corrections to the SDF field to be small, which is enforced with the following two loss terms Lemooth = >. Emen [[)]( and f.= -. Enen (z). The overall loss to training the 3D shape prior L Ludf + Li + AzC= + AaCsmooth AzCe,
with the X terms weighing the relative importance of each loss term.
3.3 POINT CLOUD LIFTING AND CANONICALIZATION DURING TESTING
Given a single RGB-D image and known camera intrinsic parameters during test time, we lift the depth image to partial 3D point cloud. In order to predict the deformation field of the observed object, we first bring the observed partial point cloud the canonical coordinate frame by leveraging Equi-pose (Li et al. (021) as our pose estimation module. Equi-pose applies a SE(3)-equivariant network to learn category-specing supervised manner. By enforcing consistency between the invariant shape reconstruction and the input point cloud transformed by the estimated pose, Equi-pose can estimate the pose of the input point cloud with respect to the learned canonical frame. Therefore, we first input complete template shape in our canonical frame to Equi-pose such that the transformation between our canonical frame and Equi-pose's canonical frame can be obtained. This way, we can transform any observed partial point cloud to our canonical frame using Equi-pose as pose estimator. However, the estimated pose from Equi- pose can only serve as noisy initialization. We show in Section 3.4 how method further optimizes for the pose to achieve accurate shape reconstruction.
canonical shape reconstruction and pose estimation
self-
3.4 JOINTLY OPTIMIZING SHAPE AND POSE
Once we trained the 3D shape prior network and the partial input point cloud is roughly aligned in the canonical space, we reconstruct the object surface by optimizing only the latent code and the object pose with the fixed prior network. As canonical 3D reconstruction methods are sensitive to minor deviations between estimated and canonical coordinate frames 4.5 we jointly optimize the latent code 2; and the initial transformation by minimizing the SDF values the observed depth points. At the same time, we sample random points in free space for the Eikonal term to ensure that the neural field is an SDE, We represent the translation as a three-dimensional vector initialized to zero and used the continuous 6D rotation parametrization from Zhou et al. (2019) for rotations. We choose a random latent code from the learned latent space as our initialization 2; and optimize
min R, Redf + X2C..
(8)
Note that we keep the weights of the prior network fixed to search for the latent representation that minimizes the SDF value at the points in the partial point cloud.
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
Datasets We include three categories in our experiments: car, chair, and airplane. We use synthetic data from the ShapeNet dataset (Chang et al. (2015) to train our deformation and template networks using 3D ground truth. Then our method trained on Shapenet directly evaluated on the following datasets: ShapeNet (Chang et al. 2015), Pix3I) chairs (Lim et al. 2013), Pascal3ID+ (Xiang et al., 2014) and the DDAD (Guizilini al., 2020) dataset. Since Pascal3D+ and Pix3I) do not contain depth scans, generate the partial point clouds by removing invisible points of the CAD models using the ground truth camera poses. Since DDAD does not provide reconstruction ground truth, we show the performance of our method real-world noisy scans qualitatively with DDAD. See the appendix for additional information on datasets, baselines, and implementation details.
Implementation Details In line with prior work train the 3D shape network three categories in the Shapenet (Chang et al. (2013) dataset, namely car, chair and plane. The networks are trained using the Adam optimizer (Kingma & Ba 2014). We use batch size 128 shapes per iteration and use 4000 points on the surface, and 4000 randomly sampled points in free space per object. Training takes 10 hours on four NVIDIA V100 GPUs.
Baselines We compare against the state-of-the-art in single view, category-specific 3D object recon- struction: i) SDF-SRN (Lin al. 2020), a neural field method that represents the object in camera coordinate frame and uses neural renderer with silhouette supervision. ii) Closest to our method TARS-3D Duggal Pathak (2022). a method that uses ground truth camera poses to render deformed template shape in canonical space to the image coordinate frame. Note that TARS-3D has access to ground truth camera poses during inference, while our method estimates this infor- mation on the fly. As our method is closely related to point cloud completion, we further compare our method against a transformer-basee point cloud completion method PoinTr (Yu et al. 2021). In contrast to the baselines, our model is only trained on Shapenet and does not have access to any 3D or camera pose information during inference except the partial depth scan.
Evaluation Metrics In this work, we follow Tatarchenko al. (2019) and report the Fl-score at threshold 1% as our primary evaluation metric. We additionally report the bidirectional chamfer distance (CD), multiplied by factor of 1e4 for readability.
4.1 3D RECONSTRUCTION ON SYNTHETIC SHAPENET DATA
Table 1 shows quantitative results of testing all approaches the holdout test of Shapenet. Our method outperforms the baselines in the car and plane categories with ground truth camera poses and is competitive in the chair category. We investigate cases where no ground truth camera poses are available and initialize our method and PoinTr with the Equi-pose estimates. Even without ac- cess to ground truth camera poses, our method performs comparable to or better than the baseline methods. We can see that PoinTr Tr suffers greatly when the coordinates are not in the canonical coor- dinate system, showing that our approach of combining canonicalization with shape reconstruction is necessary for 3D shape reconstruction real-world depth data. Our method's 3D reconstructions are more faithful to the underlying ground truth mesh. PoinTr outputs only a limited number of points and does not reconstruct the underlying surface, nor does it give correspondences between shapes in a category.
Table 1: 3D reconstruction results on synthetic test data. We report chamfer distance (CD) 4 and F-score at threshold 0.01 (F@ 1%)t. with ground truth camera poses.
Car
Chair
Planc
Methods
CD (L)
F@ 1 (t)
CD (L)
F@ 1 (f)
CD (1)
F@1 (t)
SDF-SRN Lin et al.
9.965
0.404
27.562
0.283
12.374
0.459
TARS-3D Duggal
Pathak
2022
10.175
0.412
28.823
0.272
11.302
0.418
PoinTr Yu et al. 2021
13.249
0.264
12.834
0.352
3.637
0.685
Ours!
6.181
0.497
27.292
0.343
4,495
0.768
PoinTr Yu al. 2021
56.435
0.125
36.714
0.230
23.713
0.302
Ours
9.371
0.439
32.138
0.334
16.562
0.631

---

Under review as a conference paper at ICLR 2023
4.2 3D RECONSTRUCTION ON SHAPENET WITH OCCLUSION
This section investigates how our method performs when the observed object is occluded. For this experiment, generate occluded area for the Shapenet test data as presented in Figure To study the influence of a wide variety of occlusions, we use rectangular overlap regions and generate over- lap ratios from 5 85%. We apply models trained on Shapenet on the occlusion task for all methods. As shown in Table 2 our method outperforms all the baselines without access to ground truth cam- era poses despite not seeing any occluded data during training. Compared to the other baselines, PoinTr has higher tolerance to occlusion. However, point cloud completion methods cannot pre- dict the object's surface. Moreover, these methods usually predict the complete point cloud by adding predicted points to the input. Therefore, the predicted point clouds are not guaranteed to be uniformly distributed and can have a higher density around the input points, resulting in lower Chamfer distance, as shown in Figure In contrast, our method does not have these drawbacks and can reconstruct the occluded surface by leveraging the learned category-level prior with high fidelity in terms of F-score.
Table 2: 3D reconstruction results on occluded data from the synthetic test set. We report chamfer distance (CD) 1 and F-score at threshold 0.01 (F@ 1%)1. with ground truth camera poses.
Car
Chair
Plane
Methods gt camera pose
CD (L)
F@ 1 (t)
CD (L)
F@ 1 (t)
CD (4)
F@) 1 (f)
SDF-SRN Lin et al. 2020
25,863
0.243
156.825
0.122
72.255
0.256
TARS-3D Duggal & Pathak 2022
23,806
0.241
108.976
0.131
58.998
0,263
PoinTr (Yu et al. 2021
22.110
0.208
19.079
0.297
6.168
0.601
Ours!
6.457
0.497
35.972
0.297
15.725
0.654
Poin Tr Yu ct al. 2021
59,292
0. 0.15
54.327
0.184
37.425
0.322
Ours
11.155
0.390
45.886
0.273
22.223
0.580
Input Image
SDF-SRN
TARS
PoinTr
Ours
Figure 3: Qualitative result on the occluded Shapenet dataset.
4.3 3D RECONSTRUCTION ON PASCAL3D+ AND PIX3D
For this experiment, we test the generalization capabilities of our approach. We directly apply our model trained on Shapenet for reconstructing Pascal3D+ objects. TARS-3D and PoinTr also ap- ply network weights trained on Shapenet to this task, while SDF-SRN is trained directly on the Pascal3D+ dataset. As shown in Table 3. our method again outperforms the other surface recon- struction methods without access to ground truth camera poses. Figure shows that our method generates reasonable outputs.
7

---

Under review as a conference paper at ICLR 2023
Table 3D reconstruction results on the Pascal3D+ and Pix3D dataset. We report chamfer distance (CD) 4 and F-score at threshold 1 % 1f with ground truth camera poses.
Pascul3D+
Pix3D
Car
Chair
Plane
Chair
Methods
CD (4)
F@ (1)
CD (4)
F@ (1)
CD (4)
F@ 1
CD (U)
F@] (1)
SDF-SRN Lin et 2020
16.740
0.245
24.374
0.216
29.7457
0.169
60.432
0.158
TARS-3D Duseal
Pataak 2022
19.129
0.2427
79.8922
0.1577
83,866
0.140
55.555
0.197
PoinTr Yu
17.859
0.221
11.769
0.302
4,701
0.525
13,092
0.377
Ours
13.370
0.302
16.804
0.274
54,286
0.289
31.165
0.335
Poin Yu ct al 2021
80,896
0.064
21.251
0.216
35.095
0.231
35,527
0.283
Ours
15,516
0.283
17.328
0.275
54.849
0.276
35.729
0.335
We further test our method on another chair dataset, namely the chair category of the Pix3D dataset. As shown in Table 3 our method outperforms the baselines. PoinTTr again achieves lower Chamfer distance, which does not fully represent the reconstruction quality. As shown in Figure A PoinTr generates point clouds that are not uniformly distributed while our method predicts smooth surfaces.
Input Image
SDF-SRN
TARS
Poin'Tr
Ours
Figure 4: Qualitative results the Pascal3D+ (top) and Pix3D (bottom) datasets.
4.4 3D RECONSTRUCTION ON REAL- WORLD NOISY SCANS
Finally, we apply method trained on Shapenet directly to real-world noisy LIDAR scans. To demonstrate our tolerance to noise in the point clouds, we test our method on an autonomous driving benchmark DDAD (Guizilini et al. 2020). DDAD contains urban scenes scanned using LiDARs mounted on self-driving cars. To showcase our method, we extract frames that include other driving cars and crop the LiDAR scans of other cars with masked images. Finally, these noisy LiDAR scans are fed to the pose estimation module and our deformation field to reconstruct the surfaces. Since DDAD does not contain ground truth CAD models, we present the qualitative results in Figure 5 Note that our method does not have access to the image but only the noisy LiDAR point clouds. Despite large portions of missing parts and the noise in the LiDAR scans, our method can still reconstruct reasonable car surfaces without access to ground truth camera poses.
4.5 ABLATION STUDY AND FAILURE CASES
In this section, we conduct an ablation study to asses the importance of optimizing the pose during inference. The main results are shown in Table 4 On the left, we show the ground truth mesh overlaid with the partial input points with pose optimization (blue) and without (green). We can see noticeable reduction in Fl scores and significant reduction in the reconstruction quality. The chamfer score difference between reconstruction methods across the dataset is slight, though, con- firming Tatarchenko et al. (2099) in that chamfer distance is not ideal metric for 3D reconstruc-
8

---

Under review as a conference paper at ICLR 2023
LIBAR Good
Vicepoui
Figure 5: Qualitative results on the DDAD dataset.
tion. This ablation study shows that canonical reconstruction methods are sensitive to deviations in the estimated canonical coordinate frame. This result is also confirmed by the poor performance of PoinTr input points where the estimated canonical coordinate frame is off (see Tables I and 3
Ablation of our method with and without pose optimization during inference. Left, show the ground truth mesh overlayed with the optimized and non-optimized pose. We show the resulting optimized surfaces with and without pose optimization in the middle and right.
Table 4:
Pose optim.
CD
23.008
22.223
F1
0.313
0.580
Shape
CONCLUSION AND FUTURE WORK
We introduced a new method for complete 3D surface reconstruction of object from real-world depth images. Our method relies on a representation obtained solely by training on synthetic data, which allows for extracting high-quality, category-specific geometry. We showed that even small errors in pose estimation lead to significant errors in 3D reconstruction. Therefore a simple method which uses an independently trained pose estimator followed by reconstruction in the object frame does not yield good reconstruction results. Instead, we presented finetuning scheme to optimize the object surface and pose jointly during inference. We also showed that learning strong 3D priors benefits the 3D reconstruction of occluded objects. Our method generalizes across datasets and input modalities, from dense depth images to sparse LIDAR point clouds. While our process still exhibits failure modes when the error in the estimated pose is large, this could be alleviated by combining pose estimation and 3D reconstruction in an end-to-end trainable manner. We hope our work will inspire further work in this direction.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jan Bechtold, Maxim Tatarchenko, Volker Fischer, and Thomas Brox. Fostering Generalization in Single-view 3D Reconstruction by Learning a Hierarchy of Local and Global Shape Priors. In 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition ( (CVPR), pp. 15875- 15884, Nashville, TN, USA, June 2021. IEEE.
Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pp. 187-194, 1999.
Erik Bylow, Jürgen Sturm, Christian Kerl, Fredrik Kahl, and Daniel Cremers. Real-time camera tracking and 3D reconstruction using signed distance functions. In Robotics: Science and Systems, volume 2. pp. 2013.
Angel X. Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi. and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.0301 [cs.GR], Stanford University Princeton University - Toyota Technological Institute at Chicago, 2015,
Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with interpolation-based differentiable renderer. Ad- vances in Neural Information Processing Systems, 32, 2019.
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. In Proceedings of the IEEF/CVF Conference Computer Vision and Patlern Recognition, pp. 45-54, 2020.
Jaesung Choe, Sunghoon Im, Francois Rameau, Minjun Kang, and In So Kweon. Volumefusion: Deep depth fusion for 3d scene reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16086-16095. 2021.
Angela Dai, Charles Ruizhongtai Qi, and Matthias NieBner. Shape completion using 3d-encoder- predictor cnns and shape synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5868-5877, 2017.
Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cvxnet: Learnable convex decomposition. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 31-44, 2020.
Yu Deng, Jiaolong Yang, and Xin Tong. Deformed implicit field: Modeling shapes with learned dense correspondence. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10286-10296, 2021.
Shivam Duggal and Deepak Pathak. Topologically-Aware Deformation Fields for Single- View 3D Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1536-1546, 2022.
Shivam Duggal, Zihao Wang. Wei-Chin Ma. Sivabalan Manivasagam, Justin Liang, Shenlong Wang, and Raquel Urtasun. Mending Neural Implicit Modeling for 3D Vehicle Reconstruction in the Wild. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1900-1909, 2022.
Francis Engelmann, Jörg Stückler, and Bastian Leibe. SAMP: shape and motion priors for 4d vehicle reconstruction. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 400-408. IEEE, 2017.
Haoqiang Fan, Hao Su, and Leonidas Guibas. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2463-2471 Honolulu, HI, July 2017. IEEF.
10

---

Under review as a conference paper at ICLR 2023
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li. Or Litany, Zan Gojcic, and Sanja Fidler. GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images. In Advances In Neural Information Processing Systems, 2022.
Shubham Goel, Georgia Gkioxari, and Jitendra Malik. Differentiable Stereopsis: Meshes from multiple views using differentiable rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8635-8644, 2022.
Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3D Packing for Self-Supervised Monocular Depth Estimation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Jingwei Huang, Angela Dai, Leonidas Guibas, and Matthias Nießner. 3DLite: Towards Commodity 3D Scanning for Content Creation. ACM Transactions on Graphics 2017 (TOG), 2017.
Zitian Huang, Yikuan Yu, Jiawen Xu, Feng Ni, and Xinyi Le. Pf-net: Point fractal network for 3d point cloud completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7662-7670, 2020.
Nicolai Häni, Selim Engin, Jun-Jee Chao, and Volkan Isler. Continuous Object Representation Networks: Novel View Synthesis without Target View Supervision. In Advances in Neural Infor- mation Processing Systems, 2020.
Haobo Jiang, Yaqi Shen, Jin Xie, Jun Li, Jianjun Qian, and Jian Yang. Sampling network guided cross-entropy method for unsupervised point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6128-6137, 2021.
Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In Proceedings the European Conference on Computer Vision (ECCV), pp. 371-386, 2018.
Abhishek Kar, Shubham Tulsiani, Joao Carreira, and Jitendra Malik. Sategory-Specifiic Object Reconstruction From a Single Image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
Oren Katzir, Dani Lischinski, and Daniel Cohen-Or. Shape-Pose Disentanglement using SE (3)- equivariant Vector Neurons. arXiv preprint arXiv:2204.01159, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochaattic Optimization. 44Xiv:/412.69000
[cs], December 2014.
Xiaolong Li, Yijia Weng. Li Yi, Leonidas Guibas, A Lynn Abbott, Shuran Song. and He Wang. Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation. Thirty-Fifth Conference Neural Information Processing Systems, 2021.
Joseph Lim, Hamed Pirsiavash, and Antonio Torralba. Parsing ikea objects: Fine pose estimation. In Proceedings of the IEEE international conference on computer vision, pp. 2992-2999, 2013.
Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey. Sdf-sm: Learning signed distance 3d ob- ject reconstruction from static images. Advances in Neural Information Processing Systems, 33: 11453- 1464, 2020.
Minghua Liu, Lu Sheng, Sheng Yang, Jing Shao, and Shi-Min Hu. Morphing and sampling network for dense point cloud completion. In Proceedings the AAAI conference artificial intelligence, volume 34, pp. 11596-11603, 2020.
Issue: 07.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: A skinned multi-person person linear model. ACM transactions on graphics (TOG), 34(6):1- 16, 2015. Publisher: ACM New York, NY, USA.
William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3D surface construction algorithm. ACM siggraph computer graphics, 216 1987. Publisher: ACM New York, NY, USA.
11

---

Under review as a conference paper at ICLR 2023
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4460-4470 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Proceed- ings of the European Conference on Computer Vision, pp. 405-421. Springer, 2020.
Eric Mitchell, Selim Engin, Volkan Isler, and Daniel D Lee. Higher-Order Function Networks for Learning Composable 3D Object Representations. In International Conference on Learning Representations, 2019.
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE international symposium on mixed and augmented reality, pp. 127-136. Ieee, 2011.
David Novotny, Nikhila Ravi, Benjamin Graham, Natalia Neverova, and Andrea Vedaldi. C3dpo: Canonical 3d pose networks for non-rigid structure from motion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7688-7697, 2019.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 165-174, 2019.
Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu. Geometric Trans- former for Fast and Robust Point Cloud Registration. In Proceedings of the IEEE/VVE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 11 11143-11152, June 2022.
Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojeic, Srinath Sridhar, and Leonidas Guibas. Caspr: Learning canonical spatiotemporal point cloud representations. Advances in neural infor- mation processing systems, 33: -378-1 2020.
Rahul Sajnani, Adrien Poulenard, Jivitesh Jain, Radhika Dua, Leonidas J. Guibas, and Srinath Srid- har. ConDor: Self-Supervised Canonicalization of 3D Pose for Partial Shapes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3D-structure-awer aware neural scene representations. In Advances in Neural Information Pro- cessing Systems, volume 32, pp. 1 111911130 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Riccardo Spezialetti, Federico Stella, Marlon Marcon, Luciano Silva, Samuele Salti, and Luigi Di Stefano. Learning to orient surfaces by self-supervised spherical cnns. Advances in Neural Information Processing Systems, 33:5381-5339 2020.
Weiwei Sun, Andrea Tagliasacchi, Boyang Deng. Sara Sabour, Soroosh Yazdani, Geoffrey Hinton, and Kwang Moo Yi. Canonical Capsules: Self-Supervised Capsules Canonical Pose. Neural Information Processing Systems, 2021.
Maxim Tatarchenko, Stephan R. Richter, Rene Ranftl, Zhuwen Li, Vladien Koltun, and Thomas Brox. What Do Single- View 3D Reconstruction Networks Learn? In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3400-3409, Long Beach, CA. USA, June 2019. IEEE.
View
Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, and Jitendra Malik. Multi View Supervision for Single- View Reconstruction via Differentiable Ray Consistency. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
12

---

Under review as a conference paper at ICLR 2023
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2642-
2651, 2019.
Bingli Wu, Jie Ma, Gaojie Chen, and Pei An. Feature interactive representation for point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5530-5539, 2021.
Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, Bill Freeman, and Josh Tenenbaum. Marmet:
3d shape reconstruction via 2.5 d sketches. Advances neural information processing systems, 30, 2017.
Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T Freeman, and Joshua B Tenenbaum. Learning shape priors for single-view 3d completion and reconstruction. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pp. 646-662, 2018.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang. and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE winter conference on applications of computer vision, pp. 75-82.
IEEE, 2014.
Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In Advances in Neural Information Processing Systems, volume 32, 2019.
Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. Advances in neural information processing systems, 29, 2016.
Xingguang Yan, Liqiang Lin, Niloy J. Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. ShapeFormer: Transformer-Basee Shape Completion via Sparse Representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6239- 6249, June 2022.
Guandao Yang, Yin Cui, Serge Belongie, and Bharath Hariharan. Learning single-view 3d recon- struction with limited pose supervision. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 86-101, 2018
Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4541 15500, 2019.
Yufei Ye, Shubham Tulsiani, and Abhinav Gupta. Shelf-Supervised Mesh Prediction in the Wild. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8843-88545 June 2021.
Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry- transformer. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 12498-12507 2021.
Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. In 2018 International Conference 3D Vision (3DV), pp. 728-737. IEEE, 2018.
Jason Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan. NeRS: Neural reflectance surfaces for sparse-view 3d reconstruction in the wild. Advances in Neural Information Process- ing Systems, 34:29885-29984 2021.
Zerong Zheng, Tao Yu, Qionghai Dai, and Yebin Liu. Deep implicit templates for 3d shape represen- tation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1429-1439, 2021.
13


Reviews:
[{"review_id": "vZ6u8rApGNO", "reviewer": "Reviewer_qugy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper present a deep learning based pipeline to reconstruct the 3D surface of an object from a RGBD image. A network is trained to encode the category shape priors and then deform the shape priors to fit the object point cloud. A separate network is needed to transorm the object point cloud from the camera coordinate to the canonical coordinate frame. The proposed method is evaluated on both synthetic and real datasets and shows superior performance to the compared peer works. ", "summary_of_the_review": "The work seems to be a cobination of existing ideas and the novelty is marginal. My current rating is borderline reject. ", "strength_and_weaknesses": "Strength:\n- The method achieves better results compared to some recently proposed learning based methods. It also addresses some of their limitations which moves it closer to practical application (still a long way to go though)\n- The method shows good results on the picked occlusion cases. It would be interesting to see a more thorough evaluation or demonstration. \n\nWeakness:\n- It is not clear how fast the method runs in practice, especially considering that it requires segmenting the objects from the image or point cloud and lifting the point cloud to the canonical coordinate frame.\n- The proposed method seems to be a combination of ideas from existing works. The novelties over these existing works are not clearly discussed. \n- Several parts of the paper need to be revised. \n  - Abstract \"A limitation of current color image-based 3D reconstruction models is that they do not generalize across datasets\": this should only apply to learning based methods.\n  - 2.1 \"Neural rendering and neural fields provide an alternative representation to overcome these limitations. Surface fields, such as Signed Distance Functions (SDFs) (Xu et al., 2019; Lin et al., 2020; Duggal & Pathak, 2022) or volumetric representations such as occupancy (Ye et al., 2021)\": both SDF and occupancy grid are classic represetnations, they are not proposed or provided by neural rendering and neural fields.\n  - 3.1 \"Instead of directly mapping a low dimensional latent code zi ∈ Rn to the 3D shape through concatenation\": concatenation is quite confusing.\n  - Eq 2: what is \\hat{s}?\n  - 3.1 What does \"instance space to the template space\" mean?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JN03aCEmir", "reviewer": "Reviewer_aneS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for single-view category-specific 3D reconstruction. The pipeline starts with deriving a canonical pose with a partial point cloud. Then a neural deformation is used to reconstruct the object's 3D surface. Finally a joint optimization of pose and shape is to further improve the results.\n", "summary_of_the_review": "To me the paper is clearly below the bar from multiple aspects: clarity of presentation, technical novelty and insufficient experimental validation. Thus I would vote for reject.", "strength_and_weaknesses": "### Strength\n- Qualitative and quantitative results seem impressive, there are notable improvements over previous work.\n- The appendix is useful to understand more details of the work.\n\n ### Weakness\n- Overall the work feels more like a pipeline to combine different components together. It is not fully clear to me what the major technical contribution of the paper is, compared to the previous work. It would be useful if authors can highlight and contrast over the previous work.\n- The writing and presentation of the paper can be improved. Currently it's not very easy to follow the paper.\n- Ablation study and analysis is a bit lacking, thus it's also hard to understand the individual impact of the components.\n\n### Additional Qs:\n- In table 2, why the improvement of car is larger compared to other categories?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "11bkwZIcAA", "reviewer": "Reviewer_Xiit", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper is proposed a method to recover (complete) the 3D reconstruction of an object category from a single depth picture. The method exploits synthetic training data and a point cloud pose canonicalization to learn a category-specific geometry. As a consequence, the learning of these 3D priors helps to retrieve the 3D reconstruction even for occluded bodies. The method is evaluated on different scenarios by using depth images or LIDAR-like point clouds; a comparison with competing techniques, ablation studies and failure cases are also reported. ", "summary_of_the_review": "As I commented before, the results obtained are promising, and even though some priors want to be simplified with respect to other techniques in the literature, the novelty of the proposal is somewhat incremental.", "strength_and_weaknesses": "The authors claim their method does not require ground truth camera pose or 3D reconstruction for supervision, being the goal to learn a category-specific 3D prior model. I have no problem with that, in fact, I like the idea. However, similar ideas were exploited in the non-rigid structure-from-motion community, a community that was not included in the discussion of this paper. I know the differences, but I feel a discussion of these approaches could help the reader. We can find some works on the literature, such as: “Unsupervised 3D Reconstruction and Grouping of Rigid and Non-Rigid Categories” and “Structure from category: A generic and prior-less approach.”\n\nThe proposed method is a combination of well-known ideas and architectures in the literature. In this context, the technical contribution is a bit poor. \n\nBe consistent with notation. For example, see x in Fig. 2 and Eqs. (1), (2), etc. \n\nReproducibility: some weight coefficient values, for example, see Eq. (7) are never provided in the main paper. \n\nA calibrated camera is required. This should be commented on in the introduction. \n\nWhy not use additional categories? A priori, I think more everyday categories are included in the datasets the authors are using, so I cannot understand why just three are considered. \n\nThe authors report quantitative evaluation in terms of chamfer distance and F1 score. To be honest, for a 3D reconstruction algorithm I would hope for a different metric where we evaluate the 3D model, instead of one based on chamfer deviation. \n\nThe method provides a good performance in comparison with competing techniques. Moreover, it can handle occlusion properly. However, the occlusion seems not to be very realistic. Instead of that, I would like to see some cases where the occlusion can explain a natural and real case. The method can naturally also handle missing observations.\n\nI agree with the authors; the proposed method produces more smooth solutions to the point of avoiding capturing details. Can the authors imagine an explanation of that?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject