A Simple and Scalable Shape Representation for 3D Reconstruction
Authors: Mateusz Michalkiewicz
Eugene Belilovsky
Mahsa Baktashmotagh
Anders Eriksson
Venue:  (2020)
Keywords: Computer Vision
3D Reconstruction

Abstract: Deep learning applied to the reconstruction of 3D shapes has seen growing interest. A popular approach to 3D reconstruction and generation in recent years has been the CNN decoder-encoder model often applied in voxel space. However this often scales very poorly with the resolution limiting the effectiveness of these models. Several sophisticated alternatives for decoding to 3D shapes have been proposed typically relying on alternative deep learning architectures. We show however in this work that standard benchmarks in 3D reconstruction can be tackled with a surprisingly simple approach: a linear decoder obtained by principal component analysis on the signed distance transform of the surface. This approach allows easily scaling to larger resolutions. We show in multiple experiments it is competitive with state of the art methods and also allows the decoder to be fine-tuned on the target task using a loss designed for SDF transforms, obtaining further gains.    

Under review as a conference paper at ICLR 2020
A SIMPLE AND SCALABLE SHAPE REPRESENTATION FOR 3D RECONSTRUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning applied the reconstruction of 3D shapes has growing in- terest. A popular approach to 3D reconstruction and generation in recent years has been the CNN decoder-encoder model ofien applied in voxel space. However this often scales very poorly with the resolution limiting the effectiveness of these models. Several sophisticated alternatives for decoding to 3D shapes have been proposed typically relying on alternative deep learning architectures. We show however in this work that standard benchmarks in 3D reconstruction can be tack- led with surprisingly simple approach: linear decoder obtained by principal component analysis on the signed distance transform (SDF) of the surface. This approach allows easily scaling to larger resolutions. We show in multiple exper- iments is competitive with state of the art methods and also allows the decoder to be fine-tuned on the target task using loss designed for SDF transforms, ob- taining further gains.
INTRODUCTION
In recent years there has been an increased interest in extending the successes of deep learning to problems requiring analysis and representation of 3D shapes. This includes many long standing tasks such as 3D shape reconstruction (Wu et al., 2016a Choy al. 2016) from single or multiple views, as well as shape completion Mescheder et (2018). There are number of applications of these methods in robotics, surgery, and augmented reality.
A popular approach has been the use of CNN decoder-encoder architectures Choy et al. (0016) as popularized applications like segmentation Chen al. (2017). Here for example in the single view reconstruction 2D CNN will encode the 2-D image and 3D CNN decoder model will produce the final representation full use of the structure of the object. Similar problems arise in more general attempts to learn latent variable models of shapes (Wu et al. 2016b) Gadelha et al. 2017). Here one may be interested in tasks such unconditional generation and reconstruction.
voxels. This however is ineffective in larger resolutions and does not make
More recently authors have considered alternative representations of shapes to a standard 3D dis- cretized of voxels(Choy al. 2016,] Girdhar et al. 2016b Wu et al. 2016a Yan et al. 2016 Zhu et al.) 2017), one that can permit more efficient learning and generation. These include point clouds Fan et (2017), meshes( Wang al. [2018] Georgia Gkioxari 2019). and signed distance transform based representations (Michalkiewicz et al. 2019 Park et al. 2019). To date there is not an agreed upon canonical 3-D shape representation
use with deep learning models. Further-
more authors have considered alternative architectures aimed at dealing with this problem (Richter & Roth 2018 Tatarchenko al. 2017p. However, this work, we ask whether a simpler approach can yield strong results. Building on the recent use of the Signed Distance Function (SDF) in shape representation we demonstrate simple latent shape representation that can be used downstream tasks and casily decoded. More specifically, in this work, we consider latent shape representation obtained by applying PCA on the SDF transformed shape. We show this leads to a latent shape representation that can be used directly in downstream tasks like 3D shape reconstruction from single view and 3D shape completion from a point cloud.
Our work a) reinforces the relevance of SDF representation for 3D deep learning b) demonstrates that a simple representation obtained by applying PCA on the SDF transform can lead to an effective latent shape representation. This representation allows for results competitive to state-of-the-ar

---

Under review as a conference paper at ICLR 2020
in standard benchmarks. Our work also suggests complex benchmarks than those currently considered may be needed to push forward the study of learned 3D shape reconstruction.
discuss the related work. We outline the basic methods used in the experiments in Sec. 3 We show extensive quantitative and experimental results comparing the eigenSDF approach existing methods in Sec.
The paper is structured follows. In Sec.
RELATED WORK
Point cloud based representation require tedious step of sampling points from the surface and to generate shape subsequently inferring the continuous shape from sample of points. Meshes present a challenge in that no clear way to generate valid meshes is available. Proposals have consisted of starting with template shapes and progressively deforming them Wang al. (2018). This how- ever can be problematic as it never explicitly represents the shape and may suffer issues with local coherence.
Michalkiewicz et al. 2019) and Park et al. (2019) also use the SDF representation. Unlike our work Deep Level Sets still relies on decoder encoder CNN architecture thus not removing the desired computational constraints associated with the 3D shape modeling. DeepSDF attempts to directly fit a continuous function to each shape which gives the SDF representations, Although it avoids discretization this function can lead to complex decoder model. for example an 8 layer network used to fit the SDF. Another recent work Mescheder al.) (2018) similar in spirit to PPark et al. (209)) learns classifier to predict whether point is inside or outside of the boundary, using this classifier as the shape representation. Unlike our proposal these methods cannot easily learn a latent shape representation to be applied on downstream task, since the shape is represented by the weights of the classifier or regression model. On the other hand our latent representation can easily encode an unseen shape and be easily used as prediction target for deep learning models.
Our work can also be seen as complementary to the very recent observations in Tatarchenko et al. (019) which highlights that good 3D single view reconstruction performance be achieved by using retrieval or clustering methods. We note however the descriptors used that work are more complex.
PCA has been classically used to represent shapes in a variety of contexts. For example classical methods in computer vision such the active appearance model Edwards al. (1998) and the 3D morphable model used in face analysis Blanz et al. (1999) are based on PCA shape representations. However, these typically are applied in different context requiring transforming the shape reference set of points and applying PCA on the coordinates. Leventon et al. (2002) used signed distance functions to embed 2D curves applying PCA obtain statistical models. To the best of our knowledge it has not been combined with the SDF representing a surface in 3D. We note level set methods and the SDF have only recently been revisited as an effective representation that can be combined with 3D deep learning Michalkiewicz et al. (2019); Park et al. (2019). Moreover it is enlightening that this classic approach to shape representation can be competitive with deep learning methods on standard benchmarks.
METHODS
In this section, we review the SDF transform and describe the simple latent representation that we consider.
3.1 SIGNED DISTANCE FUNCTIONS
The Signed Distance Function (SDF) of given closed surface I' C R is defined by the signed distance d IR3 x 3R3 + from any point x E R33 to the surface T. Given a discretized grid we
represent each shape an SDF on Ω by
(x)x) + inf d(z., y),
yE
with the convention that (b(x:) is positive the interior and negative the exterior of T.
2

---

Under review as a conference paper at ICLR 2020
Learn EigenSDF
SDF
PCA
Engenvediors
Downstream lasks
Encode
X
Eigenveclors
PointNet
MSE
Initialize
SDF Shape
Loss
Fine-Tuning
Figure I: Overview of our experiments. We construct eigenSDF representations and evaluate them on multiple downstream tasks. As an additional approach we also consider finetuning the decoder using the eigenvectors from PCA as initializations and the chamfer loss
In Michalkiewicz al. 2019) a CNN decoder model is used predict the SDF representation from latent space as well as to learn autoencoders. We note however that this representation is well structured and objects are often grouped by category, we thus ask if much simpler linear and non-convolutional decoder model can be effective.
Michalkiewiez al. (2019) further considers loss function for the SDF representation that approx- imately minimizes the chamfer loss.
L.( = Σ JED
Σ 8. ( LER
ΣΣ (1VV - 1)²
+
(2)
JED
with & being approximated Dirac delta, 6 inferred Signed Distance Function, and d(x)) the closest distance between grid point x and the ground truth shape. We will further leverage this representa- tion in the sequel.
3.2 EIGENSDF
We apply the PCA transform S (dee)i..NN, where N is the number of training examples. The eigenvectors E have the shape of (k:, M3) with M being the grid resolution. We project each SDF to dh F.. Here &e has shape of (1, k). In the sequel EigenSDF. Note that applying PCA to the naive voxel representation would be inappropriate as the data is binary and largely zeros. For downstream tasks we predict directly the latent representation Φ.. We will also consider using E as an initialization which is finetuned by training on the SDF shape representation directly using Eq. 2 high level overview of our framework is given in Figure [
will denote this representation
the
EXPERIMENTS
We evaluate the proposed representations on 3 tasks. These are single view 3D reconstruction, 3D reconstruction from point cloud, and 3D reconstruction with autoencoders. We take many of

---

Under review as a conference paper at ICLR 2020
baselines and comparisons from the recently proposed Mescheder et al. (2018) These applications are evaluated on 13 categories from ShapeNet repository (Chang et al. 2013).
Preprocessing In order to work on SDFs we need to have a well defined interior and exterior of an object, we have first preprocessed the meshes to make them watertight using method proposed by Stutz & Geiger (2018). Following common practice we render each ground truth mesh into 24 2D views using equally spaced azimuth angles. For each ground truth mesh get corresponding SDF & of shape 128 x 128 x 128.
Metries Following Mescheder et al. (2018) experimental setup, we report 3 metrics. The first one is Intersection Union (IoU), also known Jaccard Index, between ground truth shape S and prediction S:
SSn USUS
IoU
Second metric measures reconstruction error using symmetric chamfer distance:
chamfer(Sp..o)) P
min p
min p
ql
pEP PEQ
2
pEp
qEQ
Finally, we measure the angular distance using normal consistency (nc) metric:
Σ (Ns, No. (ns, (p)) SQ) 2|P
1
Σ
Ny. (4) (ns,
2|Q1
DEQ
where (p)p) denotes normal point p lying on surface and (4) denotes nearest neighbour of point lying surface S.
Chamfer distance and normal consistency metrics are evaluated by sampling 10a points from the surfaces, while the IoU is caleulated by sampling 10 points from within the bounding volume.
4.1 3D RECONSTRUCTION FROM SINGLE 2D VIEW
In this set of experiments, evaluate the eigenSDF approach from Sec 3 lution of 1283_ For each category we perform PCA on SDFs. We chose & to capture at least 99.5% of the variance within the dataset. The image encoder is 2D CNN whose architecture is taken from Richter & Roth 2018). We minimize the l2 loss between d.., the SDF projected into the latent space, and the prediction of the 2D CNN. We train this network for 100 epochs using 1e-3 learning rate, later dropping to le-4. Furthermore, we consider finetuning the representation starting with the eigenvectors from PCA and using Eq this baseline is referred as eigenSDF (finetuned).
We use starting reso-
In order to demonstrate that a gain is made by PCA versus just architecture we also train linear autoencoder of the same size (M) and finetune with Eq This baseline is referred to as linearSDF and IinearSD((fnettur
Finally we compare to set of standard benchmarks from the recent literature including voxel based CNN encoder/decoder Choo et al. (2016), point cloud based methods Fan et al. (2017). mesh based method Wang et al. (008)) and the recently introduced ONet Mescheder et al. (2018).
Complete results are given in Table [ First we observe that the simply using a same sized linear model linearSDF (finetune) is outperformed by using the eigenSDF: Compared to alternatives our method gives more significant gains in chamfer metric than all competitors and can be further im- proved with the finetuning. We also observe improvements in the normal consistency metric. For the IoU metric observe that eigenSDF outperforms all methods except Mescheder et al. (2018). Note that according to Sun et al. (2018) chamfer distance a far better metric for shape comparison than loU.

---

Under review as a conference paper at ICLR 2020
Table 1: Single-view 3D reconstruction results on ShapeNet. We observe that compared other state-of-the-art learning based methods eigenSDF outperforms in normal consistency and Chamfer distance. Finetuning can further improve this result. Compare to training a linear autoencoder or just finetuning the performance is substantially better, showing that eigen decomposition obtains best results.
Chamfer ONet
Cat name
3D3 R2N2
PSGN
Pix2Mesh
AtlasNet
linearSDI
linearSDDP (ft)
eigenSDD
eigenSDF (n)
airplane
0.137
0.187
0.T04
0.2.56
0.102
bench
0.194
0.181
0.201
0.138
0,155
0.247
0.241
0.085
0.072
cabinet
0.217
0.215
0.196
0.175
0.167
0.223
0.220
0.082
0.065
car
0.213
0.180
0.141
0.159
0.232
0.072
chair
0.270
01.247
0.265
0.209
0,228
0.265
0.261
0.110
0.092
display
0,314
0.284
0,239
0.198
0.278
0.286
0.258
0.115
0.115
lamp
0.633
0.465
0.377
loudspeaker
0.318
0.316
0.285
0,245
0,300
0.262
0,257
0.097
0.092
rifle
0.183
0.134
0.164
0.115
0.141
0.282
0.267
0.148
0.140
table
0.239
0.222
0.218
0,190
0,189
0.201
0,195
0.127
0.115
telephone
0.195
0.161
0.149
0.128
0.140
0.181
0.177
0.047
0.045
vessel
0.278
0.215
0.216
0.175
0.215
0.279
0.273
0.163
0.142
IoUt
Cat nume
3D R2N2
PSGN
Pix2Mesh
AclasNet
ONet
lineurSDP
lineurSDF (ft)
eigenSDD
eigenSDP
sirplane
0.426
0,420
0.571
0.425
0.433
0,465
0.487
beach
0,373
0.323
0.485
0.372
0.375
0.382
0.405
cabinet
0.667
0.664
0.733
0.658
0.661
0.670
0.686
0.552
0.665
0.669
chair
0.439
0.396
0.501
0.385
0.401
0.402
0.414
display
0.440
0.490
0.471
0.388
0,395
0.406
0.424
lamp
0,281
0.323
0.371
0,201
0.211
0,230
0,280
loudspeaker
0.611
0.599
0.647
0.554
0,561
0,599
0.609
rifle
0.375
0,4012
0.474
0,256
0.261
D.289
0.299
sofa
0.626
0.613
0.680
0.611
0.619
0.633
0.639
table
0.420
01,395
0.506
0.401
0.406
01.22
0)426
telephone
0.611
0.661
0,720
0.590
0.615
0.685
0.715
vessel
0.482
0.397
0.530
0.443
0.449
0.494
0.508
0.493
0.480
0.571
0.457
0.465
0.488
0.507
Normal Consistency!
Cat name
R2N2
PSON
Pix2Mesh
AtlasNet
ONei
lineurSDP
LinearSDP (ft)
eigenSDF
eigenSDP (It)
airplane
0.629
0.759
0.836
0.840
0.713
0.717
0.809
0.813
bench
0.678
0.732
0.779
0.813
0.754
0.762
0.815
0.832
cabinet
0.782
0.834
0.850
0.879
0.771
0.773
0.880
0.882
0.714
0.756
0.787
0.802
0.855
0.878
chair
0.663
0.746
0.791
0.823
0.757
0.777
0.824
0,828
display
0.720
0.830
0.858
0.854
0.755
0.787
0.859
0.859
lamp
0.560
0.666
0.694
0.731
0.575
0.584
0.741
0.775
loudspeaker
0.711
0.782
0.825
0.832
0.745
0.755
0.839
0.846
rifle
0.670
0.718
0.725
0.766
0.668
0.695
0.768
0.793
sofa
0.731
0.820
0.840
0.863
0.733
0.739
0.861
0.863
tuble
0.732
0.784
0.832
0.858
0.731
0.735
0.801
0,808
telephone
0.817
0.907
0.923
0.935
0.851
0.855
0.938
0.940
0.629
0.699
0.756
0.794
0.704
0.735
0.802
0,811
meun
0.695
0.772
0.811
0.834
0.734
0,747
0.830
0,840
4.2 3D SHAPE COMPLETION FROM POINT CLOUDS
We next consider shape completion from point cloud. This task has been considered in e.g- Mescheder et al. (2018). Similar experimental setup of Mescheder al. (2018) we use 13 cat- egories from ShapeNet repository and preprocess the meshes make them watertight. We randomly sample 300 points from ground truth meshes and add a Gaussian noise with 0 mean and 0.05 std. We have used the same metrics in section 4.1
We have used PointNet encoder with a bottleneck dimension 512 Qi et al. (017) and linear decoder from section 4.1 We use similar set of baselines in the previous section and compare to eigenSDF. We observe similar large gains in the chamfer metric and competitive performance in other metrics. Results shown in Table
5

---

Under review as a conference paper at ICLR 2020
We can that, similar to 3D reconstruction task, our performance in chamfer is much better, normal consistency is similar and iou is worse but still competitive.
Table 2: Results on 3D Shape completion
IoUt
Chamfer
nc
eigenSDD (ours)
0.571
0.073
0.853
3D-R2N2 (Choy et al (0166)
0.565
0.169
0.719
PSGN Fan al. 20
0,144
DMC Liao al. 2018
0.674
0.117
0.848
ONet Mescheder et al. 2018
0.778
0.079
0.895
Table 3: We compare eigenSDF to other methods in terms of reconstruction. We find that compared linear autoencoders trained voxels SDFs, it performs better. We also compare more sophisticated methods.
method
loUf
Chamfer
NCT
F-scoref
eigenSDF
0.746
0.0425
0.869
0.484
eigenSDF (ft)
0.758
0.0325
0.896
0.529
Linear (Ø) (chamfer)
0.582
0.050
0.773
0.315
Linear( voxels)
0.637
0.067
0.737
0.384
DLS Miihaakiewiez et 2019)
0.681
0.047
0.858
0.103
TL. (Girdhar et al. 201 6b))
0.656
0.082
0.847
0.081
4.3 3D RECONSTRUCTION FROM LATENTS
Finally we consider simple 3D reconstruction task Girdhar et al. (2). This can also be viewed measuring the representational power of the model Mescheder et al. (2018). This evaluates construction quality eigenSDF versus other methods particularly CNN based autoencoders. The goal is to reconstruct test set shapes. We done in other works Mescheder et al. (2018). We use cars category from ShapeNet repository and evaluate reconstruction unseen data. For the evaluations in addition the metrics used in section we further analyse the decoders using F-score Knapitsch er al. 2017
initial resolution of 128 and reduce it to k = 512 as
training time for iteration
10?
DLS Voxidense) eigenSD((t)
101
100
32
64
128
resolution
Figure 2: Training time of eigenSDF (finetuned) and dense convnets (Voxels, DLS). Figure shows on a logscale amount seconds network needs for forward and backward pass of iteration using a batchsize of 32. Resolution 256 not shown due to clarity.

---

Under review as a conference paper at ICLR 2020
category
DLS
eigenSDF(ft)
IoUt
Chamf
NCT
F-scoret
loUt
Chamff
NCt
F-scoret
cars
0.784
0.055
0,804
0,148
0.821
0.040
0.909
0.432
chairs
0.434
0.360
0.743
0.066
0.553
0.125
0.820
0.168
sofas
0.581
0.132
0.779
0.089
0.647
0.082
0.867
0.248
Table 4: Comparison to the SDF based method Michalkiewicz et al. (2019) in single view recon- struction. There is marked improvement due to the ability to model higher resolution.
4.4 COMPARISON WITH DEEP LEVEL SETS
In this section, we compare our method to the other recent approach relying on the signed distance transform Michalkiewiez al. (2099) and learning with the chamfer loss Le. This one however uses the CNN decoder model and does not learn a latent shape representation. We have chosen similar experimental setup of 3 subsets each having 2 000 examples from ShapeNet repository: cars, sofas, chairs. We observed that remaining 2 categories, bottles and phones, are too simple to see difference in higher resolution. Reported metrics might differ due different pre-processing techniques. Networks here are the one used in Michalkiewicz et al. (2019) paper and the one used in Section 4.1 We first compare the training time for both methods for various resolutions. Figure shows the result demonstrating that it is not feasible to use the CNN decoder in higher resolutions. consistent with the findings of (Richter Roth 2018).
4.5 RECONSTRUCTION AND GENERATION
Finally evaluate the performance on the single view reconstruction qualitatively. In Table 4 we can see that reconstructions (on unseen data) can be effective capturing more complex structures ignored by Michalkiewicz et al. (201)). Multiple authors have also consider generating uncondi- tionally shapes, typically using sophisticated non-linear deep learning models like GANs and VAEs. We compare some of these to sampling a gaussian in the latent space of the eigenSDF, some qualita- tive results are shown in Figure 3 We observe this simple approach yields some shapes comparable those o the complex non-linear models.
Figure 3: We compare unconditional generations of cars category. Generations from gaussian fit to eigenSDF is shown in the top row (blue). Second row are generations from Mescheder et al. (2018) and the third row is from 3D GAN Wu al. (2016b)
CONCLUSION
We have shown that using simple linear decoder coupled with the SDF representation yields com- petitive results. The SDF lends itself to the application of PCA yielding strong but simple baseline for future work in learned 3D shape analysis. Moreover, our work suggests that more complex baseline datasets may be needed to further evaluate deep learning methods on 3D shape inference.

---

Under review as a conference paper at ICLR 2020
2D view
GT
Deep Level Sets
eigenSDF
Table 5: We compare reconstruction of eigenSDF and the CNN decoder based Michalkiewicz et al. 2019) (which also uses SDF representations). eigenSDF permits operate higher resolution and generally produces more locally coherent results.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Volker Blanz, Thomas Vetter, et al. A morphable model for the synthesis of 3d faces. In Siggraph, volume 99, pp. 187-194, 1999.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012.2222 2015.
Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A uni- fied approach for single and multi-view 3d object reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
Gareth J Edwards, Christopher J Taylor, and Timothy F Cootes. Interpreting face images using active appearance models. Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, pp. 300-305, IEEE, 1998.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object recon- struction from single image. volume 2, pp. 6, 2017.
Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shape induction from 2d views of multiple objects. In 2017 International Conference on 3D Vision (3DV), pp. 402-411. IEEE, 2017.
Justin Johnson Georgia Gkioxari, Jitendra Malik. Mesh r-cnn. ICCV, 2019.
R. Girdhar, D.F. Fouhey, M. Rodriguez, and A. Gupta. Learning predictable and generative vector representation for objects. ECCV, 2016a.
Rohit Girdhar, David Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning a predictable and generative vector representation for objects. In European Conference Computer Vision, pp- 484-499. Springer, 2016b.
Arno Knapitsch, Jaesik Park, Qian Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4), 2017.
Michael E Leventon, W Eric L. Grimson, and Olivier Faugeras. Statistical shape influence in geodesic active contours. 5th IEEE EMBS International Summer School on Biomedical Imag- ing, 2002., pp. 8-pp. IEEE, 2002.
Yiyi Liao, Simon Donné, and Andreas Geiger. Deep marching cubes: Learning explicit surface representations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 2916-2925, 2018.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space, 2018
Mateusz Michalkiewicz, Jhony K. Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders P. Eriksson. Deep level sets: Implicit surface representations for 3d shape inference. In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 174, 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets
for 3d classification and segmentation. 1(2):4, 2017.
Stephan R. Richter and Stefan Roth. Matryoshka networks: Predicting 3d geometry via nested
shape layers. In CVPR, pp. 1936-19444 1944. IEEE Computer Society, 2018. URL http:://b//p uni-trier: hmmi-ttrierr.deebb/coo//////ppp
html #Ri chter018

---

Under review as a conference paper at ICLR 2020
David Stutz and Andreas Geiger. Learning 3d shape completion from laser scan data with weak supervision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, 2018.
Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree generating networks: Efficient convo- lutional architectures for high-resolution 3d outputs. In IEEE International Conference on Computer Vision (ICCV), 2017. URL http:///mme inbbbiiffomaii ure-briiburggggg de/
PubIicattoon//201//BBB
Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view 3d reconstruction networks learn? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3405-3414 2019.
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. ECCV, 2018.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a prob- abilistic latent space of object shapes via 3d aenerative-adversariall modeling. In Advances in neural information processing systems, pp. 82-90, 2016a.
Jijjun Wu, Chengkai Zhang, Tianfan Xue, William Freeman, and Joshua B Tenenbaum. Learning probabilistic latent space of object shapes via 3d generative-aaversariil modeling. In Advances in Neural Information Processing Systems, pp. 82-90, 2016b.
Xinchen Yan, Jimei Yang. Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In Advances in Neural Information Processing Systems, pp. 1696-1704, 2016.
Rui Zhu, Hamed Kiani Galoogahi, Chaoyang Wang, and Simon Lucey. Rethinking reprojection: Closing the loop for pose-aware shape reconstruction from a single image. In The IEEE Interna- tional Conference on Computer Vision (ICCV), Oct 2017.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 NUMBER OF EIGENVECTORS
In Figure we show the number of cigenvectors which captures 99.5% of variance within the dataset of approximately 6000 examples of ShapeNeiCars. Number of eigenvectors required represent shapes from other categories vary from 512 to 2048, shown in Figures 6
Amount of eigenvectors sufficient to represent ShapeNetCar category
1,0
0.9
0.8
Captured variance
0.7
car
400
800
1200
1600
2000
Number eigenvectors
Figure 4: Cumulative fraction total variance captured by eigenvectors obtained from applying PCA on ShapeNettars dataset.
Amount of eigenvectors sufficient to represent large ShapeNer categories
0.9
Captured vaniance varianee
0.7
0.6
sofa
chair
car
table
400
800
1200
1600
2000
Number eigenvectors
Figure 5: Cumulative fraction of total variance captured by eigenvectors obtained from applying PCA on large ShapeNer categories.
A.2 PCA APPLIED TO OTHER SHAPE REPRESENTATIONS
As mentioned in the Section there are commonly used shape representations: voxels, Signed Distance Functions (SDF), point clouds and meshes. Because of lack of canonical order or repre- sentation it is hard apply PCA point clouds meshes in straightforward way. Compared to
11

---

Under review as a conference paper at ICLR 2020
Amount of eigenvectors sufficient
to represent small ShapeNet categories
Captured variance
0.8
0.7
rifle
watercraft
bench
cabinet
display
phone
speaker
200
400
600
800
1000
Number eigenvectors
Figure 6: Cumulative fraction of total variance captured by eigenvectors obtained from applying PCA on small ShapeNet categories.
Figure 7: SDF-based pea reconstruction of an example from Shape dataset. Left Level set of a SDF of a car with resolution 1283, right - pea- based recostruction using 2048 eigenvectors.
Figure 8: Failed attempt of reconstructing an example from ShapeNetCars dataset using voxel-based representation. Left voxelized car with resolu- tion 1283, right pea-based recostruction using 2048 eigenvectors
the results of pea reconstruction using Signed Distance Function shape representation, applying PCA directly on voxels yeilds poor results, as shown in Figures 78
12


Reviews:
[{"review_id": "rJe5U-qh9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of learning the feature representation for predicting the 3D shape of objects, from a single image or a point cloud. The proposed approach performs PCA on the SDF field. And then the transformed feature map is learned and used as input to task-specific decoders for 3D shape prediction. The authors claims that this approach trains faster and is easier to scale, while showing competitive performance compared to state-of-the-art methods.\n\nI am leaning towards Weak Reject. The paper is generally easy to read, but with some details missing. And I found the discussion of the results to be insufficient. I think it can be an above-threshold paper if questions are addressed during rebuttal.\n\nBeing able to easily scale to higher resolutions is claimed to be one of the main advantages, but I am not convinced that this is useful under this setting. If I understand this correctly, the number of eigenvectors k is fixed, and projecting the SDF field to this space would remove the higher frequency components of the shape. So wouldn't the number of eigenvectors be the bottleneck in representational precision, not the resolution of the output space?\n\nWhat is the chosen k (number of eigenvectors)? It says k was \"chosen to capture 99.5% of the variance within the dataset\", but I could not find how exactly it was chosen and what value of k was used (I apologize if I missed).\n\nAlso, I think the PCA is category-specific (page 4, section 4.1). Is k dependent on the category or is it the same across all categories? Some of the other methods (if not all) used for comparison are not category-specific, so if this is true, I think the comparison may not be entirely fair and it should be made clearer.\n\nI think the writing could be polished as well, some minor typos:\n\nPage 2:  3rd and 4th paragraphs: continous\nPage 2: anlaysis, enlightning\nPage3: under eigenSDF: reprsentation\nPage 4: section 4.1: refered, signficant\nPage 5: “seciton”\nPage 7: tranform\n\nPage 3, Section 3.2: Is N the number of training examples and M the resolution?\nFigure 2 not referred in the main text.\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1laIOJ6Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a 3D object reconstruction/completion algorithm that utilizes a simple decoder from features generated using PCA of SDF. The approach was tested in a few experiments on public benchmarks and achieves competitive results. \n\nThe overall presentation of the paper is decent, and the network structure of the proposed approach is reasonable. It's interesting to see that using a simple PCA can help improve performance using a simple network structure. The experimental results make sense, and it's nice to see the performance is reasonable as well.\n\nI have a few questions regarding the paper:\n- Without looking at the code, I don't think I fully understand the formulation of the network structure just by reading the text. For example, what is the dashed line mean in Fig. 1? What's the meaning of the dot over \"E\" in Sec 3.2, is it \"derivatives\"? If so, why not use this symbol in Eq. (2) as well? In general, I find it's a bit hard to follow when only 2-3 paragraphs are used for describing the proposed approach. It'll be good if the authors can elaborate on the approach in a more thorough manner. \n- Regarding experimental results, it's nice to see that eigenSDF is better than linearSDF, demonstrating that the approach is quite effective. However it is not always the best in several metrics (as indicated in experimental result tables). I wonder if authors can provide more analysis or discussions on why this could happen, either the metric may not make too much sense in their setting, or if there is potential room for improvement. A few failure case visualizations could also be helpful in understanding the issues of the proposed approach.\n- Moreover, do authors have thoughts on eigenSDF vs deepSDF (cited in the paper, published in CVPR 2019)? It'll be interesting to compare those as well, as deepSDF has proven useful in a few papers already. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lQSgTfcS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thank the authors for the response. I am still in favor of the idea -- applying simple, old-school method into a new problem, and I also agree with R1 and R2 that the paper is currently lack of details and experimental results. I will keep my score, but would not fight for the acceptance if R1 and R2 insist.\n----------------------------------------\nSummary\nThis paper presents a new method for 3D shape reconstruction, based on SDF (Signed Distance Function) and PCA. The basic idea is to conduct PCA on all the shapes with SDF as feature, and encode a shape by the eigenvectors from PCA. The authors present experiments on 3D reconstruction from 2D view and point clouds, which demonstrate the effectiveness of the proposed method. I lean to vote for accepting this paper since the idea is simple but novel, and it achieves good performance.\nStrengths\n- The idea itself is simple and novel. The basic idea of this approach is simple -- keep most information / variance by using PCA, and it is also very novel, since I have not seen papers using PCA to encode 3D shapes.\n- The idea is effective. As the authors demonstrated in section 4, this approach works well, and it outperforms all other methods by a large margin according to Chamfer distance. This is impressive since such a simple method can improve the performance this much.\nWeaknesses\n- More analysis could be provided about how do the authors choose SDF. Choosing SDF here is obviously a reasonable choice, but is it the best? More analysis could be provided, or more experiments could be included.\nPossible Improvements\nAs mentioned above, more analysis about why choosing SDF or more experiments about comparing SDF to other representations under this PCA approach could be provided.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use PCS to replace the conventional decoder for 3D shape reconstruction. It shows competitive performance to the state of the art methods. While reviewer #3 is overall positive about this work, both reviewer #1 and #2 rated weak rejection. Reviewer #1 concerns that important details are missing, and the discussion of results is insufficient. Reviewer #3 has questions on the clarity of the presentation and comparison with SOTA methods. The authors provided response to the questions, but did not change the rating of the reviewers. The ACs agree that this work has merits. However, given the various concerns raised by the reviewers, this paper can not be accepted at its current state.
Decision: Reject