Title: A SIMPLE AND SCALABLE SHAPE REPRESENTATION FOR 3D RECONSTRUCTION

Abstract: Deep learning applied to the reconstruction of 3D shapes has seen growing interest. A popular approach to 3D reconstruction and generation in recent years has been the CNN decoder-encoder model often applied in voxel space. However this often scales very poorly with the resolution limiting the effectiveness of these models. Several sophisticated alternatives for decoding to 3D shapes have been proposed typically relying on alternative deep learning architectures. We show however in this work that standard benchmarks in 3D reconstruction can be tackled with a surprisingly simple approach: a linear decoder obtained by principal component analysis on the signed distance transform (SDF) of the surface. This approach allows easily scaling to larger resolutions. We show in multiple experiments it is competitive with state of the art methods and also allows the decoder to be fine-tuned on the target task using a loss designed for SDF transforms, obtaining further gains.

1. INTRODUCTION:

In recent years there has been an increased interest in extending the successes of deep learning to problems requiring analysis and representation of 3D shapes. This includes many long standing tasks such as 3D shape reconstruction (Wu et al., 2016a; Choy et al., 2016) from single or multiple views, as well as shape completion Mescheder et al. (2018) . There are a number of applications of these methods in robotics, surgery, and augmented reality.

A popular approach has been the use of CNN decoder-encoder architectures Choy et al. (2016) as popularized in applications like segmentation Chen et al. (2017) . Here for example in the single view reconstruction 2D CNN will encode the 2-D image and a 3D CNN decoder model will produce the final representation in voxels. This however is ineffective in larger resolutions and does not make full use of the structure of the object. Similar problems arise in more general attempts to learn latent variable models of shapes (Wu et al., 2016b; Gadelha et al., 2017) . Here one may be interested in tasks such as unconditional generation and reconstruction.

More recently authors have considered alternative representations of shapes to a standard 3D discretized set of voxels (Choy et al., 2016; Girdhar et al., 2016b; Wu et al., 2016a; Yan et al., 2016; Zhu et al., 2017) , one that can permit more efficient learning and generation. These include point clouds Fan et al. (2017) , meshes (Wang et al., 2018; Georgia Gkioxari, 2019) , and signed distance transform based representations (Michalkiewicz et al., 2019; Park et al., 2019) . To date there is not an agreed upon canonical 3-D shape representation for use with deep learning models. Furthermore authors have considered alternative architectures aimed at dealing with this problem (Richter & Roth, 2018; Tatarchenko et al., 2017) . However, in this work, we ask whether a simpler approach can yield strong results. Building on the recent use of the Signed Distance Function (SDF) in shape representation we demonstrate a simple latent shape representation that can be used in downstream tasks and easily decoded. More specifically, in this work, we consider a latent shape representation obtained by applying PCA on the SDF transformed shape. We show this leads to a latent shape representation that can be used directly in downstream tasks like 3D shape reconstruction from a single view and 3D shape completion from a point cloud.

Our work a) reinforces the relevance of SDF as a representation for 3D deep learning b) demonstrates that a simple representation obtained by applying PCA on the SDF transform can lead to an effective latent shape representation. This representation allows for results competitive to state-of-the-art in standard benchmarks. Our work also suggests more complex benchmarks than those currently considered may be needed to push forward the study of learned 3D shape reconstruction.

The paper is structured as follows. In Sec. 2 we discuss the related work. We outline the basic methods used in the experiments in Sec. 3. We show extensive quantitative and experimental results comparing the eigenSDF approach to existing methods in Sec. 4.

2. RELATED WORK:

Point cloud based representation require a tedious step of sampling points from the surface and to generate shape subsequently inferring the continuous shape from a sample of points. Meshes present a challenge in that no clear way to generate valid meshes is available. Proposals have consisted of starting with template shapes and progressively deforming them Wang et al. (2018) . This however can be problematic as it never explicitly represents the shape and may suffer issues with local coherence. Michalkiewicz et al. (2019) and Park et al. (2019) also use the SDF representation. Unlike our work Deep Level Sets still relies on a decoder encoder CNN architecture thus not removing the desired computational constraints associated with the 3D shape modeling. DeepSDF attempts to directly fit a continuous function to each shape which gives the SDF representations. Although it avoids discretization this function can lead to a complex decoder model, for example an 8 layer network is used to fit the SDF. Another recent work Mescheder et al. (2018) similar in spirit to Park et al. (2019) learns a classifier to predict whether a point is inside or outside of the boundary, using this classifier as the shape representation. Unlike our proposal these methods cannot easily learn a latent shape representation to be applied on downstream task, since the shape is represented by the weights of the classifier or regression model. On the other hand our latent representation can easily encode an unseen shape and be easily used as a prediction target for deep learning models.

Our work can also be seen as complementary to the very recent observations in Tatarchenko et al. (2019) which highlights that good 3D single view reconstruction performance can be achieved by using retrieval or clustering methods. We note however the descriptors used in that work are more complex.

PCA has been classically used to represent shapes in a variety of contexts. For example classical methods in computer vision such as the active appearance model Edwards et al. (1998) and the 3D morphable model used in face analysis Blanz et al. (1999) are based on PCA shape representations. However, these typically are applied in a different context requiring transforming the shape to a reference set of points and applying PCA on the coordinates. Leventon et al. (2002) used signed distance functions to embed 2D curves applying PCA to obtain statistical models. To the best of our knowledge it has not been combined with the SDF representing a surface in 3D. We note level set methods and the SDF have only recently been revisited as an effective representation that can be combined with 3D deep learning Michalkiewicz et al. (2019) ; Park et al. (2019) . Moreover it is enlightening that this classic approach to shape representation can be competitive with deep learning methods on standard benchmarks.

3. METHODS:

In this section, we review the SDF transform and describe the simple latent representation that we consider.

3.1. SIGNED DISTANCE FUNCTIONS:

The Signed Distance Function (SDF) of a given closed surface Γ ⊂ R 3 is defined by the signed distance d : R 3 × R 3 → R from any point x ∈ R 3 to the surface Γ. Given a discretized grid Ω we can represent each shape as an SDF on Ω by

EQUATION

with the convention that φ i (x) is positive on the interior and negative on the exterior of Γ. We construct eigenSDF representations and evaluate them on multiple downstream tasks. As an additional approach we also consider finetuning the decoder using the eigenvectors from PCA as initializations and the chamfer loss

In Michalkiewicz et al. (2019) a CNN decoder model is used to predict the SDF representation from a latent space as well as to learn autoencoders. We note however that this representation is well structured and objects are often grouped by category, we thus ask if a much simpler linear and non-convolutional decoder model can be effective. Michalkiewicz et al. (2019) further considers a loss function for the SDF representation that approximately minimizes the chamfer loss.

L (θ) = j∈D x∈Ω δ ( φj (x))d j (x) p 1/p + α j∈D x∈Ω ( ∇ φj (x) -1) 2 (2)

with δ being approximated Dirac delta, φ inferred Signed Distance Function, and d(x) the closest distance between grid point x and the ground truth shape. We will further leverage this representation in the sequel.

3.2. EIGENSDF:

We apply the PCA transform to S = {φ i } i=..N , where N is the number of training examples. The eigenvectors E have the shape of (k, M 3 ) with M being the grid resolution. We project each SDF to φ c = φE T . Here φ c has a shape of (1, k). In the sequel we will denote this representation as the EigenSDF. Note that applying PCA to the naive voxel representation would be inappropriate as the data is binary and largely zeros. For downstream tasks we predict directly the latent representation φ c . We will also consider using E as an initialization which is finetuned by training on the SDF shape representation directly using Eq. 2. A high level overview of our framework is given in Figure 1 .

4. EXPERIMENTS:

We evaluate the proposed representations on 3 tasks. These are single view 3D reconstruction, 3D reconstruction from point cloud, and 3D reconstruction with autoencoders. We take many of our baselines and comparisons from the recently proposed Mescheder et al. (2018) . These applications are evaluated on 13 categories from ShapeNet repository (Chang et al., 2015) .

Preprocessing In order to work on SDFs we need to have a well defined interior and exterior of an object, we have first preprocessed the meshes to make them watertight using method proposed by Stutz & Geiger (2018) . Following common practice we render each ground truth mesh into 24 2D views using equally spaced azimuth angles. For each ground truth mesh we get a corresponding SDF φ of shape 128 × 128 × 128.

Metrics Following Mescheder et al. (2018) experimental setup, we report 3 metrics. The first one is Intersection over Union (IoU), also known as Jaccard Index, between ground truth shape S and prediction S:

IoU = |S ∩ S| |S ∪ S|

Second metric measures reconstruction error using symmetric chamfer distance:

chamfer(S P , SQ ) = 1 2|P | p∈P min q∈Q |p -q| + 1 2|Q| q∈Q min p∈P |p -q|

Finally, we measure the angular distance using normal consistency (nc) metric:

nc(S P , SQ ) = 1 2|P | p∈P |N S P (p) • N SQ (n SQ (p))| + 1 2|Q| q∈Q |N SQ (q) • N S P (n S P (q))|

, where N S (p) denotes normal of point p lying on surface S and n S (q) denotes nearest neighbour of point q lying on surface S.

Chamfer distance and normal consistency metrics are evaluated by sampling 10 5 points from the surfaces, while the IoU is calculated by sampling 10 5 points from within the bounding volume.

4.1. 3D RECONSTRUCTION FROM SINGLE 2D VIEW:

In this set of experiments, we evaluate the eigenSDF approach from Sec 3. We use a starting resolution of 128 3 . For each category we perform PCA on SDFs. We chose k to capture at least 99.5% of the variance within the dataset. The image encoder is a 2D CNN whose architecture is taken from Richter & Roth (2018) . We minimize the 2 loss between φ c , the SDF projected into the latent space, and the prediction of the 2D CNN. We train this network for 100 epochs using 1e-3 learning rate, later dropping to 1e-4. Furthermore, we consider finetuning the representation starting with the eigenvectors from PCA and using Eq 2, this baseline is referred to as eigenSDF (finetuned).

In order to demonstrate that a gain is made by PCA versus just architecture we also train a linear autoencoder of the same size (M ) and finetune it with Eq 2. This baseline is referred to as linearSDF and linearSDF(finetune). Complete results are given in Table 1 . First we observe that the simply using a same sized linear model linearSDF (finetune) is outperformed by using the eigenSDF. Compared to alternatives our method gives more significant gains in chamfer metric than all competitors and can be further improved with the finetuning. We also observe improvements in the normal consistency metric. For the IoU metric we observe that eigenSDF outperforms all methods except Mescheder et al. (2018) . Note that according to Sun et al. (2018) chamfer distance is a far better metric for shape comparison than IoU. We next consider shape completion from a point cloud. This task has been considered in e.g. Mescheder et al. (2018) . Similar to experimental setup of Mescheder et al. (2018) we use 13 categories from ShapeNet repository and we preprocess the meshes to make them watertight. We randomly sample 300 points from ground truth meshes and add a Gaussian noise with 0 mean and 0.05 std. We have used the same metrics as in section 4.1.

We have used PointNet encoder with a bottleneck dimension of 512 Qi et al. ( 2017) and linear decoder from section 4.1. We use a similar set of baselines as in the previous section and compare to eigenSDF. We observe similar large gains in the chamfer metric and competitive performance in other metrics. Results are shown in Table 2 .

We can see that, similar to 3D reconstruction task, our performance in chamfer is much better, normal consistency is similar and iou is worse but still competitive. Finally we consider a simple 3D reconstruction task Girdhar et al. (2016a) . This can also be viewed as measuring the representational power of the model Mescheder et al. (2018) . This evaluates reconstruction quality of eigenSDF versus other methods particularly CNN based autoencoders. The goal is to reconstruct test set shapes. We use an initial resolution of 128 3 and reduce it to k = 512 as done in other works Mescheder et al. (2018) . We use cars category from ShapeNet repository and evaluate reconstruction on unseen data. For the evaluations in addition to the metrics used in section 4.1 we further analyse the decoders using F-score Knapitsch et al. (2017) . 2 shows the result demonstrating that it is not feasible to use the CNN decoder in higher resolutions.

It is consistent with the findings of (Richter & Roth, 2018) .

4.5. RECONSTRUCTION AND GENERATION:

Finally we evaluate the performance on the single view reconstruction qualitatively. In Table 4 we can see that reconstructions (on unseen data) can be effective capturing more complex structures ignored by Michalkiewicz et al. (2019) . Multiple authors have also consider generating unconditionally shapes, typically using sophisticated non-linear deep learning models like GANs and VAEs.

We compare some of these to sampling a gaussian in the latent space of the eigenSDF, some qualitative results are shown in Figure 3 . We observe this simple approach yields some shapes comparable to those o the complex non-linear models. 

5. CONCLUSION:

We have shown that using a simple linear decoder coupled with the SDF representation yields competitive results. The SDF lends itself to the application of PCA yielding a strong but simple baseline for future work in learned 3D shape analysis. Moreover, our work suggests that more complex baseline datasets may be needed to further evaluate deep learning methods on 3D shape inference. 

:



Figures and Tables:

Figure1: Overview of our experiments. We construct eigenSDF representations and evaluate them on multiple downstream tasks. As an additional approach we also consider finetuning the decoder using the eigenvectors from PCA as initializations and the chamfer loss

we compare to a set of standard benchmarks from the recent literature including voxel based CNN encoder/decoder Choy et al. (2016), point cloud based methods Fan et al. (2017), a mesh based method Wang et al. (2018), and the recently introduced ONet Mescheder et al. (2018).

Figure 2: Training time of eigenSDF (finetuned) and dense convnets (Voxels, DLS).Figure shows on a logscale amount of seconds a network needs for forward and backward pass of 1 iteration using a batchsize of 32. Resolution 256 not shown due to clarity.

Figure 3: We compare unconditional generations of cars category. Generations from a gaussian fit to eigenSDF is shown in the top row (blue). Second row are generations from Mescheder et al. (2018) and the third row is from a 3DGAN Wu et al. (2016b)

Figure 6: Cumulative fraction of total variance captured by eigenvectors obtained from applying PCA on small ShapeNet categories.

Figure 7: SDF-based pca reconstruction of an example from ShapeNetCars dataset. Left -Level set of a SDF of a car with resolution 128 3 , right -pcabased recostruction using 2048 eigenvectors.

Figure 8: Failed attempt of reconstructing an example from ShapeNetCars dataset using voxel-based representation. Left -voxelized car with resolution 128 3 , right -pca-based recostruction using 2048 eigenvectors

.

Single-view 3D reconstruction results on ShapeNet. We observe that compared to other state-of-the-art learning based methods eigenSDF outperforms in normal consistency and Chamfer distance. Finetuning can further improve this result. Compare to training a linear autoencoder or just finetuning the performance is substantially better, showing that eigen decomposition obtains best results.

Table content: <table><tr><td>Chamfer↓</td></tr></table>

Results on 3D Shape completion

Table content: <table><tr><td/><td colspan="3">IoU↑ Chamfer ↓ nc ↑</td></tr><tr><td>eigenSDF (ours)</td><td>0.571</td><td>0.073</td><td>0.853</td></tr><tr><td>3D-R2N2 (Choy et al. (2016))</td><td>0.565</td><td>0.169</td><td>0.719</td></tr><tr><td>PSGN (Fan et al. (2017))</td><td>-</td><td>0.144</td><td>-</td></tr><tr><td>DMC (Liao et al. (2018))</td><td>0.674</td><td>0.117</td><td>0.848</td></tr><tr><td>ONet (Mescheder et al. (2018))</td><td>0.778</td><td>0.079</td><td>0.895</td></tr></table>

We compare eigenSDF to other methods in terms of reconstruction. We find that compared to linear autoencoders trained on voxels or SDFs, it performs better. We also compare to more sophisticated methods.

Table content: <table><tr><td>method</td><td colspan="4">IoU↑ Chamfer↓ NC↑ F-score↑</td></tr><tr><td>eigenSDF</td><td>0.746</td><td>0.0425</td><td>0.869</td><td>0.484</td></tr><tr><td>eigenSDF (ft)</td><td>0.758</td><td>0.0325</td><td>0.896</td><td>0.529</td></tr><tr><td>Linear (φ) (chamfer)</td><td>0.582</td><td>0.050</td><td>0.773</td><td>0.315</td></tr><tr><td>Linear(voxels)</td><td>0.637</td><td>0.067</td><td>0.737</td><td>0.384</td></tr><tr><td colspan="2">DLS (Michalkiewicz et al. (2019)) 0.681</td><td>0.047</td><td>0.858</td><td>0.103</td></tr><tr><td>TL (Girdhar et al. (2016b))</td><td>0.656</td><td>0.082</td><td>0.847</td><td>0.081</td></tr><tr><td>4.3 3D RECONSTRUCTION FROM LATENTS</td><td/><td/><td/><td/></tr></table>

Comparison to the SDF based methodMichalkiewicz et al. (2019) in single view reconstruction. There is marked improvement due to the ability to model higher resolution.4.4 COMPARISON WITH DEEP LEVEL SETSIn this section, we compare our method to the other recent approach relying on the signed distance transformMichalkiewicz et al. (2019) and learning with the chamfer loss L . This one however uses the CNN decoder model and does not learn a latent shape representation. We have chosen a similar experimental setup of 3 subsets each having 2 000 examples from ShapeNet repository: cars, sofas, chairs. We observed that remaining 2 categories, bottles and phones, are too simple to see a difference in higher resolution. Reported metrics might differ due to different pre-processing techniques. Networks here are the one used inMichalkiewicz et al. (2019) paper and the one used in Section 4.1. We first compare the training time for both methods for various resolutions. Figure

Table content: <table><tr><td>category</td><td/><td>DLS</td><td/><td colspan="2">eigenSDF(ft)</td></tr><tr><td/><td colspan="2">IoU↑ Chamf↓ NC↑</td><td colspan="3">F-score↑ IoU↑ Chamf↓ NC↑</td><td>F-score↑</td></tr><tr><td>cars</td><td>0.784 0.055</td><td colspan="2">0.804 0.148</td><td>0.821 0.040</td><td>0.909 0.432</td></tr><tr><td>chairs</td><td>0.434 0.360</td><td colspan="2">0.743 0.066</td><td>0.553 0.125</td><td>0.820 0.168</td></tr><tr><td>sofas</td><td>0.581 0.132</td><td colspan="2">0.779 0.089</td><td>0.647 0.082</td><td>0.867 0.248</td></tr></table>

Peer review (iclr2020):
This paper studies the problem of learning the feature representation for predicting the 3D shape of objects, from a single image or a point cloud. The proposed approach performs PCA on the SDF field. And then the transformed feature map is learned and used as input to task-specific decoders for 3D shape prediction. The authors claims that this approach trains faster and is easier to scale, while showing competitive performance compared to state-of-the-art methods. I am leaning towards Weak Reject. The paper is generally easy to read, but with some details missing. And I found the discussion of the results to be insufficient. I think it can be an above-threshold paper if questions are addressed during rebuttal. Being able to easily scale to higher resolutions is claimed to be one of the main advantages, but I am not convinced that this is useful under this setting. If I understand this correctly, the number of eigenvectors k is fixed, and projecting the SDF field to this space would remove the higher frequency components of the shape. So wouldn*t the number of eigenvectors be the bottleneck in representational precision, not the resolution of the output space? What is the chosen k (number of eigenvectors)? It says k was *chosen to capture 99.5% of the variance within the dataset*, but I could not find how exactly it was chosen and what value of k was used (I apologize if I missed). Also, I think the PCA is category-specific (page 4, section 4.1). Is k dependent on the category or is it the same across all categories? Some of the other methods (if not all) used for comparison are not category-specific, so if this is true, I think the comparison may not be entirely fair and it should be made clearer. I think the writing could be polished as well, some minor typos: Page 2: 3rd and 4th paragraphs: continous Page 2: anlaysis, enlightning Page3: under eigenSDF: reprsentation Page 4: section 4.1: refered, signficant Page 5: “seciton” Page 7: tranform Page 3, Section 3.2: Is N the number of training examples and M the resolution? Figure 2 not referred in the main text.