Year: 2019

Abstract:
Deep learning applied to the reconstruction of 3D shapes has seen growing interest. A popular approach to 3D reconstruction and generation in recent years has been the CNN decoder-encoder model often applied in voxel space. However this often scales very poorly with the resolution limiting the effectiveness of these models. Several sophisticated alternatives for decoding to 3D shapes have been proposed typically relying on alternative deep learning architectures. We show however in this work that standard benchmarks in 3D reconstruction can be tackled with a surprisingly simple approach: a linear decoder obtained by principal component analysis on the signed distance transform (SDF) of the surface. This approach allows easily scaling to larger resolutions. We show in multiple experiments it is competitive with state of the art methods and also allows the decoder to be fine-tuned on the target task using a loss designed for SDF transforms, obtaining further gains.

heading: 1 INTRODUCTION
text: In recent years there has been an increased interest in extending the successes of deep learning to problems requiring analysis and representation of 3D shapes. This includes many long standing tasks such as 3D shape reconstruction (Wu et al., 2016a; Choy et al., 2016) from single or multiple views, as well as shape completion Mescheder et al. (2018). There are a number of applications of these methods in robotics, surgery, and augmented reality.
A popular approach has been the use of CNN decoder-encoder architectures Choy et al. (2016) as popularized in applications like segmentation Chen et al. (2017). Here for example in the single view reconstruction 2D CNN will encode the 2-D image and a 3D CNN decoder model will produce the final representation in voxels. This however is ineffective in larger resolutions and does not make full use of the structure of the object. Similar problems arise in more general attempts to learn latent variable models of shapes (Wu et al., 2016b; Gadelha et al., 2017). Here one may be interested in tasks such as unconditional generation and reconstruction.
More recently authors have considered alternative representations of shapes to a standard 3D discretized set of voxels(Choy et al., 2016; Girdhar et al., 2016b; Wu et al., 2016a; Yan et al., 2016; Zhu et al., 2017), one that can permit more efficient learning and generation. These include point clouds Fan et al. (2017), meshes(Wang et al., 2018; Georgia Gkioxari, 2019), and signed distance transform based representations (Michalkiewicz et al., 2019; Park et al., 2019). To date there is not an agreed upon canonical 3-D shape representation for use with deep learning models. Furthermore authors have considered alternative architectures aimed at dealing with this problem (Richter & Roth, 2018; Tatarchenko et al., 2017). However, in this work, we ask whether a simpler approach can yield strong results. Building on the recent use of the Signed Distance Function (SDF) in shape representation we demonstrate a simple latent shape representation that can be used in downstream tasks and easily decoded. More specifically, in this work, we consider a latent shape representation obtained by applying PCA on the SDF transformed shape. We show this leads to a latent shape representation that can be used directly in downstream tasks like 3D shape reconstruction from a single view and 3D shape completion from a point cloud.
Our work a) reinforces the relevance of SDF as a representation for 3D deep learning b) demonstrates that a simple representation obtained by applying PCA on the SDF transform can lead to an effective latent shape representation. This representation allows for results competitive to state-of-the-art
in standard benchmarks. Our work also suggests more complex benchmarks than those currently considered may be needed to push forward the study of learned 3D shape reconstruction.
The paper is structured as follows. In Sec. 2 we discuss the related work. We outline the basic methods used in the experiments in Sec. 3. We show extensive quantitative and experimental results comparing the eigenSDF approach to existing methods in Sec. 4.

heading: 2 RELATED WORK
text: Point cloud based representation require a tedious step of sampling points from the surface and to generate shape subsequently inferring the continuous shape from a sample of points. Meshes present a challenge in that no clear way to generate valid meshes is available. Proposals have consisted of starting with template shapes and progressively deforming them Wang et al. (2018). This however can be problematic as it never explicitly represents the shape and may suffer issues with local coherence.
Michalkiewicz et al. (2019) and Park et al. (2019) also use the SDF representation. Unlike our work Deep Level Sets still relies on a decoder encoder CNN architecture thus not removing the desired computational constraints associated with the 3D shape modeling. DeepSDF attempts to directly fit a continuous function to each shape which gives the SDF representations. Although it avoids discretization this function can lead to a complex decoder model, for example an 8 layer network is used to fit the SDF. Another recent work Mescheder et al. (2018) similar in spirit to Park et al. (2019) learns a classifier to predict whether a point is inside or outside of the boundary, using this classifier as the shape representation. Unlike our proposal these methods cannot easily learn a latent shape representation to be applied on downstream task, since the shape is represented by the weights of the classifier or regression model. On the other hand our latent representation can easily encode an unseen shape and be easily used as a prediction target for deep learning models.
Our work can also be seen as complementary to the very recent observations in Tatarchenko et al. (2019) which highlights that good 3D single view reconstruction performance can be achieved by using retrieval or clustering methods. We note however the descriptors used in that work are more complex.
PCA has been classically used to represent shapes in a variety of contexts. For example classical methods in computer vision such as the active appearance model Edwards et al. (1998) and the 3D morphable model used in face analysis Blanz et al. (1999) are based on PCA shape representations. However, these typically are applied in a different context requiring transforming the shape to a reference set of points and applying PCA on the coordinates. Leventon et al. (2002) used signed distance functions to embed 2D curves applying PCA to obtain statistical models. To the best of our knowledge it has not been combined with the SDF representing a surface in 3D. We note level set methods and the SDF have only recently been revisited as an effective representation that can be combined with 3D deep learning Michalkiewicz et al. (2019); Park et al. (2019). Moreover it is enlightening that this classic approach to shape representation can be competitive with deep learning methods on standard benchmarks.

heading: 3 METHODS
text: In this section, we review the SDF transform and describe the simple latent representation that we consider.

heading: 3.1 SIGNED DISTANCE FUNCTIONS
text: The Signed Distance Function (SDF) of a given closed surface Γ ⊂ R3 is defined by the signed distance d : R3 × R3 7→ R from any point x ∈ R3 to the surface Γ. Given a discretized grid Ω we can represent each shape as an SDF on Ω by
φi(x) = ± inf y∈Γ d(x, y), (1)
with the convention that φi(x) is positive on the interior and negative on the exterior of Γ.
In Michalkiewicz et al. (2019) a CNN decoder model is used to predict the SDF representation from a latent space as well as to learn autoencoders. We note however that this representation is well structured and objects are often grouped by category, we thus ask if a much simpler linear and non-convolutional decoder model can be effective.
Michalkiewicz et al. (2019) further considers a loss function for the SDF representation that approximately minimizes the chamfer loss.
L (θ) = ∑ j∈D (∑ x∈Ω δ (φ̃ j(x))dj(x)p )1/p + α ∑ j∈D ∑ x∈Ω (‖∇φ̃j(x)‖ − 1)2 (2)
with δ being approximated Dirac delta, φ̃ inferred Signed Distance Function, and d(x) the closest distance between grid point x and the ground truth shape. We will further leverage this representation in the sequel.

heading: 3.2 EIGENSDF
text: We apply the PCA transform to S = {φi}i=..N , where N is the number of training examples. The eigenvectors E have the shape of (k,M3) with M being the grid resolution. We project each SDF to φc = φET . Here φc has a shape of (1, k). In the sequel we will denote this representation as the EigenSDF. Note that applying PCA to the naive voxel representation would be inappropriate as the data is binary and largely zeros. For downstream tasks we predict directly the latent representation φc. We will also consider usingE as an initialization which is finetuned by training on the SDF shape representation directly using Eq. 2. A high level overview of our framework is given in Figure 1.

heading: 4 EXPERIMENTS
text: We evaluate the proposed representations on 3 tasks. These are single view 3D reconstruction, 3D reconstruction from point cloud, and 3D reconstruction with autoencoders. We take many of our
baselines and comparisons from the recently proposed Mescheder et al. (2018). These applications are evaluated on 13 categories from ShapeNet repository (Chang et al., 2015).
Preprocessing In order to work on SDFs we need to have a well defined interior and exterior of an object, we have first preprocessed the meshes to make them watertight using method proposed by Stutz & Geiger (2018). Following common practice we render each ground truth mesh into 24 2D views using equally spaced azimuth angles. For each ground truth mesh we get a corresponding SDF φ of shape 128× 128× 128.
Metrics Following Mescheder et al. (2018) experimental setup, we report 3 metrics. The first one is Intersection over Union (IoU), also known as Jaccard Index, between ground truth shape S and prediction S̃:
IoU = |S ∩ S̃| |S ∪ S̃|
Second metric measures reconstruction error using symmetric chamfer distance:
chamfer(SP , S̃Q) = 1 2|P | ∑ p∈P min q∈Q |p− q|+ 1 2|Q| ∑ q∈Q min p∈P |p− q|
Finally, we measure the angular distance using normal consistency (nc) metric:
nc(SP , S̃Q) = 1 2|P | ∑ p∈P |NSP (p) ·NS̃Q(nS̃Q(p))|+ 1 2|Q| ∑ q∈Q |NS̃Q(q) ·NSP (nSP (q))|
,
where NS(p) denotes normal of point p lying on surface S and nS(q) denotes nearest neighbour of point q lying on surface S.
Chamfer distance and normal consistency metrics are evaluated by sampling 105 points from the surfaces, while the IoU is calculated by sampling 105 points from within the bounding volume.

heading: 4.1 3D RECONSTRUCTION FROM SINGLE 2D VIEW
text: In this set of experiments, we evaluate the eigenSDF approach from Sec 3. We use a starting resolution of 1283. For each category we perform PCA on SDFs. We chose k to capture at least 99.5% of the variance within the dataset. The image encoder is a 2D CNN whose architecture is taken from Richter & Roth (2018). We minimize the `2 loss between φc, the SDF projected into the latent space, and the prediction of the 2D CNN. We train this network for 100 epochs using 1e-3 learning rate, later dropping to 1e-4. Furthermore, we consider finetuning the representation starting with the eigenvectors from PCA and using Eq 2, this baseline is referred to as eigenSDF (finetuned).
In order to demonstrate that a gain is made by PCA versus just architecture we also train a linear autoencoder of the same size (M ) and finetune it with Eq 2. This baseline is referred to as linearSDF and linearSDF(finetune).
Finally we compare to a set of standard benchmarks from the recent literature including voxel based CNN encoder/decoder Choy et al. (2016), point cloud based methods Fan et al. (2017), a mesh based method Wang et al. (2018), and the recently introduced ONet Mescheder et al. (2018).
Complete results are given in Table 1. First we observe that the simply using a same sized linear model linearSDF (finetune) is outperformed by using the eigenSDF. Compared to alternatives our method gives more significant gains in chamfer metric than all competitors and can be further improved with the finetuning. We also observe improvements in the normal consistency metric. For the IoU metric we observe that eigenSDF outperforms all methods except Mescheder et al. (2018). Note that according to Sun et al. (2018) chamfer distance is a far better metric for shape comparison than IoU.

heading: 4.2 3D SHAPE COMPLETION FROM POINT CLOUDS
text: We next consider shape completion from a point cloud. This task has been considered in e.g. Mescheder et al. (2018). Similar to experimental setup of Mescheder et al. (2018) we use 13 categories from ShapeNet repository and we preprocess the meshes to make them watertight. We randomly sample 300 points from ground truth meshes and add a Gaussian noise with 0 mean and 0.05 std. We have used the same metrics as in section 4.1.
We have used PointNet encoder with a bottleneck dimension of 512 Qi et al. (2017) and linear decoder from section 4.1. We use a similar set of baselines as in the previous section and compare to eigenSDF. We observe similar large gains in the chamfer metric and competitive performance in other metrics. Results are shown in Table 2.
We can see that, similar to 3D reconstruction task, our performance in chamfer is much better, normal consistency is similar and iou is worse but still competitive.

heading: 4.3 3D RECONSTRUCTION FROM LATENTS
text: Finally we consider a simple 3D reconstruction task Girdhar et al. (2016a). This can also be viewed as measuring the representational power of the model Mescheder et al. (2018). This evaluates reconstruction quality of eigenSDF versus other methods particularly CNN based autoencoders. The goal is to reconstruct test set shapes. We use an initial resolution of 1283 and reduce it to k = 512 as done in other works Mescheder et al. (2018). We use cars category from ShapeNet repository and evaluate reconstruction on unseen data. For the evaluations in addition to the metrics used in section 4.1 we further analyse the decoders using F-score Knapitsch et al. (2017).

heading: 4.4 COMPARISON WITH DEEP LEVEL SETS
text: In this section, we compare our method to the other recent approach relying on the signed distance transform Michalkiewicz et al. (2019) and learning with the chamfer loss L . This one however uses the CNN decoder model and does not learn a latent shape representation. We have chosen a similar experimental setup of 3 subsets each having 2 000 examples from ShapeNet repository: cars, sofas, chairs. We observed that remaining 2 categories, bottles and phones, are too simple to see a difference in higher resolution. Reported metrics might differ due to different pre-processing techniques. Networks here are the one used in Michalkiewicz et al. (2019) paper and the one used in Section 4.1. We first compare the training time for both methods for various resolutions. Figure 2 shows the result demonstrating that it is not feasible to use the CNN decoder in higher resolutions. It is consistent with the findings of (Richter & Roth, 2018).

heading: 4.5 RECONSTRUCTION AND GENERATION
text: Finally we evaluate the performance on the single view reconstruction qualitatively. In Table 4 we can see that reconstructions (on unseen data) can be effective capturing more complex structures ignored by Michalkiewicz et al. (2019). Multiple authors have also consider generating unconditionally shapes, typically using sophisticated non-linear deep learning models like GANs and VAEs. We compare some of these to sampling a gaussian in the latent space of the eigenSDF, some qualitative results are shown in Figure 3. We observe this simple approach yields some shapes comparable to those o the complex non-linear models.

heading: 5 CONCLUSION
text: We have shown that using a simple linear decoder coupled with the SDF representation yields competitive results. The SDF lends itself to the application of PCA yielding a strong but simple baseline for future work in learned 3D shape analysis. Moreover, our work suggests that more complex baseline datasets may be needed to further evaluate deep learning methods on 3D shape inference.

heading: A APPENDIX
text: A.1 NUMBER OF EIGENVECTORS
In Figure 4, we show the number of eigenvectors which captures 99.5% of variance within the dataset of approximately 6000 examples of ShapeNetCars. Number of eigenvectors required to represent shapes from other categories vary from 512 to 2048, shown in Figures 5-6.
A.2 PCA APPLIED TO OTHER SHAPE REPRESENTATIONS
As mentioned in the Section 1, there are 4 commonly used shape representations: voxels, Signed Distance Functions (SDF), point clouds and meshes. Because of lack of canonical order or representation it is hard to apply PCA on point clouds or meshes in a straightforward way. Compared to
the results of pca reconstruction using Signed Distance Function as shape representation, applying PCA directly on voxels yeilds poor results, as shown in Figures 7-8.

Review summaries:
This paper studies the problem of learning the feature representation for predicting the 3D shape of objects, from a single image or a point cloud. The proposed approach performs PCA on the SDF field. And then the transformed feature map is learned and used as input to task-specific decoders for 3D shape prediction. The authors claims that this approach trains faster and is easier to scale, while showing competitive performance compared to state-of-the-art methods.