A Simple and Scalable Shape Representation for 3D Reconstruction

1 INTRODUCTION . In recent years there has been an increased interest in extending the successes of deep learning to problems requiring analysis and representation of 3D shapes . This includes many long standing tasks such as 3D shape reconstruction ( Wu et al. , 2016a ; Choy et al. , 2016 ) from single or multiple views , as well as shape completion Mescheder et al . ( 2018 ) . There are a number of applications of these methods in robotics , surgery , and augmented reality . A popular approach has been the use of CNN decoder-encoder architectures Choy et al . ( 2016 ) as popularized in applications like segmentation Chen et al . ( 2017 ) . Here for example in the single view reconstruction 2D CNN will encode the 2-D image and a 3D CNN decoder model will produce the final representation in voxels . This however is ineffective in larger resolutions and does not make full use of the structure of the object . Similar problems arise in more general attempts to learn latent variable models of shapes ( Wu et al. , 2016b ; Gadelha et al. , 2017 ) . Here one may be interested in tasks such as unconditional generation and reconstruction . More recently authors have considered alternative representations of shapes to a standard 3D discretized set of voxels ( Choy et al. , 2016 ; Girdhar et al. , 2016b ; Wu et al. , 2016a ; Yan et al. , 2016 ; Zhu et al. , 2017 ) , one that can permit more efficient learning and generation . These include point clouds Fan et al . ( 2017 ) , meshes ( Wang et al. , 2018 ; Georgia Gkioxari , 2019 ) , and signed distance transform based representations ( Michalkiewicz et al. , 2019 ; Park et al. , 2019 ) . To date there is not an agreed upon canonical 3-D shape representation for use with deep learning models . Furthermore authors have considered alternative architectures aimed at dealing with this problem ( Richter & Roth , 2018 ; Tatarchenko et al. , 2017 ) . However , in this work , we ask whether a simpler approach can yield strong results . Building on the recent use of the Signed Distance Function ( SDF ) in shape representation we demonstrate a simple latent shape representation that can be used in downstream tasks and easily decoded . More specifically , in this work , we consider a latent shape representation obtained by applying PCA on the SDF transformed shape . We show this leads to a latent shape representation that can be used directly in downstream tasks like 3D shape reconstruction from a single view and 3D shape completion from a point cloud . Our work a ) reinforces the relevance of SDF as a representation for 3D deep learning b ) demonstrates that a simple representation obtained by applying PCA on the SDF transform can lead to an effective latent shape representation . This representation allows for results competitive to state-of-the-art in standard benchmarks . Our work also suggests more complex benchmarks than those currently considered may be needed to push forward the study of learned 3D shape reconstruction . The paper is structured as follows . In Sec . 2 we discuss the related work . We outline the basic methods used in the experiments in Sec . 3 . We show extensive quantitative and experimental results comparing the eigenSDF approach to existing methods in Sec . 4 . 2 RELATED WORK . Point cloud based representation require a tedious step of sampling points from the surface and to generate shape subsequently inferring the continuous shape from a sample of points . Meshes present a challenge in that no clear way to generate valid meshes is available . Proposals have consisted of starting with template shapes and progressively deforming them Wang et al . ( 2018 ) . This however can be problematic as it never explicitly represents the shape and may suffer issues with local coherence . Michalkiewicz et al . ( 2019 ) and Park et al . ( 2019 ) also use the SDF representation . Unlike our work Deep Level Sets still relies on a decoder encoder CNN architecture thus not removing the desired computational constraints associated with the 3D shape modeling . DeepSDF attempts to directly fit a continuous function to each shape which gives the SDF representations . Although it avoids discretization this function can lead to a complex decoder model , for example an 8 layer network is used to fit the SDF . Another recent work Mescheder et al . ( 2018 ) similar in spirit to Park et al . ( 2019 ) learns a classifier to predict whether a point is inside or outside of the boundary , using this classifier as the shape representation . Unlike our proposal these methods can not easily learn a latent shape representation to be applied on downstream task , since the shape is represented by the weights of the classifier or regression model . On the other hand our latent representation can easily encode an unseen shape and be easily used as a prediction target for deep learning models . Our work can also be seen as complementary to the very recent observations in Tatarchenko et al . ( 2019 ) which highlights that good 3D single view reconstruction performance can be achieved by using retrieval or clustering methods . We note however the descriptors used in that work are more complex . PCA has been classically used to represent shapes in a variety of contexts . For example classical methods in computer vision such as the active appearance model Edwards et al . ( 1998 ) and the 3D morphable model used in face analysis Blanz et al . ( 1999 ) are based on PCA shape representations . However , these typically are applied in a different context requiring transforming the shape to a reference set of points and applying PCA on the coordinates . Leventon et al . ( 2002 ) used signed distance functions to embed 2D curves applying PCA to obtain statistical models . To the best of our knowledge it has not been combined with the SDF representing a surface in 3D . We note level set methods and the SDF have only recently been revisited as an effective representation that can be combined with 3D deep learning Michalkiewicz et al . ( 2019 ) ; Park et al . ( 2019 ) . Moreover it is enlightening that this classic approach to shape representation can be competitive with deep learning methods on standard benchmarks . 3 METHODS . In this section , we review the SDF transform and describe the simple latent representation that we consider . 3.1 SIGNED DISTANCE FUNCTIONS . The Signed Distance Function ( SDF ) of a given closed surface Γ ⊂ R3 is defined by the signed distance d : R3 × R3 7→ R from any point x ∈ R3 to the surface Γ . Given a discretized grid Ω we can represent each shape as an SDF on Ω by φi ( x ) = ± inf y∈Γ d ( x , y ) , ( 1 ) with the convention that φi ( x ) is positive on the interior and negative on the exterior of Γ . In Michalkiewicz et al . ( 2019 ) a CNN decoder model is used to predict the SDF representation from a latent space as well as to learn autoencoders . We note however that this representation is well structured and objects are often grouped by category , we thus ask if a much simpler linear and non-convolutional decoder model can be effective . Michalkiewicz et al . ( 2019 ) further considers a loss function for the SDF representation that approximately minimizes the chamfer loss . L ( θ ) = ∑ j∈D ( ∑ x∈Ω δ ( φ̃ j ( x ) ) dj ( x ) p ) 1/p + α ∑ j∈D ∑ x∈Ω ( ‖∇φ̃j ( x ) ‖ − 1 ) 2 ( 2 ) with δ being approximated Dirac delta , φ̃ inferred Signed Distance Function , and d ( x ) the closest distance between grid point x and the ground truth shape . We will further leverage this representation in the sequel . 3.2 EIGENSDF . We apply the PCA transform to S = { φi } i= .. N , where N is the number of training examples . The eigenvectors E have the shape of ( k , M3 ) with M being the grid resolution . We project each SDF to φc = φET . Here φc has a shape of ( 1 , k ) . In the sequel we will denote this representation as the EigenSDF . Note that applying PCA to the naive voxel representation would be inappropriate as the data is binary and largely zeros . For downstream tasks we predict directly the latent representation φc . We will also consider usingE as an initialization which is finetuned by training on the SDF shape representation directly using Eq . 2 . A high level overview of our framework is given in Figure 1 . 4 EXPERIMENTS . We evaluate the proposed representations on 3 tasks . These are single view 3D reconstruction , 3D reconstruction from point cloud , and 3D reconstruction with autoencoders . We take many of our baselines and comparisons from the recently proposed Mescheder et al . ( 2018 ) . These applications are evaluated on 13 categories from ShapeNet repository ( Chang et al. , 2015 ) . Preprocessing In order to work on SDFs we need to have a well defined interior and exterior of an object , we have first preprocessed the meshes to make them watertight using method proposed by Stutz & Geiger ( 2018 ) . Following common practice we render each ground truth mesh into 24 2D views using equally spaced azimuth angles . For each ground truth mesh we get a corresponding SDF φ of shape 128× 128× 128 . Metrics Following Mescheder et al . ( 2018 ) experimental setup , we report 3 metrics . The first one is Intersection over Union ( IoU ) , also known as Jaccard Index , between ground truth shape S and prediction S̃ : IoU = |S ∩ S̃| |S ∪ S̃| Second metric measures reconstruction error using symmetric chamfer distance : chamfer ( SP , S̃Q ) = 1 2|P | ∑ p∈P min q∈Q |p− q|+ 1 2|Q| ∑ q∈Q min p∈P |p− q| Finally , we measure the angular distance using normal consistency ( nc ) metric : nc ( SP , S̃Q ) = 1 2|P | ∑ p∈P |NSP ( p ) ·NS̃Q ( nS̃Q ( p ) ) |+ 1 2|Q| ∑ q∈Q |NS̃Q ( q ) ·NSP ( nSP ( q ) ) | , where NS ( p ) denotes normal of point p lying on surface S and nS ( q ) denotes nearest neighbour of point q lying on surface S. Chamfer distance and normal consistency metrics are evaluated by sampling 105 points from the surfaces , while the IoU is calculated by sampling 105 points from within the bounding volume . 4.1 3D RECONSTRUCTION FROM SINGLE 2D VIEW . In this set of experiments , we evaluate the eigenSDF approach from Sec 3 . We use a starting resolution of 1283 . For each category we perform PCA on SDFs . We chose k to capture at least 99.5 % of the variance within the dataset . The image encoder is a 2D CNN whose architecture is taken from Richter & Roth ( 2018 ) . We minimize the ` 2 loss between φc , the SDF projected into the latent space , and the prediction of the 2D CNN . We train this network for 100 epochs using 1e-3 learning rate , later dropping to 1e-4 . Furthermore , we consider finetuning the representation starting with the eigenvectors from PCA and using Eq 2 , this baseline is referred to as eigenSDF ( finetuned ) . In order to demonstrate that a gain is made by PCA versus just architecture we also train a linear autoencoder of the same size ( M ) and finetune it with Eq 2 . This baseline is referred to as linearSDF and linearSDF ( finetune ) . Finally we compare to a set of standard benchmarks from the recent literature including voxel based CNN encoder/decoder Choy et al . ( 2016 ) , point cloud based methods Fan et al . ( 2017 ) , a mesh based method Wang et al . ( 2018 ) , and the recently introduced ONet Mescheder et al . ( 2018 ) . Complete results are given in Table 1 . First we observe that the simply using a same sized linear model linearSDF ( finetune ) is outperformed by using the eigenSDF . Compared to alternatives our method gives more significant gains in chamfer metric than all competitors and can be further improved with the finetuning . We also observe improvements in the normal consistency metric . For the IoU metric we observe that eigenSDF outperforms all methods except Mescheder et al . ( 2018 ) . Note that according to Sun et al . ( 2018 ) chamfer distance is a far better metric for shape comparison than IoU .

Summary: This paper studies the problem of learning the feature representation for predicting the 3D shape of objects, from a single image or a point cloud. The proposed approach performs PCA on the SDF field. And then the transformed feature map is learned and used as input to task-specific decoders for 3D shape prediction. The authors claims that this approach trains faster and is easier to scale, while showing competitive performance compared to state-of-the-art methods.