SALD: Sign Agnostic Learning with Derivatives

1 INTRODUCTION . Recently , neural networks ( NN ) have been used for representing and reconstructing 3D surfaces . Current NN-based 3D learning approaches differ in two aspects : the choice of surface representation , and the supervision method . Common representations of surfaces include using NN as parametric charts of surfaces ( Groueix et al. , 2018b ; Williams et al. , 2019 ) ; volumetric implicit function representation defined over regular grids ( Wu et al. , 2016 ; Tatarchenko et al. , 2017 ; Jiang et al. , 2020 ) ; and NN used directly as volumetric implicit functions ( Park et al. , 2019 ; Mescheder et al. , 2019 ; Atzmon et al. , 2019 ; Chen & Zhang , 2019 ) , referred henceforth as implicit neural representations . Supervision methods include regression of known or approximated volumetric implicit representations ( Park et al. , 2019 ; Mescheder et al. , 2019 ; Chen & Zhang , 2019 ) , regression directly with raw 3D data ( Atzmon & Lipman , 2020 ; Gropp et al. , 2020 ; Atzmon & Lipman , 2020 ) , and differentiable rendering using 2D data ( i.e. , images ) supervision ( Niemeyer et al. , 2020 ; Liu et al. , 2019 ; Saito et al. , 2019 ; Yariv et al. , 2020 ) . The goal of this paper is to introduce SALD , a method for learning implicit neural representations of surfaces directly from raw 3D data . The benefit in learning directly from raw data , e.g. , nonoriented point clouds or triangle soups ( e.g. , Chang et al . ( 2015 ) ) and raw scans ( e.g. , Bogo et al . ( 2017 ) ) , is avoiding the need for a ground truth signed distance representation of all train surfaces for supervision . This allows working with complex models with inconsistent normals and/or missing parts . In Figure 1 we show reconstructions of zero level sets of SALD learned implicit neural representations of car models from the ShapeNet dataset ( Chang et al. , 2015 ) with variational autoencoder ; notice the high detail level and the interior , which would not have been possible with , e.g. , previous data pre-processing techniques using renderings of visible parts ( Park et al. , 2019 ) . Our approach improves upon the recent Sign Agnostic Learning ( SAL ) method ( Atzmon & Lipman , 2020 ) and shows that incorporating derivatives in a sign agnostic manner provides a significant improvement in surface approximation and detail . SAL is based on the observation that given an unsigned distance function h to some raw 3D data X ⊂ R3 , a sign agnostic regression to h will introduce new local minima that are signed versions of h ; in turn , these signed distance functions can be used as implicit representations of the underlying surface . In this paper we show how the sign agnostic regression loss can be extended to compare both function values h and derivatives ∇h , up to a sign . The main motivation for performing NN regression with derivatives is that it reduces the sample complexity of the problem ( Czarnecki et al. , 2017 ) , leading to better accuracy and generalization . For example , consider a one hidden layer NN of the form f ( x ) = max { ax , bx } +c . Prescribing two function samples at { −1 , 1 } are not sufficient for uniquely determining f , while adding derivative information at these points determines f uniquely . We provide empirical evidence as well as theoretical motivation suggesting that both SAL and SALD possess the favorable minimal surface property ( Zhao et al. , 2001 ) , that is , in areas of missing parts and holes they will prefer zero level sets with minimal area . We justify this property by proving that , in 2D , when restricted to the zero level-set ( a curve in this case ) , the SAL and SALD losses would encourage a straight line solution connecting neighboring data points . We have tested SALD on the dataset of man-made models , ShapeNet ( Chang et al. , 2015 ) , and human raw scan dataset , D-Faust ( Bogo et al. , 2017 ) , and compared to state-of-the-art methods . In all cases we have used the raw input data X as is and considered the unsigned distance function to X , i.e. , hX , in the SALD loss to produce an approximate signed distance function in the form of a neural network . Comparing to state-of-the-art methods we find that SALD achieves superior results on this dataset . On the D-Faust dataset , when comparing to ground truth reconstructions we report state-of-the-art results , striking a balance between approximating details of the scans and avoiding overfitting noise and ghost geometry . Summarizing the contributions of this paper : • Introducing sign agnostic learning with derivatives . • Identifying and providing a theoretical justification for the minimal surface property of sign agnostic learning in 2D . • Training directly on raw data ( end-to-end ) including unoriented or not consistently oriented triangle soups and raw 3D scans . 2 PREVIOUS WORK . Learning 3D shapes with neural networks and 3D supervision has shown great progress recently . We review related works , where we categorize the existing methods based on their choice of 3D surface representation . Parametric representations . The most fundamental surface representation is an atlas , that is a collection of parametric charts f : R2 → R3 with certain coverage and transition properties ( Do Carmo , 2016 ) . Groueix et al . ( 2018b ) adapted this idea using neural network to represent a surface as union of such charts ; Williams et al . ( 2019 ) improved this construction by introducing better transitions between charts ; Sinha et al . ( 2016 ) use geometry images ( Gu et al. , 2002 ) to represent an entire shape using a single chart ; Maron et al . ( 2017 ) use global conformal parameterization for learning surface data ; Ben-Hamu et al . ( 2018 ) use a collection of overlapping global conformal charts for human-shape generative model . Hanocka et al . ( 2020 ) shrink-wraps a template mesh to fits a point cloud . The benefit in parametric representations is in the ease of sampling the learned surface ( i.e. , forward pass ) and work directly with raw data ( e.g. , Chamfer loss ) ; their main struggle is in producing charts that are collectively consistent , of low distortion , and covering the shape . Implicit representations . Another approach for representing surfaces is as zero level sets of a function , called an implicit function . There are two popular methods to model implicit volumetric functions with neural networks : i ) Convolutional neural network predicting scalar values over a predefined fixed volumetric structure ( e.g. , grid or octree ) in space ( Tatarchenko et al. , 2017 ; Wu et al. , 2016 ) ; and ii ) Multilayer Perceptron of the form f : R3 → R defining a continuous volumetric function ( Park et al. , 2019 ; Mescheder et al. , 2019 ; Chen & Zhang , 2019 ) . Currently , neural networks are trained to be implicit function representations with two types of supervision : ( i ) regression of samples taken from a known or pre-computed implicit function representation such as occupancy function ( Mescheder et al. , 2019 ; Chen & Zhang , 2019 ) or a signed distance function ( Park et al. , 2019 ) ; and ( ii ) working with raw 3D supervision , by particle methods relating points on the level sets to the model parameters ( Atzmon et al. , 2019 ) , using sign agnostic losses ( Atzmon & Lipman , 2020 ) , or supervision with PDEs defining signed distance functions ( Gropp et al. , 2020 ) . Primitives . Another type of representation is to learn shapes as composition or unions of a family of primitives . Gradient information have been used to improve and facilitate fitting of invariant polynomial representations ( Tasdizen et al. , 1999 ; Birdal et al. , 2019 ) . Li et al . ( 2019 ) represent a shape using a parametric collection of primitives . Genova et al . ( 2019 ; 2020 ) use a collection of Gaussians and learn consistent shape decompositions . Chen et al . ( 2020 ) suggest a differentiable Binary Space Partitioning tree ( BSP-tree ) for representing shapes . Deprelle et al . ( 2019 ) combine points and charts representations to learn basic shape structures . Deng et al . ( 2020 ) represent a shape as a union of convex sets . Williams et al . ( 2020 ) learn cites of Voronoi cells for implicit shape representation . Template fitting . Lastly , several methods learn 3D shapes of a certain class ( e.g. , humans ) by learning the deformation from a template model . Classical methods use matching techniques and geometric loss minimization for non-rigid template matching ( Allen et al. , 2002 ; 2003 ; Anguelov et al. , 2005 ) . Groueix et al . ( 2018a ) use an auto-encoder architecture and Chamfer distance to match target shapes . Litany et al . ( 2018 ) use graph convolutional autoencoder to learn deformable template for shape completion . 3 METHOD . Given raw geometric input data X ⊂ R3 , e.g. , a triangle soup , our goal is to find a multilayer perceptron ( MLP ) f : R3 × Rm → R whose zero level-set , S = { x ∈ R3 | f ( x ; θ ) = 0 } ( 1 ) is a manifold surface that approximates X . Sign agnostic learning . Similarly to SAL , our approach is to consider the ( readily available ) unsigned distance function to the raw input geometry , h ( y ) = min x∈X ‖y − x‖ ( 2 ) and perform sign agnostic regression to get a signed version f of h. SAL uses a loss of the form loss ( θ ) = Ex∼D τ ( f ( x ; θ ) , h ( x ) ) , ( 3 ) where D is some probability distribution , e.g. , a sum of gaussians with centers uniformly sampled over the input geometry X , and τ is an unsigned similarity . That is , τ ( a , b ) is measuring the difference between scalars a , b ∈ R up-to a sign . For example τ ( a , b ) = ∣∣|a| − b∣∣ ( 4 ) is an example that is used in Atzmon & Lipman ( 2020 ) . The key property of the sign agnostic loss in equation 3 is that , with proper weights initialization θ0 , it finds a new signed local minimum f which in absolute value is similar to h. In turn , the zero level set S of f is a valid manifold describing the data X . Sign agnostic learning with derivatives . Our goal is to generalize the SAL loss ( equation 3 ) to include derivative data of h and show that optimizing this loss provides implicit neural representations , S , that enjoy better approximation properties with respect to the underlying geometry X . Generalizing equation 3 requires designing an unsigned similarity measure τ for vector valued functions . The key observation is that equation 4 can be written as τ ( a , b ) = min { |a− b| , |a+ b| } , a , b ∈ R , and can be generalized to vectors a , b ∈ Rd by τ ( a , b ) = min { ‖a− b‖ , ‖a+ b‖ } . ( 5 ) We define the SALD loss : loss ( θ ) = Ex∼D τ ( f ( x ; θ ) , h ( x ) ) + λEx∼D′ τ ( ∇xf ( x ; θ ) , ∇xh ( x ) ) ( 6 ) where λ > 0 is a parameter , D′ is a probability distribution , e.g. , it could be identical to D , or uniform over the input geometry X , and ∇xf ( x ; θ ) , ∇xh ( x ) are the gradients f , h ( resp . ) with respect to their input x . In Figure 2 we show the unsigned distance h to an L-shaped curve ( left ) , and the level sets of the MLPs optimized with the SALD loss ( middle ) and the SAL loss ( right ) ; note that SALD loss reconstructed the sharp features ( i.e. , corners ) of the shape and the level sets of h , while SAL loss smoothed them out ; the implementation details of this experiment can be found in Appendix A.4 . Minimal surface property . We show that the SAL and SALD losses possess a minimal surface property ( Zhao et al. , 2001 ) , that is , they strive to minimize surface area of missing parts . For example , Figure 4 shows the unsigned distance to a curve with a missing segment ( left ) , and the zero level sets of MLPs optimized with SALD loss ( middle ) , and SAL loss ( right ) . Note that in both cases the zero level set in the missing part area is the minimal length curve ( i.e. , a line ) connecting the end points of that missing part . SALD also preserves sharp features of the rest of the shape . Figure A1 in the supplementary shows additional 2D experiments comparing to the Implicit Geometric Regularization ( IGR ) method ( Gropp et al. , 2020 ) that learns implicit representations by regularizing the gradient norm and do not posses the minimal surface property . We will provide a theoretical justification to this property in the 2D case . We consider a geometry defined by two points in the plane , X = { x1 , x2 } ⊂ R2 and possible solutions where the zero level set curve S is connecting x1 and x2 . We prove that among a class of curves U connecting x1 and x2 , the straight line minimizes the losses in equation 3 and equation 6 restricted to U , when assuming uniform distributions D , D′ . We assume ( without losing generality ) that x1 = ( 0 , 0 ) T , x2 = ( ` , 0 ) T and consider curves u ∈ U defined by u ( s ) = ( s , t ( s ) ) T , where s ∈ [ 0 , ` ] , and t : R→ R is some differentiable function such that t ( 0 ) = 0 = t ( ` ) , see Figure 3 . For the SALD loss we prove the claim for a slightly simplified agnostic loss motivated by the following lemma proved in Appendix A.1 : Lemma 1 . For any pair of unit vectors a , b : min { ‖a− b‖ , ‖a+ b‖ } ≥ |sin∠ ( a , b ) | . We consider τ ( a , b ) = |sin∠ ( a , b ) | for the derivative part of the loss in equation 6 , which is also sign agnostic . Theorem 1 . Let X = { x1 , x2 } ⊂ R2 , and the family of curves U connecting x1 and x2 . Furthermore , let lossSAL ( u ) and lossSALD ( u ) denote the losses in equation 3 and equation 6 ( resp . ) when restricted to u with uniform distributions D , D′ . Then in both cases the straight line , i.e. , the curve u ( s ) = ( s , 0 ) , is the strict global minimizer of these losses . Proof . The unsigned distance function is h ( u ) = { √ s2 + t2 s ∈ [ 0 , ` /2 ] √ ( s− ` ) 2 + t2 s ∈ ( ` /2 , ` ] . From symmetry it is enough to consider only the first half of the curve , i.e. , s ∈ [ 0 , ` /2 ) . Then , the SAL loss , equation 3 , restricted to the curve u ( i.e. , where f vanishes ) takes the form lossSAL ( u ) = ∫ ` /2 0 τ ( f ( u ; θ ) , h ( u ) ) ‖u̇‖ ds = ∫ ` /2 0 √ s2 + t2 √ 1 + ṫ2 ds , where √ 1 + ṫ2 ds is the length element on the curve u , and τ ( f ( s , t ; θ ) , h ( s , t ) ) = |h ( s , t ) | =√ s2 + t2 , since f ( s , t ; θ ) = 0 over the curve u . Plugging t ( s ) ≡ 0 in lossSAL ( u ) we see that the curve u = ( s , 0 ) T , namely the straight line curve from x1 to 0.5 ( x1 + x2 ) is a strict global minimizer of lossSAL ( u ) . Similar argument on s ∈ [ ` /2 , ` ] prove the claim for the SAL case . For the SALD case , we want to calculate τ ( ∇xf ( u ; θ ) , ∇xh ( u ) ) restricted to the curve u ; let a = ∇xf ( u ; θ ) and b = ∇xh ( u ) . First , b = ( s2+ t2 ) −1/2 ( s , t ) T . Second , a is normal to the curve u , therefore it is proportional to u̇⊥ = ( −ṫ , 1 ) T . Next , note that |sin∠ ( a , b ) | = ∣∣∣∣det ( −ṫ s1 t ) ∣∣∣∣√ 1 + ṫ2 √ s2 + t2 = 1√ 1 + ṫ2 ∣∣∣∣ dds ‖ ( s , t ) ‖ ∣∣∣∣ , where the last equality can be checked by differentiating ‖ ( s , t ) ‖ w.r.t . s. Therefore , lossSALD ( u ) − lossSAL ( u ) λ = ∫ ` /2 0 τ ( a , b ) ‖u̇‖ ds = ∫ ` /2 0 ∣∣∣∣ dds ‖ ( s , t ) ‖ ∣∣∣∣ ds ≥ ∥∥∥∥ ( ` 2 , t ( ` 2 ) ) ∥∥∥∥ ≥ ` 2 . This bound is achieved for the curve u = ( s , 0 ) , which is also a minimizer of the SAL loss . The straight line also minimizes this version of the SALD loss since lossSALD ( u ) = ( lossSALD ( u ) − lossSAL ( u ) ) + lossSAL ( u ) .

Summary: This paper presents SALD, a new type of implicit shape representation that, in addition to predicting the signed distance function, aligns the gradients of the distance function with that of the neural distance field. The resulting algorithm, for example, has improved approximation power and better preserves the sharp features than the ancestor SAL (sign agnostic learning). The formulation is such that the architecture can consume raw point clouds.