SALD: Sign Agnostic Learning with Derivatives
Authors: Matan Atzmon
Yaron Lipman
Venue: ICLR 2021 Poster (2021)
Keywords: implicit neural representations
3D shapes learning
sign agnostic learning

Abstract: Learning 3D geometry directly from raw data, such as point clouds, triangle soups, or unoriented meshes is still a challenging task that feeds many downstream computer vision and graphics applications. 

In this paper, we introduce SALD: a method for learning implicit neural representations of shapes directly from raw data. We generalize sign agnostic learning (SAL) to include derivatives: given an unsigned distance function to the input raw data, we advocate a novel sign agnostic regression loss, incorporating both pointwise values and gradients of the unsigned distance function. Optimizing this loss leads to a signed implicit function solution, the zero level set of which is a high quality and valid manifold approximation to the input 3D data. The motivation behind SALD is that incorporating derivatives in a regression loss leads to a lower sample complexity, and consequently better fitting. In addition, we provide empirical evidence, as well as theoretical motivation in 2D that SAL enjoys a minimal surface property, favoring minimal area solutions. More importantly, we are able to show that this property still holds for SALD, i.e.,  with derivatives included.

We demonstrate the efficacy of SALD for shape space learning on two challenging datasets: ShapeNet that contains inconsistent orientation and non-manifold meshes, and D-Faust that contains raw 3D scans (triangle soups). On both these datasets, we present state-of-the-art results.

Published as a conference paper at ICLR 2021
SALD: SIGN AGNOSTIC LEARNING WITH DERIVATIVES
Matan Atzmon & Yaron Lipman
Weizmann Institute of Science (matan aaztnn, yaron. 1 pman @wei zmann ac. 11
ABSTRACT
Learning 3D geometry directly from raw data, such as point clouds, triangle soups, unoriented meshes is still challenging task that feeds many downstream com- puter vision and graphics applications.
In this paper, we introduce SALD: a method for learning implicit neural repre- sentations of shapes directly from raw data. We generalize sign agnostic learning (SAL) to include derivatives: given an unsigned distance function to the input raw data, we advocate novel sign agnostic regression loss, incorporating both point- wise values and gradients of the unsigned distance function. Optimizing this loss leads to a signed implicit function solution, the zero level set which a high quality and valid manifold approximation to the input 3D data. The motivation behind SALD is that incorporating derivatives in a regression loss leads to a lower sample complexity, and consequently better fitting. In addition, we provide empir- ical evidence, as well as theoretical motivation in 2D that SAL enjoys minimal surface property, favoring minimal area solutions. More importantly, we are able to show that this property still holds for SALD, i.e., with derivatives included.
We demonstrate the efficacy of SALD for shape space learning on two challenging datasets: ShapeNet (Chang et al., 2013) that contains inconsistent orientation and non manifold meshes, and D- D-Faust (Bogo et al. (017)) that contains raw 3D scans (triangle soups). On both these datasets, we present state-of satte-of-the-are t--hhhe-rtt results.
INTRODUCTION
Recently, neural networks (NN) have been used for representing and reconstructing 3D surfaces. Current NN- 3D learning approaches differ in two aspects: the choice of surface representa- tion, and the supervision method. Common representations of surfaces include using NN as para- metric charts of surfaces (Groueix et al. 201 8b Williams et al. 2001); volumetric implicit function representation defined over regular grids (Wu et al. 2016; [Ttarccennko et al., 2017| Jiang et al, 2020); and NN used directly volumetric implicit functions (Park et al. 2019 Mescheder et al.
2019 Atzmon et al., 2019 [Chen & ZZhng 2019), referred henceforth as implicil neural repre- sentations. Supervision methods include regression of known approximated volumetric implicit representations (Park et al. M01ch/e et al. 2019 Chen & Zhang regression directly
with raw 3D data (Atzmon Lipman 2020; Gropp al. 2020 Atzmon & Lipman 2020 and
differentiable rendering using 2D data (I.c., images) supervision (Niemeyer et al. 2020 Liu et al., Saito et al. 2019 Yariv et 2020).
2019
The goal of this paper is to introduce SALD, method for learning implicit neural representations of surfaces directly from raw 3D data. The benefit in learning directly from raw data, e.g., non- oriented point clouds or triangle soups (e.g., CCang et al. (2015)) and raw scans (e.g., Bogo et al.) (017)) ) avoiding the need for ground truth signed distance representation of all train surfaces for supervision. This allows working with complex models with inconsistent normals and/or missing parts. In Figure we show reconstructions of zero level sets of SALD learned implicit neural representations of car models from the ShapeNet dataset (Chang al. 2015) with variational auto- encoder; notice the high detail level and the interior, which would not have been possible with, e.g., previous data pre-proceee rre-processing techniques using renderings of visible parts (Park et al. 2019).
Our approach improves upon the recent Sign Agnostic Learning (SAL) method (Atzmon & Lipman 2020) and shows that incorporating derivatives in sign agnostic manner provides a significant

---

Published as a conference paper at ICLR 2021
Learning the shape space of ShapeNet (Chang ct al. 2015) cars directly from raw data using SALD. Note the interior details; top row depicts SALD reconstructions of train data, and bottom row SALD reconstructions of test data.
Figure I:
improvement in surface approximation and detail. SAL is based on the observation that given unsigned distance function to some raw 3D data C R2, a sign agnostic regression to will introduce new local minima that signed versions of h; in turn, these signed distance functions can be used as implicit representations the underlying surface. In this paper we show how the sign agnostic regression loss can be extended to compare both function values h and derivatives Vh, up to sign.
The main motivation for performing NN regression with derivatives is that it reduces the sample complexity of the problem (Czarnecki et al. 2017). leading to better accuracy and generalization. For example, consider one hidden layer NN the form f() = max {az, br) +c. Prescribing two function samples at {-1, 1} are not sufficient for uniquely determining f. while adding derivative information these points determines uniquely.
We provide empirical evidence well theoretical motivation suggesting that both SAL and SALD possess the favorable minimal surface property (Zhao et al. al.] (0001). that is, areas of missing parts and holes they will prefer zero level sets with minimal area. We justify this property by proving that, 2D, when restricted to the zero level-set curve in this case), the SAL and SALD losses would encourage straight line solution connecting neighboring data points.
We have tested SALD on the dataset man made models, ShapeNet (Chang et al., 2015). and human raw scan dataset, D-Faust (Bogo et al. 2017), and compared to state-of-the-arr methods. In all cases we have used the raw input data X' as is and considered the unsigned distance function to X, i.e., hx, in the SALD loss to produce neural network. Comparing to state-of-the offtfhh-aart methods we find that SALD achieves superior results on this dataset. On the D-Faust dataset, when comparing ground truth reconstructions we report state-of-the-ar results, striking a balance between approximating details of the scans and avoiding overfitting noise and ghost geometry.
approximate signed distance function in the form of
Summarizing the contributions this paper:
Introducing sign agnostic learning with derivatives.
Identifying and providing a theoretical justification for the minimal surface property of sign agnostic learning in 2D.
Training directly on raw data (end-to-end) including unoriented or not consistently oriented triangle soups and 3D scans.
PREVIOUS WORK
Learning 3D shapes with neural networks and 3D supervision has shown great progress recently. We review related works, where we categorize the existing methods based on their choice of 3D surface representation.
2

---

Published as a conference paper at ICLR 2021
Parametric representations. The most fundamental surface representation is atlas, that is col- lection of parametric charts f IR² - IR with certain coverage and transition properties (Do Carmo 2016). Groueix et al. (2018b) adapted this idea using neural network to represent a surface as union of such charts: Williams et al. (2019) improved this construction by introducing better transitions between charts; [Sinha et al. (2016) use geometry images (Gu al., 2002) represent entire shape using a single chart; Maron et al. (2017) use global conformal parameterization for learn- ing surface data; Ben-Hamu al. (2018) use a collection of overlapping global conformal charts for human-shape generative model. Hanocka et al. (2020) shrink-wraps template mesh to fits point cloud. The benefit parametric representations in the ease of sampling the learned surface (i.e., forward pass) and work directly with raw data (e.g., Chamfer loss); their main struggle is in producing charts that are collectively consistent, of low distortion, and covering the shape.
Implicit representations. Another approach for representing surfaces is as zero level sets of function, called an implicit function. There are two popular methods to model implicit volumetric functions with neural networks: i) Convolutional neural network predicting scalar values over predefined fixed volumetric structure (e.g., grid or octree) in space (Tatarchenko et al., 2017 Wu et al. 2016); and ii) Multilayer Perceptron of the form f R -
R defining continuous volumetric function (Park et al. 2019)] | Mescheder et al. (019 [Chen & Zhang 2019). Currently, neural networks types of supervision: (i) regression of samples taken from a known or pre-computed implicit function representation such as occupancy function (Mescheder al., 2019 Chen & Zhang, 2019) or a signed distance function (Park al., 2019]: and (ii) working with raw 3D supervision, by particle methods relating points on the level sets the model parameters (Atzmon al.) 2019), using sign agnostic losses (Atzmon Lipman 2020), or supervision with PDEs defining signed distance functions (Gropp et al.) 2020).
trained be implicit function representations with
Primitives. Another type of representation is learn shapes as composition or unions of family of primitives. Gradient information have been used to improve and facilitate fitting of invariant polynomial representations (Tasdizen et al.) 1999, Birdal et al., 2019). Li al. (2019) represent a shape using parametric collection of primitives. Genova et al. (2019) 2020) use a collection of Gaussians and learn consistent shape decompositions. [Chen et al. (2020) suggest a differentiable Binary Space Partitioning tree (BSP-tree) for representing shapes. Deprelle al. (2019) combine points and charts representations to learn basic shape structures. Deng et al. (2020) represent a shape as union of convex sets. Williams et al. (2020) learn cites of Voronoi cells for implicit shape representation.
Template fitting. Lastly, several methods learn 3D shapes of certain class (e.g., humans) by learning the deformation from a template model. Classical methods use matching techniques and geometric loss minimization for non-rigid template matching (Allen et al. [002 2003) AAnuelovv
itt al. 2005). Groueix et al. (2018a) 8a) use an auto-encoder architecture and Chamfer distance to match
target shapes. Litany et al. (2018) use graph convolutional autoencoder to learn deformable template for shape completion.
3 METHOD
Given raw geometric input data X C R33 e.g., a triangle soup, our goal is to find a multilayer perceptron (MLP) f R x Rm
+ R whose zero level-set,
s f E 3 f(x; 0) = =0}
is a manifold surface that approximates X.
Sign agnostic learning. Similarly to SAL, our approach is to consider the (readily available) un- signed distance function to the raw input geometry,
h(y) = min I|y a:|
#EX
and perform sign agnostic regression to get a signed version f of h. SAL uses a loss of the form loss(0) ()) = EnLD 7 (f(x; (z)), where is some probability distribution, c.g., a sum of gaussians with centers uniformly sampled over the input geometry X, and 7 is an unsigned similarity. That is, T(a, b) is measuring the differ- ence between scalars a, b E R up-to a sign. For example
T(a b)
|||
3

---

Published as a conference paper at ICLR 2021
unsigned distance
SALD
SAL
Sign agnostic learning of an unsigned distance function to an shape (left). Red colors depict positive values, and blue-green green colors depict negative values. In the middle, the result of optimizing the SALD loss (equation 6); on the right, the result of SAL loss (equation Note that SALD better preserves sharp features of the shape and the isolevels.
Figure 2:
an example that is used in Atzmon & Lipman (020). The key property of the sign agnostic loss in equation 3iis that, with proper weights initialization do. it finds a new signed local minimum f which absolute value is similar to In turn, the zero level S of is a valid manifold describing the data X'
Sign agnostic learning with derivatives. Our goal is generalize the SAL loss (equation 3 clude derivative data of h and show that optimizing this loss provides implicit neural representations, S, that enjoy better approximation properties with respect to the underlying geometry X.
Generalizing equation equationßrequirer designing an unsigned similarity measure for vector valued func-
tions. The key observation is that equation 4 can be written as 7(a, b) min {la bl + b a, b E R, and can be generalized to vectors a, b Rd by
r(a,b) b) = min {Ila bll, l|a + bl]}
(5)
We define the SALD loss:
loss(0) EE-D T (zx; 0), h(x)) AE 7 (V=, 0), Vah(z))
where À > 0 is parameter, D' is probability distribution, e.g., it could be identical to D, or uniform over the input geometry X, and fx 0), Vah(x) are the gradients f, h (resp.) with respect their input æ.
In Figure 2 we show the unsigned distance to an L-shaped curve (left), and the level sets of the MLPs optimized with the SALD loss (middle) and the SAL loss (right); note that SALD loss reconstructed the sharp features (i.e., corners) of the shape and the level sets of ht while SAL loss smoothed them out; the implementation details of this experiment can be found in Appendix A.4
Minimal surface property. We show that the SAL and SALD losses possess a minimal sur- face property (Zhao al. 2001), that is, they strive to minimize surface area of missing parts. For example, Figure [4] shows the unsigned distance to a curve with a missing segment (left), and the zero level
of MLPs optimized with SALD loss (middle), and SAL loss (right). Note that in both cases the zero level set in the missing part area the minimal length curve (i.e., line) connecting the end points of that missing part. SALD also preserves sharp features of the rest of the shape. Figure AT the supplementary shows additional 2D experiments comparing the Implicit Geometrie Regulariza- tion (IGR) method (Gropp et al.) 2020) that learns implicit repre- sentations by regularizing the gradient minimal surface property.
E2
and do not posses the
Figure 3: Minimal surface
We will provide theoretical justification to this property in the 2D case. We consider a geometry defined by two points in the plane, X' [2x, 22 } C R² and possible
property in 2D.
solutions where the zero level curve S is connecting # and 22. We prove that among class of

---

Published as a conference paper at ICLR 2021
unsigned distance
SALD
SAL
Figure 4: Minimal surface property: using SALD (middle) and SAL (right) with the input unsigned distance function of curve with missing part (left) leads to solution (black line, middle and right) with approximately minimal length the missing part area. Note that the SALD solution also preserves sharp features of the original shape, better than SAL.
curves U connecting and 22, the straight line minimizes the losses in equation 3 and equation restricted U, when assuming uniform distributions D, D' We assume (without losing generality) (e, O)" and consider curves E 24 defined by u(s) (s. t(s)s( where
that Φ1 (0, 0)" 22
s E [0, E]. and t: R
some differentiable function such that t(0) = = t(E), see Figure 3
For the SALD loss we prove the claim for slightly simplified agnostic loss motivated by the fol- lowing lemma proved Appendix A.I
Lemma For any pair of unit vectors a, b. min { - bll ||a + bll} 2 sin L(a,
We consider r(a, b) = |sin ((a, b) for the derivative part of the loss in equation which is also sign agnostic.
Theorem 1. Let == x2 C R.?. and the family of curves U connecting æ, and x2 Fur- thermore, let losSsa(u1) and losssuo ()) denote the losses in equation 3 and equation 6 (resp.) when restricted to u with uniform distributions D, D'. Then in both cases the straight line, Le., the curve us)) = (s, 0), is the strict global minimizer of these losses.
Proof. The unsigned distance function is
+
E [0,6/2]
huw)
(f/2,
From symmetry it is enough to consider only the first half of the curve, i.e., E [0, f/2). Then, the SAL loss, equation 3 restricted to the curve u (i.e., where f vanishes) takes the form
10" 2/2
T(J(uu (()))) ||l ds 1
2/2
lo888ar. (w)
2 12
ds,
where VI + t ds is the length element on the curve and T(f(s, t: f), h(s,t)) (h(s, t)
V + 12, since [1 0) 0 over the curve Plugging ((s) 0 in lossea (1) we see that the curve u. (s. O). namely the straight line curve from æ1 to 0.5(m) + 222) is a strict global minimizer of Losss (u). Similar argument
[ prove the claim for the SAL case.
For the SALD case, we want to calculate 7 (uV 0), vah(w) restricted to the curve u; let a= V=)uuu 0) and (u)) First, b = (s2 - 1)" Second, a is normal to the curve u. therefore is proportional to át (-t, 1) Next, note that
- det
d (1 t)
Isin <(a,b)
12 +
V1+ 13 ds
5

---

Published as a conference paper at ICLR 2021
where the last equality can be checked by differentiating [|(s, w.r.t. Therefore,
losSua((u) - 1osssar(1))
2/2 fe²= T(a, b) | b| ds 1. 2/2 d
ds I1 1[s.021 do ( 2
t 2
This bound is achieved for the curve SAL loss. 'he straight line also minimizes this version of the SALD loss since loss (u) (LoSSSALD (u) IosssA (u)) + los8su( (u).
(s,0), which is also minimizer of the
4 EXPERIMENTS
We tested SALD on the task of shape space learning from raw 3D data. We experimented with two different datasets: i) ShapeNet dataset (Chang al. (015), containing synthetic 3D Meshes; and ii) D-Faust dataset (Bogo et al. 2017) containing raw 3D scans. Furthermore, we empirically test our sample complexity hypothesis (i.e., that incorporating derivatives improve sample complexity) by sets.
inspecting surface reconstruction accuracy for SAL and SALD when trained with fixed size sample
Shape space learning architecture. Our method can be easily incorporated into existing shape space learning architectures: i) Auto-Decoder (AD) suggested in Park et al. (2019); and the ii) Modified Variational Auto-Encoder (VAE) used in|Atzmon & Lipman (2020). For VAË, the encoder taken to be PointNet (Qi et al.) 2017). For both options, the decoder is the implicit representation in equation [ where f(x;; 0) is taken to be an 8-layer MIPP with 512 hidden units in each layer and Softplus activation. In addition, to enable sign agnostic learning we initialize the decoder weights, 0, using the geometric initialization from Atzmon & Lipman (2020). See Appendix AA.2.4 for more details regarding the architecture. The point samples æ; x for the empirical computation of the expectations in equation 6 are drawn according to distributions D, D' explained in Appendix 1.2.
The baseline methods selected for comparison cover both existing supervision method- ologies: DeepSDF (Park al., (2019) is chosen representative out of the methods that require pre-computed implicit representation for training. For methods that train directly on raw 3D data, we compare versus SAL (Atzmon & Lipman 2020 and IGR (Gropp et al. 2020). See Appendix
Baselines.
A.6 for a detailed description of the quantitative metrics used for evaluation.
Category
Sofas
Chairs
Tables
Planes
Lamps
Mean
Median
Mean
Median
Mean
Median
Mean
Median
Mean
Median
DeepSDF
0.329
0.230
0.341
0.133
0,839
0.149
0.177
0.076
0.909
0.344
SAL
0.704
0.523
0.494
0.259
0.543
0.231
0.429
0.146
4.913
1,515
SALD(VAE)
0.391
0.244
0.415
0.255
0.679
0.279
0.197
0.062
1.808
1.172
SALD(AD)
0.207
0.147
0.281
0.157
0.408
0.25
0.098
0.032
0.506
0.327
Table 1: ShapeNet quantitative results. We log the mean and median of the Chamfer distances (dc)
between the reconstructed 3D surfaces and the ground truth meshes. Numbers are reported x1033.
4.1 SHAPENET
In this experiment we tested the ability of SALD to learn a shape space by train- ing on a challenging 3D data such as non- manifold/non orientable meshes. We tested SALD with both AD and VAE architectures. In both settings, we set X = 0.1 for the SALD loss. We follow the evaluation proto- col as in DeepSDF (Park et al.) 2019): using the same train/test splits, we train and evaluate our method on 5 different categorics. Note that comparison versus IGR is omitted as IGR requires consistently oriented normals for shape space learning, which is not available for ShapeNet, where many models have non-consistent triangles' orientation.
Figure 5: AD versus VAE.
Results. Table M and Figure 6 show quantitative and qualitative results (resp.) for the held-out test set, comparing SAL, DeepSDF and SALD. As can be read from the table and inspected in the

---

Published as a conference paper at ICLR 2021
ShapeNet qualitative test results. Each quadruple shows (columns from left to right): ground truth model, SAL-reconstruction, DeepSDF reconstruction, SALD reconstruction.
Figure 6:
figure, our method, when used with the same auto-decoder as in DeepSDF, compares favorably to DeepSDF's reconstruction performance on this data.
Qualitatively the surfaces produced by SALD are smoother, mostly with more accurate sharp fea- tures, than SAL and DeepSDF generated surfaces. the Cars class with VAE. Figure | shows a comparison between SALD shape space learning with test car model (left). Note that the AD (middle) seems to produce
[ typical train and test results from
VAE and AD in reconstruction more details of the test model than the VAE (right), e.g., steering wheel and headlights. Figure show SALD (AD) generated shapes via latent space interpolation between two test models.
Figure 7: ShapeNet latent interpolation. In each group, the leftmost and rightmost columns are test examples reconstructions; latent space generated shapes are coloured in yellow.
4.2 D-FAUST
The D-Faust dataset (Bogo et al. 2017) contains raw scans (triangle soups) of 10 humans in multiple poses. There are approximately 4lk scans in the dataset. Due the low variety between adjacent scans, sample each pose scans ratio of 5. The leftmost column in Figure 8 shows examples of raw scans used for training. For evaluation we use the registrations provided with the data set. Note that the registrations where not used for training. We tested SALD using the VAE architecture, with À 1.0 set for the SALD loss. We followed the evaluation protocol as in Atzmon & Lipman (2020), using the train/test split. Note that Atzmon Lipman (2020) already conducted a comprehensive comparison of SAL versus DeepSDF and AtlasNet (Groueix al. 2018b). establishing SAL as state-of-the-an method for this dataset. Thus, we focus on comparison versus SAL and IGR.
Results. Table and Figure show quantitative and qualitative results (resp.); although SALD does not produces the best test quantitative results, it is roughly comparable in every measure to the best among the two baselines. That is. produces details comparable IGR while maintaining the minimal surface property as SAL and not adding undesired surface sheets as IGR; see the figure for visual illustrations of these properties: the high level of details of SALD and IGR compared to SAL. and the extraneous parts added by IGR, avoided by SALD. These phenomena can also be seen quantitatively, c.g., the reconstructioontto-regli rergitttttttt loss of IGR. Figure 9 show SALD generated shapes via latent space interpolation between test scans. Notice the ability of SALD generate novel mixed faces and body parts.

---

Published as a conference paper at ICLR 2021
Figure 8: D-Faust qualitative results test examples. Each quadruple shows (columns from left to right): raw scans (magenta depict back back- faces), IGR, SALD, and SAL.
Figure 9: D-Faust latent interpolation. In each group, the leftmost and rightmost columns are test scans reconstructions; latent space generated shapes are coloured in yellow.
4.3 SAMPLE COMPLEXITY
In this experiment test the sam- ple complexity hypothesis: namely, whether regressing with derivatives improves shape reconstruction accu- racy, under fixed budger of point samples. This experiment considers different shapes chosen randomly from the chair, sofa and table test sets of the ShapeNet dataset. For each shape, we prepared fixed sample set of points {a; ( where E {1K,5K, 10K, 20K). to- gether with the unsigned distance value and derivative (h(x;), (=;))) The point samples are drawn according to distribution 'D explained Appendix A.2.I We separately trained the SAL and SALD losses on the same sample data, using the same hyper-parameters, in two different scenarios: i) Individual shape reconstruction: optimizing the weights of a randomly initialized 8-layer MLP f(x; 0); and ii) latent shape reconstruction: given trained auto-decoder network fæx 2; 8) (as 4.1). we optimize solely the latent code 2, keeping the weights fixed. Lastly, computed the Chamfer distance between the learned shape S and the input geometry X. For the individual shape reconstruction, the inset figure shows the Chamfer distance, dc(S. ()), as a function of the sample size Figure T0 shows for each sample size, the learned sofa and table. Note that SALD demonstrates better approximation to the input geometry in comparison to SAL,
SALL
SAL
SAL
(chair)
(sofa)
(table)
det (reg. recon.)
de' (reg.. recon.)
d5 (recon., reg.)
dę (recon., reg.)
d5" (scan, recon.)
dę (scan, recon.)
Mean
Median
Mean
Median
Mean
Median
Mean
Median
Mean
Median
Mean
Median
SAL
0.418
0.328
13.21
12.459
0.344
0.256
11.354
10.522
0.429
0.246
10.0%
9.096
IGR
0.276
0.187
10.328
9,822
3,806
3.627
17.124
17.902
0.241
0.11
5,829
5.295
SALD
0.428
0.346
11.67
11.07
0.362
11,035
10.371
0.397
0.279
7.884
7.227
Table D-Faust quantitative results. We log mean and median of the one-sided Chamfer and normal distances between registration meshes (reg), reconstructions (recon) and raw input scans (scan). The de numbers are reported #10".
8

---

Published as a conference paper at ICLR 2021
1 K
5K
10K
20K
1K
5K
10K
20K
Figure 10: Sample complexity experiment: SALD (bottom row) shows better shape approximation than SAL (top row), especially for small sample sets; numbers indicate sample sizes.
in particular as the sample size gets smaller, and thus supporting the sample complexity hypothesis. When optimizing the latent code of fully trained auto-decoder, the sample size has a little to no effect on the approximation quality of test shape reconstruction. This can be explained by the fact that the auto-decoder is trained on the maximal sample size, and therefore provides strong prior for the latent reconstruction. See the supplementary A.5 for the results.
4.4 LIMITATIONS
Figure II shows typical failure cases of our method from the ShapeNet experiment de- scribed above. We mainly suffer from two types of failures: First, since inside and out- side information is not known (and often not even well defined in ShapeNet models) SALD can add surface sheets closing what should be open areas (e.g., the bottom side of the lamp, or holes in the chair). Second, thin structures can be missed (e.g., the electric cord of the lamp on the left). useful strategy sample thin struc- tures is to make sure the sample frequency is inversely proportional to the distance to the medial axis Amenta al. [1998), where an approximation can be made using curvature estimation. Furthermore, it is important to note that implicit representations of the type presented in equation [ cannot model surfaces with boundaries and therefore cannot represent flat dimensionless surfaces with boundaries. A potential solution could be incorporating additional implicits to handle boundaries.
Figure 11: Failure cases.
5 CONCLUSIONS
We introduced SALD, method for learning implicit neural representations from raw data. The method is based on generalization the sign agnostic learning idea to include derivative data. We demonstrated that the addition of a sign agnostic derivative term to the loss improves the approxima- tion power of the resulting signed implicit neural network. In particular, showing improvement in the level of details and sharp features of the reconstructions. Furthermore, we identify the favorable minimal surface property of the SAL and SALD losses and provide a theoretical justification 2D. Generalizing this theoretical analysis to 3D is marked interesting future work.
We see two more possible venues for future work: First, it is clear that there is room for further improvement in approximation properties of implicit neural representations. Although the results in D-Faust are already close to the input quality, in ShapeNet we still see a gap between input models and their implicit neural representations; this challenge already exists in overfitting large collection of diverse shapes in the training stage. Improvement can come from adding expressive power to the neural networks, or further improving the training losses; adding derivatives as done in this paper is one step in that direction but does not solves the problem completely. Combining sign agnostic learning with the recent positional encoding method (Tancik et al. 2020) could also be an interesting future research venue. Another interesting project is to combine the sign-agnostic losses with gradient regularization such as the one employed IGR (Gropp et al. 2020). Second, it is interesting to think of applications settings in which SALD improve the current state-of- the-art. Generative 3D modeling, learning geometry with 2D supervision, or other types of partially observed scans such depth images all potentially fruitful options.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
The research was supported by the European Research Council (ERC Consolidator Grant, "Lift- Match" 771136). the Israel Science Foundation (Grant No. 1830/17) and by research grant from the Carolito Stiftung (WAIC).
REFERENCES
Brett Allen, Brian Curless, and Zoran Popović. Articulated body deformation from range scan data. ACM Transactions on Graphics (TOG), (3)):612-619) 2002.
Brett Allen, Brian Curless, and Zoran Popović. The space of human body shapes: reconstruction and parameterization from range scans. ACM transactions on graphics (TOG), 22(3):587-594, 2003.
Nina Amenta, Marshall Bern, and Manolis Kamvysselis. A new voronoi-based surface reconstruc- tion algorithm. In Proceedings of the 25th annual conference on Computer graphics and interac- five techniques, pp. 415 421, 1998.
Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. ACM SIGGRAPHI 2005 Papers, pp. 408 416. 2005.
Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, Haggai Maron, and Yaron Lipman. Controlling neural level sets. In Advances in Neural Information Processing Systems, pp. 2032-2041, 2019.
Atilim Günes Baydin, Barak Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: survey. The Journal of Machine Learning Re- search, (1):55595--667,,211
Heli Ben-Hamu, Haggai Maron, Itay Kezurer, Gal Avineri, and Yaron Lipman. Multi-chart genera- tive surface modeling. ACM Transactions on Graphics (TOG), 37(6):1-15, 2018.
Tolga Birdal, Benjamin Busam, Nassir Navab, Slobodan Ilie, and Peter Sturm. Generic primitive detection in point clouds using novel minimal quadric fits. IEEE transactions on pattern analysis and machine intelligence, 42(6):1333-133 2019.
Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael Black. Dynamic faust: Registering human bodies in motion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6233-6242, 2017.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rie 3d model repository. arXiv preprint arXiv: 2015.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev training for neural networks. In Advances in Neural Information Processing Systems, pp. 4278 4287, 2017.
Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cyxnet: Learnable convex decomposition. June 2020.
10

---

Published as a conference paper at ICLR 2021
Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir Kim, Bryan Russell, and Mathieu Aubry. Learning elementary structures for shape generation and matching. In Advances in Neural Information Processing Systems, pp. 1433-7443, 2019.
Manfredo P Do Carmo. Differential Geometry of Curves and Surfaces: Revised and Updated Second Edition. Courier Dover Publications, 2016.
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Saraa, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7154-71171 1564,164, 2019.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3857-4866, 2020.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometrie regu- larization for learning shapes. Proceedings of Machine Learning and Systems 2020, 2020.
Thibault Groueix, Matthew Fisher. Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. 3d- coded: 3d correspondences by deep deformation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 230 246, 2018a.
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier-måché approach to learning 3d surface generation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 216-224, 2018b.
Xianfeng Gu, Steven J Gortler, and Hugues Hoppe. Geometry images. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques, pp. 355-361, 2002.
Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Point2mesh: A self-prior for de- formable meshes. arXiv preprint 20Xiv:2005.11085 2020.
Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. Sdfdiff:: Differentiable rendering of signed distance fields for 3d shape optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1251-1261, 2020.
Diederík Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6941, 2014.
Lingxiao Li, Minhyuk Sung, Anastasia Dubrovina, Li Yi, and Leonidas Guibas. Supervised fitting of geometric primitives to 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2652-2660, 2019.
Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia. Deformable shape completion with graph convolutional autoencoders. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1886-1895, 2018
Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3d supervision. In Advances in Neural Information Processing Systems, pp.
2019.
William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, volume 21, pp. 163-169. ACM, 1987.
Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, Vladimir G Kim, and Yaron Lipman. Convolutional neural networks on surfaces via seamless toric covers.
ACM Trans. Graph., 36(4):71-1, 2017.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference Computer Vision and Pattenn Recognition, pp. 4460-4470, 2019.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 3504-3515, 2020.

---

Published as a conference paper at ICLR 2021
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Charles Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 652-660, 2017.
Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceed- ings of the IEEE International Conference on Computer Vision, pp. 2304-2314, 2019.
Ayan Sinha, Jing Bai, and Karthik Ramani. Deep learning 3d shape surfaces using geometry images. In European Conference on Computer Vision, pp. 223-240. Springer, 2016.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let net- works learn high frequency functions in low dimensional domains. NeurIPS, 2020.
Tolga Tasdizen, J-P Tarel, and David B Cooper. Algebraic curves that work better. Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), volume 2, pp. 35-41. IEEE, 1999.
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In Proceedings of the IEFE Interna- tional Conference on Computer Vision, pp. 2088-2096, 2017.
The CGAL Project. CGAL User and Reference Manual. CGAL Editorial Board, 5.0.2 edition, 2020. URL
http:oo...a.....o....///aaaaaaaa .html
Francis Williams, Teseo Schneider, Claudio Silva, Denis Zorin, Joan Bruna, and Daniele Panozzo. Deep geometric prior for surface reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 101 10130-10199, 2019.
Francis Williams, Jerome Parent-Levesque, Derek Nowrouzezahrai, Daniele Panozzo, Kwang Moo Yi, and Andrea Tagliasacchi. Voronoinet: General functional approximators with local sup- port. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 264 265,-26552200 2020.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a prob- abilistic latent space of object shapes via 3d generative-adversariall modeling. In Advances in neural information processing systems, pp. 82-90, 2016.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. Ad- vances in Neural Information Processing Systems, 33, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp- 3391-3401, 2017.
Hong-Kai Zhao, Stanley Osher, and Ronald Fedkiw. Fast surface reconstruction using the level set method. In Proceedings IEEE Workshop on Variational and Level Set Methods in Computer Vision, pp. 194-201. IEEE, 2001.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 PROOF OF LEMMA
Lemma 1. For any pair of unit vectors b: min {Ila bl1, |aa + bbll] 2 sin L(a, b)].
Proof. Let a, Rd be arbitrary unit norm vectors. Then,
min {Ila bll ||a + bl]} = [min (2 + 2 (a, b) 2 (a. b)))
2 1 (a, b)
2 coss L(a, b]l 1/2
2 Isin (a. b)l
Where the last inequality can be proved by considering two cases: a E [0, x x/2) and a E [T/2, where we denote a = La,, b). In the first case a E [0, x/2], cos a and this case
T).
1-cos
2
isin g) The inequality is proved by considering
a (1
sin 2 ssn a:| = 2sin 2
sin a = 2 ssn
cos
2
2
for x/2] For the case [T/2, x] considering
we have
1+cos
cos g.l.
This case is proved by
cos
sin a
cos
a cos
o/22 (1
sin
2
2
2
for a E xx
unsigned distance
SALD Figure Al: 2D reconstruction additional results.
IGR (Gropp al. 2020
13

---

Published as a conference paper at ICLR 2021
A.2
IMPLEMENTATION DETAILS
A.2.1 DATA PREPARATION
Given some raw 3D data X, SALD loss (See equation 60 is computed on points and corresponding unsigned distance derivatives, {h (x)) and (Vah (a') b') (resp.) sampled from some distri- butions D and D'. In this paper, we points {y) from X' and placing two isotropic Gaussians, (y, ofl) and (y o².) ) for each y. The distribution parameter σi depends on each point y. set to be as the distance the 50th closest point to y. whereas 72 is set to 0.3 fixed. D2 is chosen by projecting D1 to S. The distribution D is set to uniform on X:: note that on X, (z') is a sub-differential which is the convex hull the two possible normal vectors (=n) at x'; the sign-agnostic loss does not differ between the two normal choices, we arbitrarily use one them in the loss. Computing the unsigned distance to X' is done using the CGAL library (The CGAL Project 2020). To speed up training, we precomputed for each shape in the dataset, 500K samples of the form {h (x)) MED and (Vah (2'))
D1 D2, where D2 is chosen by uniform sampling
A.2.2 GRADIENT COMPUTATION
The SALD loss requires incorporating the term V= 0) differentiable manner. Our com- putation of V fx:( () is based AUTOMATIC DUFFERENTAAION (Baydin et al. 2017)) forward mode. Similarly Gropp et al. 2020), f 0) is constructed as a network consists of layers of the form
V f+1 diag (o' (We+13² + be+1)) We+1 Vay
where yf denotes the output of the & layer in ((x; 0) and 0 = (We, by) are the learnable parameters.
A.2.3 TIMINGS AND NETWORK SIZE
In figure A2 we report the timings and memory footprint of 8-layer MLP with 512 hidden units. As the gradients calculation, V..((x; (), is based on automatic differentiation forward mode, in theory should yield doubling of the forward time. However, in practice we see that the gap increases as increase the number of points for evaluation. For the DFaust experiment (which is the largest dataset in the paper), training was done a batch of 64 shapes and a sample size of 922. It took around 1.5 days complete 3000 epochs with 4 Nvidia V100 32GB gpus. Note that for VAE, the computational cost in test time is equivalent between SAL and SALD.
30
2500
25
SALD
SALD
Porward Time (milisecond)
SAL
SSLL
Memory (MB)
1500
15
10
1000
2500
9000
7400
10000
17500
19000
11800
200000
2500
9000
7500
10000
13800
15000
17900
20000
Sample
Sample Size
Figure A2: Timings (left) and network memory footprint (right), reported on various sample size.
14

---

Published as a conference paper at ICLR 2021
A.2.4 ARCHITECTURE DETAILS
VAE ARCHITECTURE
Our VAE architecture is based on the one used in Atzmon Lipman 2020). The encoder g (X;01), 01). where X Ig is the input point cloud, is composed of DeepSets (aaheer al. 2017 and PointNet @t al. 2017) layers. Each layer consists of
PFC(din doa)): X + (XW + 1b") PL(dm: 2dm) Y + [Y, max (Y)1]
where [s.-] is the concat operation, W & Rdaxdd and b E Rdd are the layer weights and bias and (-) is the pointwise non-linear ReLU activation function. Our encoder architecture is:
PFC(3, 128) PFC(118, 128) + PL(128, 256)
PFC(256, 128) - (1.(128, 256) - PFC 128) +
PL(128, 256) P2C6,56, 128) PL(128, 256)
PFC(256, 256) -> MaxPool X3 FC(256, 256),
v (Wa + b) denotes fully connected layer. The final two fully con- nected layers outputs vectors μ E R and 17 IR 256 used for parametrization multiviariate Gaussian N(j, diag exp 7) used for sampling latent vector R,256 Our encoder architecture is similar to the one used in Mescheder et al. (2019).
where FC(dm, doat)
Our decoder ([x,, 2] 02) is composition of 8 layers where the first layer is FC(256 3, 512), middle layers are FC(512, 512) and the final layer is Linear(512, 1). Notice that the input for the decoder is (æe z] where æ R and is the latent vector. In addition, we add skip connection between the input to the middle fourth layer. We chose the Softplus with 5 = 100 for the non linear activation in the FC layers. For regulrization of the latent x, we add the following term training loss
0.001 (I]ell + |17 + 1111),
similarly to Atzmon & Lipman (2020).
AUTO-DECODER ARCHITECTURE
We use an auto-decoder architecture, similar to the one suggested in Park et al. (2019). We defined the latent vector z E 256 The decoder architecture is the same the one described above for the VAE. For regulrization of the latent z, we add the following term to the loss
0.001 ||
similarly to Park et al. (2019).
A.3 TRAINING DETAILS
We trained our networks using the ADAM (Kingma & Ba 2014) optimizer, setting the batch size to 64. On each training step the SALD loss is evaluated on a random draw 922 points out of the precomputed 500K samples. For the VAE, we set fixed learning rate of 0.0005, whereas for the AD we scheduled the learning rate to start from 0.0005 and decrease by a factor of 0.5 every 500 epochs. All models using PYTORCH deep learning framework (Paszke et al. 2017).
trained for 3000 epochs. Training done 4 Nvidia V-100 GPUs,
A.4 2 AND
For the two dimensional experiments figures and we have used the same decoder as in the VAE architecture with the only difference that the first layer is FC(2, 512) (no concatenation of latent vector to the 2D input). We optimized using the ADAM (Kingma & Ba 2014) optimizer, for 5000 epochs. The parameter À in the SALD loss was set to 0.1.
15

---

Published as a conference paper at ICLR 2021
SALD
Chamter detence
Chamter disennce
Chumter dittence
-ALLE -SAL
0.00
Sample Scoe
Sumple Stoee
Sample Sime
(chair)
(sofa)
(table)
Figure A3: Latent reconstruction sample complexity experiment: Chamfer distance to the input, as function of the sample size. Note the Chamfer distance of the latent reconstruction is oblivious to sample size.
A.5 SAMPLE COMPLEXITY
Figures A3 and A4 show quantitative and qualitative results for auto-decodes latent test shape construction on samples sizes 1K,5K, 5K, 10K, 20K. Note that the reconstruction is oblivious to the sample size. This is possibly due to the fact that the auto-decoder was trained with the maximal sample size.
1K
5K
10K
20K
IK
5K
10K
20K
Figure A4: Latent reconstruction sample complexity experiment: SAL is left, SALD is right. Note the latent reconstruction is oblivious to sample size.
A.6 EVALUATION
Evaluation metrics.
We the following Chamfer distance metrics measure similarity be-
tween shapes:
de (2) (d (X1,))) de (i)
(7)
where
de (2) = [2
Σ
min ||
x2
(8)
26 e
16

---

Published as a conference paper at ICLR 2021
and the sets X either point clouds or triangle soups. In addition, measure similarity of the normals of triangle soups 71, T2, we define:
1 (dw (T1, T2) du (T2, Ti)),
ds (7i, T2)
(9)
where
d (Ti, 72)
1 T
Σ ((m(21), 2
(10)
where Z(a, b) is the positive angle between vectors a, b E R n (21) denotes the face normal of point 2; in triangle soup Ti, and a is the projection of x; on T2-
Tables [ and 2 in the main paper report quantitative evaluation of our method, compared to other baselines. The meshing of the learned implicit representation CUBES algorithm (Lorensen & Cline 1987 on uniform cubical grid of size [512] Comput- ing the evaluation metrics de and dy is done on a uniform sample of 30K points from the meshed surface.
done using the MARCHING
17


Reviews:
[{"review_id": "uwD4wkOHf2b", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper presents SALD, a new type of implicit shape representation that, in addition to predicting the signed distance function, aligns the gradients of the distance function with that of the neural distance field. The resulting algorithm, for example, has improved approximation power and better preserves the sharp features than the ancestor SAL (sign agnostic learning). The formulation is such that the architecture can consume raw point clouds. \n\nSTRENGTHS\n\nThis paper certainly speaks to me. First of all, learning implicit representations directly from raw point clouds can allow for interesting applications such as better generative models or efficient 3D reconstruction networks. The approach is very sensible. In fact, aligning gradients of the implicit surface with the ones of the data is not a new idea and has been done for instance in quadric fitting:\n* Birdal, T., Busam, B., Navab, N., Ilic, S., & Sturm, P. (2019). Generic primitive detection in point clouds using novel minimal quadric fits. IEEE transactions on pattern analysis and machine intelligence, 42(6), 1333-1347.\n* Tasdizen, T., Tarel, J. P., & Cooper, D. B. (1999, June). Algebraic curves that work better. In Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149) (Vol. 2, pp. 35-41). IEEE.\n\n[the paper might benefit from including those especially because it has related work sections called 'primitives' and 'implicit representations'.]. \n\nThis is not a drawback but just the opposite: there is a strong prior evidence that such approaches are useful. I also like that the authors spend a reasonable amount of effort for theoretical analysis. Though, I believe that this can be extended to more realistic scenarios (as the authors aptly explained in the limitations).  \n\n\nWEAKNESSES / ISSUES \n\n- In addition to aligning the gradients, many works benefit from constraining the gradient norm of the implicit function be |\\nabla| = 1. See for instance:\n* Slavcheva, Miroslava, et al. \"Killingfusion: Non-rigid 3d reconstruction without correspondences.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.\n\nCan we think of a similar approach here? Could the paper show some ablations with regularizers concerning the gradient norm?\n\n- Nowadays, the use of implicit 3D representations is omnipresent. In the evaluations, would it be possible to compare against the variants of DeepSDF (e.g. Curriculum DeepSDF or MetaSDF etc.)? With that, it might also be nice to include some more qualitative results in the supplementary. \n\n- Would it be possible to include additional real objects that are non-humans? This might involve for instance cars in an autonomous driving scenario.\n\n- Some discussions on the following aspects could be valuable for the reader: (i) What would be a good suggestion to handle thin-structures? It seems to be a common issue among many SDF-like methods. (ii) The use of raw point sets is good, but such data usually come partially observed. Could this method support partial observations? If not, could there be workaround?\n\n- The Chamfer distance and the variations thereon are obviously not well suited to assess the accuracy of the deep implicit representations. This creates an urge for better quantitative metrics, maybe the data driven ones. For the future, I would strongly suggest thinking about those to have more meaningful evaluation data.\n\n- Some minor remarks:\n* Can we already compare D and D' and give an intuition about what they might refer to at the place they are first defined?\n* \"they strives to\" -> they strive to\n* \"tested SALD ability\" -> tested SALD's ability\n* \"the surfaces produces\" -> \"the surfaces produced\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nc-cVYiVKW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how to generate meshes from raw point clouds. In particular, this paper proposes a framework which is built on top of recent \"sign agnostic learning (SAL)\" work. Compared to SAL, this work adds a gradient penalty term, which encourages the derivative consistency. The problem studied in this paper is important, however, the proposed method is very incremental and has several motivation issues. I summarize the pros and con as follows.\n\nPros:\n1. The idea of using gradient penalty to learn \"sharp\" signed distance function seems convincing. In Figure 4, the proposed method preserves sharp features compared to its counterpart SAL.\n2. This paper presents a theoretic intuition why SALD works -- under uniform distribution assumption, SALD finds the global minimum. \n\nCons:\n1. My biggest concern is the motivation to learn sign distance function from its unsigned observations. For data (ShapeNet and FAUST) used in this paper, signed distances are immediately available -- one can easily convert a mesh to its implicit representation. To me, learning signed distance function (as DeepSDF does) is more convincing since the direct supervision is available. So why does this method bother to learn the proxy objective (unsigned distance function)? \n2. Following 1, the most obvious application of this paper would be learning signed distance function when the distances are not available -- the input is either LiDAR scan or depth image. In that case, if the paper can reconstruct realistic 3D models, it will be much stronger. \n3. To some extent, this paper uses neural networks to learn sign priors from data. There are multiple existing works on this direction which this paper doesn't mention (or briefly mentions but doesn't compare to). E.g, \"Deep geometric prior for surface reconstruction\" and \"Point2Mesh: A Self-Prior for Deformable Meshes\". The paper should at least explain the differences of the tasks if it doesn't compare to them. \n4. In the implementation detail, the paper says it uses a similar architecture to DeepSDF in the auto-decoding case. However, the method shows improvements over DeepSDF. This seems impossible given that DeepSDF learns from direct signed distance supervision. So I am wondering if this is due to model size difference. I'd like to see more comparisons to DeepSDF under exactly the same model capacity. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "GuoA1qVc2I", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is based on the \"sign agnostic learning\" (SAL) method for capturing signed distance functions with neural networks. It extends this method by incorporating derivative information, which interestingly can likewise be handled in a sign agnostic manner. (Maybe I missed this somewhere, but if the derivatives are sign agnostic, couldn't it happen that the inside is positive? Did the authors encounter that in some cases?)\n\nThe paper presents and motivates this extension together with an additional theoretical insight about the minimal surface property of SAL and SALD. In line with SAL, the paper presents a nice variety of results for shapes from different shape databases. The quantitative results are also convincing. It's interesting to see the substantial difference between the VAE and AD architectures. For the comparison with SAL it's good to see the direct improvements from the derivative loss with a VAE.\n\nThe paper leans heavily on SAL, and the change in terms of the overall method seems to be fairly small. Nonetheless, I think it's an interesting insight that the sign agnostic derivatives can be included in this way, and I found it interesting to see how much they improve the results.\n\nGiven that learning signed distance functions is a very active topic, and a very useful building block for a variety of adjacent works that use learned SDFs, the proposed SALD approach seems like a very nice advancement of the state of the art.\n\nSo, overall, I really liked the paper. Figure 2 alone is impressive, and makes a good case for the method. Together with the nice presentation and set of results I think this paper makes for a very good addition to ICLR.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9vsv9fecdRA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary of paper and contributions\nSALD extends prior work on Sign Agnostic neural implicit shape representations to include a loss term on the derivative of the implicit function. The authors justify the benefits of derivatives in 2 ways: (a) By citing prior work [1] which shows empirically that derivatives decrease sample complexity of deep ReLU networks, and (b) By showing qualitative improvements over SAL without derivatives.\n\nThe authors show qualitative evidence that global minimizers of sign agnostic losses (with and without derivatives) satisfy the *minimal surface property*, a desirable property of solutions in commonly discussed the surface reconstruction literature. They demonstrate this property via 2D experiments and via a motivating theoretical example. \nFinally, the authors show their loss function can be integrated into existing generative shape modelling pipelines, comparing results on ShapeNet and D-FAUST against DeepSDF which requires pre-computed SDF data, and SAL which can operate on raw inputs.\n\n## On the benefit of using derivatives\nThe authors cite [1] to motivate the benefit of including derivative terms in the loss. In the case of deep ReLU networks such as the one used by the authors, this prior work shows an empirical reduction in sample complexity when regressing low dimensional functions (Section 4.1) motivated by a theoretical intuition (Section 3).  While the neural implicit functions learned by SALD are indeed low dimensional, the shape-space learning problem is not: It learns a map from a point set (consisting of many points) or a high dimensional (256 in the SALD case) latent code to an implicit function. Given this, I don't believe the authors can simply claim a reduction in sample complexity by citing [1] without demonstrating further experimental evidence, especially given the fact that the experiments in the paper do not show SALD drastically improving over SAL.\n\nIn particular I would be more convinced by an experiment showing the degradation of SAL vs SALD as the number of available samples for a shape is decreased when (a) regressing a single shape directly from data (such as in IGR [2] Section 6), and (b) regressing a shape using an auto-decoder. \n\n## Minimal surface property\nShowing that global minima to SAL may satisfy the minimal surface property is indeed quite interesting. I do feel however that the claim in the paper regarding this is a bit oversold. In particular \"We prove that SAL enjoys a minimal length property in 2D\" (Abstract) and \"Identifying and providing a theoretical justification for the minimal surface property of [sal].\" (end of Section 1). The minimal surface property is well known in the surface reconstruction literature (e.g. [3] cited by the authors in Section 3) and the theorem shown by the authors appears to be for a specific case in 2D unless I am missing something. While these results are not trivial, I feel the contribution should be rephrased to something along the lines of\n\"We give empirical evidence and theoretical motivation that minimizers of SAL-type losses produce solutions satisfying the minimal surface property [citation]\"\n\n## Experimental Evidence\nI feel like the choices of datasets and baselines are sufficient to show the effectiveness of SALD. There are two experiments however which I feel are missing from the paper:\n 1. The sample complexity experiment described above.\n 2. Some kind of performance evaluation. I imagine that computing losses on gradients of networks is quite expensive. How much is the increase in runtime compared to the gains in accuracy?\n\n## Summary of review\nGeneralizing SAL to include derivative quantities is a natural next step for this line of work. The authors show that SALD improves performance over the state of the art on Shapenet and performs comparably on D-FAUST. While these results are great, I feel the paper is missing a few key experiments described above, and that the claims around the minimal surface property are a bit overblown. I am rating this paper as marginally below the acceptance threshold but am more than willing to increase my score if the authors make the requested revisions or give a strong justification as to why they are unnecessary in their rebuttal. \n\n## References\n[1] Czarnecki et. al. - Sobolev Training for Neural Networks\n[2] Gropp e.t. al. - Implicit Geometric Regularization for Learning Shapes\n[3] Zhao et. al. - Fast surface reconstruction using the level set method", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Congratulations!  The reviewers unanimously viewed this work positively and were in favor of acceptance to ICLR.

While the current revision already addresses many reviewer concerns, it may be worth adding some of the datasets pointed out by R3 or comparing to some of the papers suggested by R1.
Decision: Accept (Poster)