StEik: Stabilizing the Optimization of Neural Signed Distance Functions and Finer Shape Representation
Authors: Huizong Yang
Yuxin Sun
Ganesh Sundaramoorthi
Anthony Yezzi
Venue: NeurIPS 2023 poster (2023)
Keywords: Implicit Neural Representation
Surface Reconstruction

Abstract: We present new insights and a novel paradigm for learning implicit neural representations (INR) of shapes. In particular, we shed light on the popular eikonal loss used for imposing a signed distance function constraint in INR. We show analytically that as the representation power of the network increases, the optimization approaches a partial differential equation (PDE) in the continuum limit that is unstable. We show that this instability can manifest in existing network optimization, leading to irregularities in the reconstructed surface and/or convergence to sub-optimal local minima, and thus fails to capture fine geometric and topological structure. We show analytically how other terms added to the loss, currently used in the literature for other purposes, can actually eliminate these instabilities. However, such terms can over-regularize the surface, preventing the representation of fine shape detail. Based on a similar PDE theory for the continuum limit, we introduce a new regularization term that still counteracts the eikonal instability but without over-regularizing. Furthermore, since stability is now guaranteed in the continuum limit, this stabilization also allows for considering new network structures that are able to represent finer shape detail. We introduce such a structure based on quadratic layers. Experiments on multiple benchmark data sets show that our new regularization and network are able to capture more precise shape details and more accurate topology than existing state-of-the-art.

StEik: Stabilizing the Optimization of Neural Signed Distance Functions and Finer Shape Representation
Huizong Yang"
Yuxin Sun"
Ganesh Sundaramoorthi
Anthony Yezzi!
'Georgia Institute of Technology Chuizong, yang, syuxin3, (anzzii@@attecl edu, ganesh. sundaramoorthhi@rtzz. com
RRytheen Technologies
'Equal contribution
Abstract
We present new insights and a novel paradigm (StEik) for learning implicit neural representations (INR) of shapes. In particular, we shed light on the popular eikonal loss used for imposing a signed distance function constraint in INR. We show analytically that as the representation power of the network increases, the optimiza- tion approaches partial differential equation (PDE) in the continuum limit that is unstable. We show that this instability can manifest in existing network optimiza- tion, leading to irregularities in the reconstructed surface and/or convergence sub-optimal local minima, and thus fails to capture fine geometric and topological structure. We show analytically how other terms added to the loss, currently used the literature for other purposes, actually eliminate these instabilities. However, such terms can over-regularize the surface, preventing the representation of fine shape detail. Based on similar PDE theory for the continuum limit, we introduce a new regularization term that still counteracts the eikonal instability but without over-regularizing. Furthermore, since stability is now guaranteed in the continuum limit, this stabilization also allows for considering new network structures that are able represent finer shape detail. We introduce such structure based on quadratic layers. Experiments on multiple benchmark data sets show that our new regularization and network able to capture more precise shape details and more accurate topology than existing state-of-the-ar)
Introduction
Implicit neural representations (INR) [ 17). which are neural network representations for implicit representations of signals (c.g., shape, images), have recently become powerful tool for modeling shape in learning based frameworks for surface reconstruction tasks [ -- 10 17 in computer vision and graphics. INRs typically represent shape as the zero level of its corresponding signed distance function (SDF), which is represented with a neural network (e.g.. a multi-layer perceptron minimizes loss consisting of data fidelity term (e.g., fidelity to known points on the surface, i.e., a point cloud, for the point cloud to surface reconstruction task [1-3, 10 [17) and regularization terms. A regularization term used is the eikonal loss (TI) which constrains the neural representation to be an SDF, While existing methods have shown the ability recover complex scenes and objects, in many cases as datasets become more complex, finer scale geometric and topological structures may not always be recovered.
MLP). To learn an INR,
In this paper, show that the continuum limit of the optimization of neural SDFs the network representation power increases (to recover finer scale shape features) can be unstable due to the eikonal loss. This limits recovery of fine shape details and/or can lead to convergence sub-optimal
Code:
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

local minima resulting in gross errors. We provide theoretical framework based on geometric PDEs and PDE stability analysis [18)-21) which has recently been proven to be a powerful tool
for analyzing deep learning optimization [22 24), to study the optimization of the eikonal loss. Within this framework, we show how other terms (such as the normal constraint (TO) and divergence loss (14) in state-of-the-arr proposed for various end goals can also have stabilizing effect, giving new justification for such terms. However, we show (both in theory and empirically) that such terms, which we show are penalties surface area and mean curvature, can over-regularize the resulting surface, thus failing to capture fine details of the shape (e.g., thin structures). Based on geometric PDEs, show how construct a regularization term that stabilizes the PDE, but does not over-smooth the surface.
Furthermore, stabilizing the eikonal loss optimization with our new regularization enables the use of new neural networks with higher representation power that can capture finer scale details of shape, without suffering from the destabilizing effects of the continuum PDE. We demonstrate this point by proposing a new network structure for neural SDFs, based on quadratic layers (3), which have thus far been applied mainly to classification tasks. Unlike compositions of linear layers that preserve linearity, compositions of quadratic functions are not quadratic, but can be a higher-order polynomial. Thus, we can represent shape with finer piece-wise Taylor approximations, resulting in finer shape representation with fewer network parameters than sfate-of-the-art (14).
Our contributions are: 1. We provide theoretical framework, based on geometric PDEs, to study the optimization of neural SDFs. In particular. we use PDEs to analyze the eikonal loss, and show can be unstable. This theory sheds light on the design of neural SDFs and serves as framework to design new methods. 2. We use this framework to analyze existing terms proposed in the literature, and show how some can provide a stabilizing effect, even though they were motivated for other end goals. This provides new theoretical justification for these methods while also providing a rigorous analysis of their limitations. 3. We use geometric PDEs to propose new loss regularization, i.e., second order derivative in the normal direction, that avoids over-regularization while stabilizing the eikonal loss, 4. We propose the use of quadratic layers for neural SDFs, which provide an arbitrary order piece-wise wise polynomial approximation of the shape. This provides a finer shape representation than existing art, without being subject to the instabilities of the eikonal loss. 5. We provide an extensive benchmark comparison to state of the art on three datasets: the Surface Reconstruction Benchmark (32). ShapeNet (33), and large scene reconstruction [12). We demonstrate that our method consistently improves state-of-the-art, especially on challenging geometries and topologies.
We call our method StEik for stabilizing the eikonal equation. Note that while we benchmark our methods on the task of surface recovery from point cloud data, our theory/methods can be applied to other problems that aim to recover neural SDFs.
2 Related Work
Shape implicit neural representations (INRs): Traditional approaches for representing 3D shapes such as meshes pose difficulties in integrating them with deep learning methods. Deep learning ap- proaches operating directly on traditional representations have limited quality and flexibility. Recently, neural networks have been proposed to represent shapes and scenes using signed distance functions (SDF) [T. 110 [6] or an occupancy functions (2). 33 [17). They have proven more convenient and accurate than traditional representations within deep learning- based solutions. DeepSDF (13) was the first to introduce the use of SDFs in INRs, and was used to represent a collection of shapes. regresses on ground truth SDFs. In many applications, such SDF ground truth is difficult to obtain. Thus, some methods learn INRs directly from raw data, e.g., point clouds (from range scanners) or 2D images (c.g., 3. 16 in multi-view reconstruction applications.
SAL 1 aims recover SDFs from point clouds. SALD [T0] further improves SAL by incorporating surface normal data. IGR (11) proposes loss function based on the eikonal equation, which helps regularize the learned function towards an SDF. FFN 177 and SIREN (12) introduce high frequencies into their architecture to avoid bias towards low- fowffrquencyy solutions in different ways. FFN uses a ReLU MLP that is paired with a Fourier feature layer. SIREN uses the sine activation. Recently, DIGS 14] improves the performance of SIREN on shape representation tasks by proposing a soft constraint on the divergence of the gradient field and proposing new initialization method. DIGS is motivated by avoiding the use of normal data, which is not available for many applications.

---

where f is the artificial parameter of the evolution (the continuum equivalent of the iteration index), VL satisfies the relation ST. 8u = (Su, VI(u)) and the latter expression is the IL2 inner product between the gradient and the (infinite dimensional) perturbation of u, du. Note that while we analyze stability of gradient descent, our analysis also applies to second order optimizers (e.g., Nesterov momentum) as such optimizers do not change stability properties (18). (24). Suppose now that is Approaches above recovering SDFs use the eikonal constraint in training, which we show limits to an unstable PDE, causing artifacts that limit recovery of fine details or convergence to sub-optimal local minima. We provide a theoretical framework to understand this instability and explain how some existing approaches can unknowingly mitigate this instability, however, producing an undesirable over-regularizing effect. Our framework enables us to design new regularizer that stabilizes the eikonal term while recovering finer geometric details (without over-regularizing.
Our new regularization enables us to use new neural networks for SDFs to represent finer shape details without suffering from destabilizing effects of the eikonal term that become more prominent in higher capacity networks. We use quadratic layers in INRs, which is novel, illustrate this point. Quadratic Deep Neural Networks (QDNNs), proposed back in 1990s (25). (26). have been recently used to enhance the learning capability of Deep Neural Networks (DNNs) 17 1-11. Rather than linear functions used in conventional linear layers, a quadratic function is used. Since compositions of quadratic functions can be higher-order polynomials, such QDNNS can represent piece-wise polynomial functions. Thus, QDNNS have better model efficiency because they can approximate polynomial decision boundaries using smaller network depth/width. However, the improvement is limited when it is applied to Convolution Neural Networks (CNN)
Quadratic Deep Neural Networks:
34 39]. In contrast, we demonstrate that using quadratic neurons in MLPs for representing shapes as implicit functions is highly effective.
3 Theory and Analysis of the Stability of Neural SDF Optimization
In this section, we present geometric PDEs as a framework for analyzing the continuum limit of neural SDF optimization, show the instability in the eikonal loss, and show how existing neural SDF approaches can mitigate the instability. This serves as a framework for our new methods in Section 4. To analyze stability of the optimization/PDE. we use spectral methods from numerical PDEs and Von Neumann analysis. Our exposition is self-contained, however, for more detailed introduction, see Ch. 4 of [40) and Ch. 4.3 of [41]. and for a tutorial on stability analysis in the context of deep networks, see 24
3.1 PDE as the Continuum Limit of Neural SDF Optimization
Let Ω C R" + F be the function that is evolving in the continuum (c.g.. level set representation; the hyper-surface of interest is the zero level set, i.c., {x e Ω u(x) = 0}). This is the continuum limit of the typical neural SDF evolutions. Suppose the loss of interest (defined on u) is L. Then the gradient descent given by the PDE:
Ou V-),(
(1)
Ōt
parameterized by 0, denoted ug. as in neural SDFs. We compute the projected gradient descent of the loss with respect to the parameters Note that if we wish to perturb according to a perturbation 50, then du
8u 50. Therefore, 36
Du [ (L(W)==)
) dg (z) 80 dr S. vz(()(z) 00
Ou (x) dx 80.
8L 50
Thus, the projected gradient descent in parameter-space is
de
Ou (x) da.
dt vL(w)(z) 00
(2)
The corresponding PDE evolution of the neural representation (in function space) is
Ou du d0
du
du (V((),,
Σ
(3)
J
00 dl
00;
00,
12
3

---

where 3- au. is a basis for the sub-space of the tangent space of function representations that 00: f;
is spanned by the parameterization of the network (e.g., neural SDF). The evolution above is simply projection of the continuum gradient of the loss onto the basis of the tangent space formed by the neural representation.
Note that as the neural network representation gains more representational power (more capacity to represent finer scale and more divers shapes), the basis B approaches spanning the entire tangent space of functions, i.e., in R", and hence the projected PDE approaches the full PDE (T). Therefore, analyzing the unconstrained PDE ①1 gives insight into the neural representation. In the next sub- sections, we will focus on the notion of stability of the PDEs, which impacts the accuracy of the neural representation.
3.2 PDE Stability Analysis: Theoretical Framework for Analysis of Neural SDF Optimization
Current approaches for learning neural signed distance function minimize loss that consists of data fidelity term and regularization. Regularization aims to keep the representation close to signed distance function, and can also include terms that regularize the underlying shape (e.g.. keep the shape smooth). In this sub-section, we will focus on the eikonal loss that is part of the regularization. necessary condition for a signed distance function is that it satisfies the eikonal PDE and thus the eikonal loss penalizes deviation from that constraint:
Len(u) 1 ||Vu(z)] 1|" dæ.
Vu(x)) 1, E Ω
(4)
where p = 1 or 2 a L' or L loss, respectively and Vu is the spatial gradient.
We claim that the gradient descent PDE for the eikonal loss maybe unstable at some space-time locations. By stability, we mean that the solution of the PDE converges as ( - 0o. By Von Neumann analysis (40). if the homogeneous component of the linearization is non-zero, and the evolution in the frequency (Fourier) domain has an unbounded amplifier, the PDE is unstable. We use Von Neumann analysis to show that the gradient descent PDE of the cikonal loss is unstable. By arguments in the previous section, this means that as the representation of the power of the neural SDF increases, the optimization can become unstable. The gradient descent PDE for the Eikonal loss
du V (Ke Vu),
{ |sgn| [Vu(=)]/[Vu(x)) Vu
p=1
(x)
(5)
p 2=
where sgn is the sign function. The local linearization of this equation is obtained by treating Ke as constant, which is true locally; this results in the linearization:
du a
Au,
(6)
where denotes Laplacian, and note Ke can be positive or negative. When Ki < 0, the process is backward diffusion, which is ill-posed and therefore fundamentally unstable, regardless of the numerical implementation scheme to be used. To see this, we may compute the spatial Fourier transform of the above equation, which yields:
aū Øt (t, w) = (ee = ū(t, w) x e NN kolwi
(7)
where w (Wi..... ..,)) is the frequency variable, and is the Fourier transform of u. Notice that when Ke. < O, the process diverges and so is unstable. Therefore, the projected gradient descent PDE of the Eikonal loss when is represented with a (parametric) neural representation can become unstable as the representational power of the neural SDF increases (approaching the continuum limit).
One may wonder, if the optimization of the Eikonal loss is unstable, why the network optimization seems to converge. There may be several reasons for this. Firstly, since Ke can be positive or negative certain locations, the PDE could go from unstable to stable and even oscillate between these two states without fully blowing up. However, this can cause irregularities in the evolution and recovered shape (see Fi. Second, due to the finite parameterization of neural representations, networks with less capacity may project to a flow that annihilates some of the unstable components. Lastly, as we will see the next sub-section through analysis, various regularization terms introduced (for other purposes) can have stabilizing effect. Nevertheless, these approaches can limit the representational power of the network to represent fine-scale shape details. Our approaches in the next section, built upon our theory, stabilize while allowing more complex networks to have finer shape representation.

---

without reg.
with reg.
(a) Iter. m (before instability)
(b) Iter. m +
(c) Iter. m
Final
Visual demonstration of the eikonal instability in the INR. (a) shows the level set at the iteration, m, before instability of the non-regularized SDF optimization. [Top rowl: shows the level set at various subsequent iterations (b)-(d) of the continued non-regularized SDF evolution. [Bottom row]: shows the level at various iterations of the SDF evolution when adding our proposed regularization (directional divergence) after (a).
Figure 1:
3.3 PDE Stability Analysis of Existing Neural SDF Representations
We now theory analyze existing methods. In [14] a regularization term is added to the loss function for training neural SDFs; the loss (called the divergence loss) is as follows:
Lai- = ff10 (
(8)
where Qn points the ground truth surface (e.g., points of point cloud the zero level set of the ground truth). The authors observe empirically that the Laplacian of a SDF is close to zero and thus this is added constraint. Although show in the next section that this is not always only partially true, we will now show that this term has another beneficial property, ie.. that stabilizes the instability of the eikonal loss gradient descent. The gradient descent PDE for the sum of the above divergence loss and the eikonal loss (o. Teik + az Lavv, where Oe. Old >> O are weights) is
du
AAsgnn Au)]
0re V
[N
Vul
Oid
A
p p=
which is a fourth-order PDE. Note that in implementations, one would have to approximate the sign function with differentiable approximation. We will assume sgn(x:) 2x 1, where is the sigmoid function, i.e., the key property is that the approximation is positively sloped near the origin, and close to constant away from the origin either side. Note that the stability of the PDE is typically dominated by the highest-order terms, which in the above case is stable. To see this, we linearize the first term done previously (assuming K. is constant, and approximating sign as linear near the origin and constant elsewhere). In this case,
Ka (A)(()
Au(x) Au(x)) 0
A|sgn (x)
where Ka > is the slope of the sign approximation at zero. Therefore, in both and p = 2, the linearization of the PDE (near Au for and everywhere for p 2) is given by
du 01 a.ksAu AA]
8u
Ht u
agKK
(10)
j=k=1
5

---

Computing the spatial Fourier transform of the above linearized equation yields:
dū Ōt --
[aKakw]"]] a(t,w) = A(w)ū(t,w)
u(t,w) awwe,
(1I)
Note that in any local approximation of Ki with constant, Kid > O. Thus, regardless of the sign of Kie so long as od is chosen large enough, the set in which A(w) is positive can be minimized, and so the process is stable. Thus, besides aiming to enforce the empirically observed property that the Laplacian of the neural SDF is close to zero, that term also adds stability to the neural SDF optimization, adding a regularizing effect.
In several works, term is included to penalize the deviation between the normal to the SDF and the ground truth normal direction to the surface (or point cloud), which provides further constraints on the recovered SDF. In some problems this ground truth data is available. In addition to serving as an additional constraint, for particular forms that constraint [1]), we show that term can stabilize the eikonal term. The normal constraint is given by the loss:
mormm (u) / /V((z) - NN dx,
(12)
where Qt are points on the ground truth surface. The gradient descent of this term is given by
du at [An (Vu -ou--a Ngat): Kip
Vu Ngel -1 p=
(13)
p p=2
which includes forward diffusion, which if the weight on this term is chosen larger than -ee would stabilize the (unstable) backward diffusion of the eikonal loss.
4
New Shape Regularization and Representation for Finer Neural SDFs
4.1 A New Stabilizing Term Without
We introduce a new stabilizing term for eikonal loss. To motivate our approach, we first shed some more insight into the divergence loss (penalty on the Laplacian of u, the SDF repre- sentation). We first recall fact from differential geometry. For hyper--urface in R", the mean curvature H! of the hyper-surface measures the average turning of the unit normal with respect to n principal directions of the surface [42). We avoid precisely defining the mean curvature due to the lengthy technical details needed, and refer the reader to [42). Of particular interest is pressing the mean curvature of a surface in terms of its level set embedding. If is level set fune- tion, then the mean curvature of the level sets can be written as the divergence of the normal vector field to the level sets [43), i.c.,
Iter. 0
Iter. 1.5k
Iter. 3k
Figure 2: Illustration of the ability of our new reg- ularization to capture fine-scale details of shape while still stabilizing the optimization. Without a penalty on the mean curvature, our directional divergence term restores the shape more quickly and captures fine details (bottom). On the other hand, the full divergence term (top) unnecessarily penalizes the mean curvature of the level sets, re- sulting in over-smoothness. Note the dark black lines represent the zero level set (lighter ones indi- cate other level sets). Ground truth is mlake like shape (dotted gray). Note that both divergence terms avoid instabilities, but our proposed one re- covers the correct curvatures of the shape.
Vu
H div
(14)
Vnl
Note that if is signed distance function, sat- isfies the eikonal equation and thus the mean cur- vature is the Laplacian of the SDF, i.e Hl Au. Hence, for arbitrary shapes, the Laplacian of an SDF is the mean curvature of the level sets, which is not always close to zero. If we would like to represent shapes with fine detail and com- plex curvatures, penalizing the Laplacian of u in the loss would not necessarily beneficial, although the term stabilizes the eikonal loss.

---

However, we note that there is a component of the Laplacian of a SDF that is zero. Indeed, if we compute the gradient of both sides of the eikonal equation ([Vu(x) 1), we obtain that
Vu(x) D²)(x) Vu(x),
0 = DD
(15)
Vu(x:)
where (2)(x) indicates the Hessian of the SDF, and the dot indicates matrix-vector multiplication. Note that the above quantity dotted with Vu is the second derivative of u in the normal direction of the level sets, which a component of the full Laplacian of Hence, we introduce new loss term as replacement for the penalty on the full Laplacian, which we refer to as Laplacian normal regularization or directional divergence:
au) = S Fru(z)DPu((e Vu(z)]
(16)
This loss enforces the constraint in SDFs that the second derivative in the normal direction is zero, without enforcing unwanted smoothness by penalizing the fine detail (points of high mean curvature) of the level sets. This will lead to a fourth-order (non-linear) PDE for its gradient descent. The gradient descent PDE includes term that - A A/Du], an isotropic fourth order term, which from the previous analysis would stabilize the lower order eikonal instability. Although the full flow only regularizes in the normal direction, over the evolution it regularizes over other directions as the normal vector changes direction, killing the cikonal instability.
New Training Loss: We combine the new stabilizing term with the loss function used in SIREN [22 without the normal constraint (for more applicability) to form our proposed training loss:
L = Ote Leik + Om Lmanifold + On Lnon manifold + oz Lt...
Limanifola = 110 |u(a)] dx, Lmon manifold J
(17)
exp (-alu(x)))
where a, O.e: Ch,, Om Or are syper-parametens and f2g are known points the surface of interest (e.g., point cloud data). Imaaiffll penalizes surface points away from the zero level set. Tnon manifold penalizes points not on the surface of interest from being close the zero level set. We use p 1 for the Eikonal loss, the same as in SIREN [12) and DiGS [4]. For aj we use the annealing strategy as in DiGS
[4]. J. See supplementary for details.
4.2 A New Representation for Finer Shape Representation in Neural SDFs
We now introduce a new neural network representation for SDFs, motivated by our result that allows stabilizing the eikonal loss even when the representational power of the network increases. Note in ReLu MLP, the network represents a piecewise-lines function. Activations partition the domain where various linear approximations are used. To capture finer details of shape (without resorting to heavy linear networks), it is natural to leverage more general Taylor series (quadratic and beyond) approximations to capture the curvature of the shape. Motivated by this observation, we propose to use quadratic layers rather than linear layers. Notice that the composition of a quadratic with quadratic function is quartic function, and thus composing quadratic layers many times can approximate any desired order of a Taylor series, even without the use of activations. We still use activations, however, to partition the domain into regions where different Taylor approximations are used. Without stabilizing the eikonal term in the optimization, such finer-scale representations become unstable; thus, regularization plays crucial role. Note quadratic layers have been proposed for neural networks [30]: however, proposing them for shape representation in neural SDFs is novel to the best of our knowledge. Note also that SIREN [12] uses sinusoidal activation to obtain a representation beyond piecewise linear in ReLu MIPs; in that representation, however, the activation serves to both partition the domain in pieces, and represent each piece with more complex (polynomial) functions. Quadratic layers allow more complex (polynomial) functions in the pieces, without overloading the activation with both partitioning the domain and more complex function representation.
As in 30 we define a quadratic layer using the following representation: a(x) (Wix+ + by) (W2x b2) + W3x² b3, where W3 E I Xm x E R is the input vector, x" is the element-wise square, bj E R are biases, and denotes the element wise product. We replace the linear neurons in the SIREN [12 network with quadratic neurons to obtain a high-order expression for the signed distance function. For implementation, use the combination of three linear layer modules in PyTorch
(18)
7

---

Experiments
We now demonstrate the effectiveness of our method on the task of surface reconstruction from point clouds. For all the experiments in this section, we follow the same mesh generation procedure and evaluation setting as the state-of-the-ar method DiGS [14.. We experiment on three benchmarks: Surface Reconstruction Benchmark (SRB) (3). ShapeNet [33], and the Scene Reconstruction Benchmark (12]. We use network with hidden layers and 128 hidden channel for SRB and ShapeNet, and we use 8 hidden layers, and 256 channels for scene reconstruction. The number
of training iterations is the same as DiGS (14), 10k for SRB and ShapeNet, and 100k for scene
reconstruction. We provide all of the training details in the supplementary.
5.1 Surface Reconstruction Benchmark (SRB)
SRB consists of 5 noisy range scans and each contains point cloud and normal data. We com- pare method against the current state-of- the-art methods on this benchmark without us- ing normal data (as in [14) as normal data may be difficult to obtain). Results are shown in Table I We report the Chamfer (dc) and Hausdorff (dz) distances between the recon- structed meshes and the ground truth meshes. Furthermore, we report their corresponding one- sided distances (dg and d between the re- constructed meshes and the input noisy point cloud, which measures how much the recon- struction overfits noise in the input. Results show that StEik is better than SoTA methods on the ground truth metrics, but can slightly overfit the noisy input due to the fine representation property of our method. The improvement is not so dramatic compared to DiGS [T4] because this SRB is a relatively easy task without many thin structures and complex structures, and DiGS (14) already has a good performance. However, we still achieve better result with 25% fewer parameters than DiGS.
GT
Scans
Method
dc
dæ
de
d
IGR wo
1.38
16.33
0.25
2.96
SIREN wo
0.42
7.67
0.08
1.42
I
0.36
7.47
0.13
3.50
IGR+FF
0.96
11.06
0.32
4.75
PHASE+FF 17
0.22
4.96
0.07
1.56
DiGSI 14
0.19
3.52
0.08
1.47
Our Strnk
0.180
2.800
0.096
1.454
Table 1: Quantitative results on the Surface Recon-
struction Benchmark 32 using only point data (no normals).
5.2 ShapeNet
of Shape 33 which consists of 20 shapes in each of 13 categories with only surface point data. Note points are sampled from the shapes (as in (44)) to simulate point clouds. We compare StEik against the current state-of-the-art methods on this dataset without using normal data and report the results in Table 2 As criteria for the benchmark, we consider the Intersection over Union (IoU) and Chamfer Distance between the reconstructed shapes and the ground truth shapes. The Intersection over Union (IoU) captures the accuracy of the predicted occupancy function, while the Chamfer Distance captures the accuracy of the predicted surface. Under both metries, StEik outperforms all other methods by a large margin. This demonstrates that StEik is particularly effective for reconstructing thin structures. A visual example is shown Figure] (see supplementary for more).
We evaluated our method on a preprocessed subset 44).
45
squared Chamfer 1
IoU median
Method SIREN wo n S
mean
median std
mean
3.08c-4
2.58c-4
3,26c-4
0.3085
0.2952
0.2014
1.14e-3
2.11e-4
3.63e-3
0.4030
0.3944
0.2722
DiGS
1.32e-4
2.55e-5
4.73e-4
0.9390
0.9764
0.1262
Ablation (of Regularizations & Linear vs Quad Layers)
Lin+LLa Qua+Laiv
1.71e-4
1.23e-5
1.20e-3
0.9586
0.9809
0.0993
5.45e-5
1.05e-5
3.60e-4
0,9593
0.9852
0.1130
Our StEik (Qua+La..)
6.86c-5
6.33e-6
3.34e-4
0.9671
0.9841
0.0878
Table 2: Quantitative results on the ShapeNet 33
using only point data (no normals).
Ablation Study: Below the middle line in Table 2 we study the effectiveness of each of our novel
contributions (the Laplacian normal regularization and quadratic layers). On 5 out of 6 metries,
8

---

our normal Laplacian regularization using linear networks outperforms DiGs (which uses linear networks), showing the utility the new regularization. On 4 out of metrics, our normal Laplacian regularization out-performs the standard Laplacian regularization using quadratic networks, again showing the utility of our new regularization separately. Note that both cases the metrics where the normal Laplacian normal performs worse are just slightly worse compared to the amount of increase in the other metrics. On all metrics, the quadratic network using the same Laplacian regularization as DiGs out- nut-performs DiGs, showing the utility of quadratic networks alone. Note that each of our contributions, i.e., Laplacian normal regularization and quadratic layers, separately show increase performance against DiGs (except metric in the linear case) even though the ppper-prrameters were not optimized in the approaches compared against DIGS.
Ground Truth
DIGS
Linear LL...
Quadratic Ldiv
Our StEik
Example Visual results on ShapeNet [3 We manifest the effectiveness the new regular- ization and the new representation of Neural SDFs independently. Furthermore, the combination of two modules demonstrates an extra improvement. See supplement for more visual results.
Figure 3:
5.3 Scene Reconstruction
show the reconstruction of room scene point cloud from roughly 10M points and compare our method with D14)) the current SoTA method without normals. This is the same scene used in [12] and contains many thin features that difficult reconstruct. The surface produced by DiGS over-smoothed so that the thin structures like picture frames and sofa legs are not recovered, while in StEik those fine details are recovered.
In Figure 4
DIGS
Our StEik
Figure 4: Visual results the Scene Reconstruction Benchmark using only point data (no normals).
5.4 Timing Performance and Model Size
Table 3 compares the training time of one itera- tion and the number of parameters of DiGS [14] and our method. The evaluation is performed on
Method
Structure
Runtime
Parameters
DiGS
5x256
37.86ms
0.26M
single Nvidia Tesla A000 GPU. The setting above
Lin+LLa
5x256
32.52ms
0.26M
the line is for SRB (32 and ShapeNet (33] ex- periments. The setting below the line is for the scene reconstruction experiment [12]. We achieve better performance than DiGS [14] with only 3/4 the number of parameters. There is an increase in training time (per iteration) for StEik compared to DiGS due to the extra computation cost intro- duced by quadratic neurons. Note that our T.r.. regularization is computationally less expensive
Qua+ Lake
5x 1x8
50.92ms
0.20M
Our StEik
5x128
42.20ms
0.20M
DiGS
8x512
63.28ms
I 1.84M
Lin+Taa
8x512
50.90ms
1 1.84M
Qua+ Laie
8x:256
100.27mm
1 1.39M
Our StEik
8x256
80.62ms
1.39M
Table 3: Network structure, speed, and size com- parison. 5x256 means layers with 256 neurons.

---

than Ldiv (proposed in DiGS) as LL.x can be expressed 2 V (Vu(x), Vu(x))), which can be computed with a single back-propagation call. contrast there is no such simpler expression for the Laplacian.
5.5 Limitations
Due to the lack of efficient implementation of quadratic layers in the deep learning libraries, the increase in training time not negligible. Indeed, there is no native Pytorch implementation of a quadratic layer, and used, we expect that a native Pytorch implementation become available, which would address the current drawback. In addition, there is still improvement space for reconstruction results, in some cases the surface is not perfectly recovered.
currently stack linear layers. As quadratic layers become frequently
Conclusion
We showed that stability is an important consideration in the design of neural SDF representations. We showed that the eikonal loss can result in instabilities that can cause artifacts in both the optimization converge to sub-optimal local minima. Our theory allows for
and the recovered shape, understanding the instability and existing methods for neural SDFs a common framework. Our framework enabled the construction of new regularization term for neural SDFs that stabilizes the instability while avoiding over-regularization. The regularization enabled us to consider finer shape representations with neural SDFs that piecewise polynomial while stabilizing the eikonal term. Empirical results validated our theoretical findings. This work opens up the possibility of exploring broader range of geometric regularizations that naturally arise from PDEs, and the possibility of exploring new finer-scale network representations.
Acknowledgements
Supported in part by Army Research Office (ARO) W911NF-22-1-0261 and by the Intelligence Advanced Research Projects Activity (LARPA) via Department of Interior/ Interior Business Center (DOI/IBC) contract number 140D0423C0075. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of TARPA, DOI/BBC the U.S. Government.
References
[1] M. Atzmon and Y. Lipman, Sal: Sign agnostic learning of shapes from raw data, 2020. arXiv:
1911. 10414 [cs.CV]
[2] Y. Lipman, Phase transitions, distance functions, and implicit neural representations, 2021.
arxiv: 2106. 07689 [C].LG]
[3] M. Niemeyer, I.. Mescheder, M. Oechsle, and A. Geiger, Differentiable volumetric rendering:
Learning implicit 3d representations without 3d supervision, 2020. arXiv: 1912. 07372
[C]]
[4] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman, Volume rendering of neural implicit surfaces, 2021. arXiv: 2106. 12052 [cs
[5] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, Nerf: Representing scenes neural radiance fields for view synthesis, 2020. arXiv: 2003 08934 [cs.CV]
[6] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, Mip-nerf: multiscale representation for anti-aliasing neural radiance fields, 2021. arXiv: 2103. 13415 cvs]cv]
[7] K. Zhang, G. Riegler, N. Snavely, and V. Koltun, Nerf++: Analyzing and improving neural radiance fields, 2020. arXiv: 2010. 07492 [cs.CV]
[8] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and Hedman, Mip-nerf 360: Un-
bounded anti anti-aliased neural radiance fields, 2022. arXiv: 2111. 12077 [cs.CV]
10

---

[9] T. Müller, A. Evans, C. Schied, and A. Keller, "Instant neural graphics primitives with multiresolution hash encoding." ACM Transactions on Graphics, vol. 41, no. 4. pp. 1-15. Jul. 2022. DOL: 10. 1145/11455555 0045/3528223.. 0553822233331127 [Online]. Available: https: //dpi: /ttp::///oo.or/10000/
1145/2F3628223.1 33330127
[10] M. Atzmon and Y. Lipman, Sald: Sign agnostic learning with derivatives, 2020. arXiv:
2006. 05400 [v]
[11]
Gropp, L. Yariv, N. Haim, M. Atzmon, and Y. Lipman, Implicit geometric regularization for learning shapes, 2020. arXiv: 22022 10099 [cs. [G]]
[12] V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein, Implicil neural representations with periodic activation functions, 2020. arXiv: htt::2206..vv 09661
[cs.cv]
[13] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, Deepsdf: Learning continu- ous signed distance functions for shape representation, 2019. arXiv: 1 05103 [cs CV]
[14] Y. Ben- Ben-Shabat, C. H. Koneputugodage, and S. Gould, Digs Divergence guided shape implicir neural representation for unoriented point clouds, 2022. arXiv: 22066 10811 [cs.CV]
[15] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction, 2023. arXiv: 2106 10689 [cs.CV]
[16] L. Yariv, Y. Kasten, D. Moran, al., Multiview neural surface reconstruction by disentangling geometry and appearance, 2020. arXiv: 2003. 09852 [cs.CV)
[17] M. Tancik, P. P. Srinivasan, B. Mildenhall, et al., Fourier features let networks learn high frequency functions in low dimensional domains, 2020. uxXiv:/206...0 220.. 10739 [cs.CV]
[18] M. Benyamin, J. Calder, G. Sundaramoorthi, and A. Yezzi, "Accelerated variational pdes for efficient solution of regularized inversion problems," Journal of mathematical imaging and vision, vol. 62, no. 1, pp. 10-36, 2020.
[19] G. Sundaramoorthi and A. Yezzi, "Variational pdes for acceleration manifolds and appli- cation diffeomorphisms. Advances in Neural Information Processing Systems, vol. 31, 2018.
[20] A. Yezzi, G. Sundaramoorthi, and M. Benyamin, "Accelerated optimization in the pde frame- work formulations for the active contour case," " SIAM journal on imaging sciences, vol. 13, 2020.
no. 4, pp.
[21] G. Sundaramoorthi, A. Yezzi, and M. Benyamin, "Accelerated optimization in the pde frame- work: Formulations for the manifold of diffcomorphisms: SIAM Journal on Imaging Sciences, vol. 15, no. Pp. 324-366, 2022.
[22] Y. Sun, D. Lao, G. Sundaramoorthi, and A. Yezzi, "Accelerated PDEs for construction and theoretical analysis of an SGD extension," in The Symbiosis of Deep Learning and Differ- ential Equations, 2021. [Online]. Available: https: /penrevinr net forum? id j3nedszy5Vc
[23] Y. Sun, D. LAO, G. Sundaramoorthi, and A. Yezzi, "Surprising instabilities in training deep networks and a theoretical analysis," in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35, Curran Asso- ciates, Inc., 2022, pp. 197567199 578. [Online]. httpllablllitttps:////oooo http:///ppppo000/../oo//////aaa.
neur ips Paper
Conference pdf
[24] Y. Sun, D. Lao. G. Sundaramoorthi, and A. Yezzi, Surprising instabilities in training deep networks and a theoretical analysis, 2023. arXiv: 2206. 02001 [cs.LG]
[25] C.-. Chiang and H.-C. Fu, "A variant of second-order multilayer perceptron and its application to function approximations." in [Proceedings 1992] IJCNN Internatioonl Joint Conference on Neural Networks, vol. 3, 1992 887-892 vol.3. DOI: 10. 1109/ /009/I/CNN...... 1992. 227087
[26] B.-L. Lu, Y. Bai, H. Kita, and Y. Nishikawa, "An efficient multilayer quadratic perceptron for pattern classification and function approximation," Proceedings of 1993 International Conference on Neurul Networks (JJCNN-33
Japan), vol. 2, 1993, 1385-1388 vol.2.
DO1: 101:|10.11099 11w./ (..009IIJ/MM IJCNN 1993. 199..7168022
[27] G. Zoumpourlis, A. Doumanoglou, N. Vretos, and P. Daras, "Non-linear convolution filters for enn-based learning." " CoRR, vol. abs/1100.000000 2017, arXiv: 1708. 07038 [Online]. Available: httilable/http:////ee http///xwwv..or/////////// 07038

---

[28] P. Mantini and S. K. Shah, "Cqnn: Convolutional quadratic neural networks," in 2020 25th International Conference on Pattern Recognition (ICPR), 2021, pp. 9819-9826. Do1: 10
2021. 9413207
[29] G. G. Chrysos, S. Moschoglou, G. Bouritsas, Y. Panagakis, J. Deng, and S. Zafeiriou, IeNts:: Deep polynomial neural networks, 2020. arXiv: 2003 03828 [cs. LG]
[30] F. Fan, J. Xiong, and G. Wang, Universal approximation with quadratic deep networks, 2019.
arXiv: 1808. 00098 [CS.LG]
[31] Z. Xu, F. Yu, Xiong, and X. Chen, Quadralib: A performant quadratic neural network library for architecture optimization and design exploration, 2022. arXiv: 22004. 01701 [cs. Lc]
[32] M. Berger, J. A. Levine, L. G. Nonato, G. Taubin, and C. T. Silva, "A benchmark for surface reconstruction," " ACM Trans. Graph., vol. 32, no. 2, Apr. 2013, ISSN: 0730-0301. DO1: 10 [Online]. Available: https: //ooi org/ 10 1145/2451122
1145/2451233 2451246 2451246
[33] A. X. Chang. T. Funkhouser, L. Guibas, et al., Shapenet: An information-rich 3d model repository, 2015. arXiv: 1512. 03012 [cs. [Rs
[34] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." " in Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[35] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," " arXiv preprint arXiv:1409. 1550, 2014.
[36] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in Proceedings of the IEEE conference computer vision and pattern recognition, 2017, pp. 4700-4708.
[37] F. Chollet, "Xception: Deep learning with depthwise separable convolutions," in Proceedings of the IEEE conference on computer vision and patlern recognition, 2017, pp. 1251-1258.
[38] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, "Mobilenetv2: Inverted residuals and linear bottlenecks." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510-4520.
[39] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, "Shufflenet v2: Practical guidelines for efficient cnn architecture design," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 116-131.
[40] L. N. Trefethen, "Finite difference and spectral methods for ordinary and partial differential equations," 1996.
[41] L. C. Evans, Partial differential equations. American Mathematical Society, 2022, vol. 19.
[42] M. Spivak, A comprehensive introduction to differential geometry. Publish or Perish, Incorpo- rated, 1970, vol. 4.
[43] L. Ambrosio and H. M. Soner, "Level set approach to mean curvature flow in arbitrary codimension." Journal of differential geometry, vol. 43, no. 4, pp. 693-737, 1996.
[44] F. Williams, M. Trager, J. Bruna, and D. Zorin, "Neural splines: Fitting 3d surfaces with infinitelywwide neural networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1149-999999 9958.
[45] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger, Occupancy networks: Learning 3d reconstruction in finction space, 2019. arXiv: 1812. 03828 [cs.CV]
12


Reviews:
[{"review_id": "F9ZaU1zX2V", "reviewer": "Reviewer_Bytg", "summary": "This paper presents a novel approach to learning implicit neural representations (INR) of shapes. The paper tries to address the issue of instability in optimizing neural signed distance functions (SDFs) using the eikonal loss, which can lead to irregularities in the reconstructed surfaces and convergence to sub-optimal solutions. The paper provides a theoretical framework based on geometric partial differential equations (PDEs) to analyze the eikonal loss and its instabilities.  They introduce a new Laplacian normal regularization that stabilizes the optimization without over-smoothing the surfaces.\nFurthermore, the paper proposes the use of quadratic layers in neural SDFs, which allow for higher-order polynomial approximations of shape and capture finer details. The paper validates the approach through extensive experiments on benchmark datasets, showcasing the superiority of their method in reconstructing shapes compared to state-of-the-art methods.", "questions": "- What are the differences between the linear backbone and Digs? Why does $L_{div}$ perform slower than $L_{L.n.}$ in terms of time efficiency?\n\n- Why does the proposed method show limited improvement on the Surface Reconstruction Benchmark dataset? Can you provide some explanations or insights into this observation?", "limitations": "Limitations of the proposed method have been discussed in a sub-section of the paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper demonstrates good writing skills, presenting complex concepts and theories in a clear and understandable manner.\n\n- The paper provides strong theoretical explanations for the optimization of neural signed distance functions. By using geometric PDEs, the authors analyze the eikonal loss and its instabilities, revealing the underlying mechanisms and challenges in optimizing neural SDFs.\n\n- Building upon the theoretical insights, the authors propose a new regularization term that addresses the instability issue without over-smoothing the surfaces. ", "weaknesses": "- The paper states that their method improves the representation of shapes compared to state-of-the-art methods. However, the evaluation on the surface reconstruction benchmark dataset does not demonstrate significant performance improvements over other methods. \n\n- The paper introduces a new regularization term to address the instability issue in optimizing neural signed distance functions (SDFs). However, it does not include ablation experiments to investigate the impact of different hyperparameters of the loss function. A comprehensive analysis of the regularization term's sensitivity to hyperparameters could provide valuable insights into its effectiveness and guide the selection of optimal settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1** *The paper states that their method improves the representation of shapes compared to state-of-the-art methods. However, the evaluation on the surface reconstruction benchmark dataset does not demonstrate significant performance improvements over other methods.*\n\n**->** Since SRB is less challenging than the other datasets, without many thin and/or complex structures, DiGS already performs well. Nevertheless, noting that the metrics for SRB represent the average of 5 shapes in the dataset, we go on to offer a finer inspection in the supplementary by providing the performance for each individual shape. DC, Garagoyle and LordQuas  are close to saturated already so large improvements are hard. However, there is a significant improvement for Anchor and Daratech. Especially for the visual result of the Anchor, you can clearly see our method fixes the hole reconstruction.\n\n**Weakness 2** *The paper introduces a new regularization term to address the instability issue in optimizing neural signed distance functions (SDFs). However, it does not include ablation experiments to investigate the impact of different hyperparameters of the loss function. A comprehensive analysis of the regularization term's sensitivity to hyperparameters could provide valuable insights into its effectiveness and guide the selection of optimal settings.*\n\n**->** We have conducted this experiment on SRB varying regularization weights. It shows that around the optimal weight choice, the results are not sensitive.  Furthermore, increasing the weight beyond the optimal, only degrades results slightly since that simply enforces further a constraint that is true of all SDFs, without smoothing the geometry much.  This is consistent Lagrange multiplier theory for enforcing a constraint into the optimization problem.\n\n|            | GT    |       | Scans         |               |\n|-----------|-------|-------|---------------|---------------|\n| $\\alpha_l$ | $d_C$ | $d_H$ | $d_{\\vec{C}}$ | $d_{\\vec{H}}$ |\n| 10         | 0.264 | 6.089 | 0.099         | 1.513         |\n| 50         | 0.191 | 3.799 | 0.096         | 1.485         |\n| $100^*$    | **0.180** | **2.800** | **0.096**         | **1.453**         |\n| 200        | 0.188 | 3.520 | 0.097         | 1.495         |\n| 300        | 0.192 | 3.497 | 0.102         | 1.535         |\n| 400        | 0.187 | 3.177 | 0.102         | 1.537         |\n| 500        | 0.194 | 3.557 | 0.098         | 1.499         |\nTable: Varying $\\alpha_l$ and performance. The relationship between $\\alpha_l$ and the performance is not salient. We may mention that the weight needs to be tuned based on different tasks, but a relatively larger weight is preferred given the annealing strategy.\n\n**Q1.1** *What are the differences between the linear backbone and Digs?*\n\n**->** The DiGS result presented in Table 1 is the result reported in their original paper. To show the improvement of each component, we run all experiments using the same experiment setting(hyperparameter, environment, GPU). $Lin + L_{div}$ is the result of DiGS we got.\n\n**Q1.2** *Why does $L_{div}$  perform slower than $L_{l.n.}$  in terms of time efficiency?*\n\n**->** The major savings comes from the number of back-propagation calls.  In our approach, we only need to compute derivatives of scalar functions in comparison to the divergence term. Since $D^2(x)\\nabla u(x)$ can be computed conveniently as half the derivative of $\\langle \\nabla  u(x), \\nabla  u(x) \\rangle$ with respect to $x$, we can compute $L_{L.n} = \\langle D^2(x)\\nabla u(x), \\nabla u(x)\\rangle$ by a single back-propagation call. In contrast, there is no simpler expression to compute Laplacian, thereby necessitating multiple back-propagation calls to compute each double derivative. Please refer to the supplementary code/models/losses.py line 37-53.\n\n\n**Q2** *Why does the proposed method show limited improvement on the Surface Reconstruction Benchmark dataset? Can you provide some explanations or insights into this observation?*\n\n**->** See Weakness1."}, {"review_id": "Mbw5CMfPXb", "reviewer": "Reviewer_biQ8", "summary": "The paper gives a theoretical framework for analysis the stability of neural SDF optimisation, by considering the gradient descent optimisation as a PDE. They can then analysis the gradient descent on Eikonal loss PDE, and show that it is inherently unstable, and further show that normal loss or divergence loss (which are widely used in the literature to train these methods) help stabilise the PDE. Finally they explain why the divergence loss oversmoothes (as it is equivalent to mean curvature when the eikonal term is satisfied) so they introduce a new loss, Laplacian normal regularisation, that removes this issue by being faithful to SDFs but giving enough second order information to stabilise the PDE. To do this they take the derivative of both sides of the Eikonal equation, and enforce this constraint as a loss. They also introduce quadratic layers in the MLP, in order to fit better to curved surfaces. Their results show good improvements in surface reconstruction performance, however their ablation studies do not make it clear whether the improvement is mainly from quadratic layers or from the new regularisation term.", "questions": "- How important is the annealing strategy on Laplacian normal regularisation? In DiGS, the motivation for the annealing strategy was that the divergence loss, while very \"stabilising\", caused oversmoothing, hence the need to anneal the loss. Your analysis explains that the oversmoothing is due to the mean curvature being penalised everywhere which is not ideal, and instead introduce Lapacian normal regularisation to mitigate this, so it sounds as if annealing would not be necessary?\n- In equation 15/16, it should be a matrix-vector product not a dot product, i.e. D^2u(x)\\nabla u(x) rather than D^2u(x)\\cdot \\nabla u(x).", "limitations": "- Some discussion of limitations w.r.t. DiGS (I guess they assume the reader understands they inherit the limitations of DiGS, e.g. the extra time required for computing and backpropagating through second derivatives, which makes sense)\n- Would like some discussion of whether the author believes that this new loss fixes the issues with thin features", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Interesting theoretical framework for analysis the stability of neural SDF optimisation, and explanation for why normals or divergence loss helps with stability\n- Great insight into why the divergence loss from DiGS is not optimal in terms of mean curvature\n- Using second order information derived from the Eikonal equation makes sense", "weaknesses": "- The ablation results on SRB seem to show that quadratic layers could be more important than the Lapacian normal regularisation, but it is hard to tell as the overall improvements are minor. It would be better to run the ablation on ShapeNet, where there is a much larger improvement in the full model compared to DiGS. \n- If the motivation of quadratic layers is for a better fit to curved surfaces, then a simpler alternative with a lot less parameter increase would be to change the input to quadratic basis functions, e.g. [x,y] -> [1,x,y,xy,x^2,y^2]. Have the authors tried this, or is there more motivation for quadratic layers?\n\nOverall the theoretical analysis is a great contribution. While the results are strong (small improvement on SRB but SRB is close to saturated already so large improvements are hard, and there are impressive improvements on ShapeNet), the ablation study being on SRB doesn't make it clear how much of the contribution is from the main point of the paper, Lapacian normal regularisation, or from quadratic layers. Doing the ablation study on ShapeNet would clear this up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness1** *The ablation results on SRB seem to show that quadratic layers could be more important than the Laplacian normal regularisation, but it is hard to tell as the overall improvements are minor. It would be better to run the ablation on ShapeNet, where there is a much larger improvement in the full model compared to DiGS.*\n\n**->** See Global Response.\n\n**Weakness2** *If the motivation of quadratic layers is for a better fit to curved surfaces, then a simpler alternative with a lot less parameter increase would be to change the input to quadratic basis functions, e.g. $[x,y] -> [1,x,y,xy,x^2,y^2]$. Have the authors tried this, or is there more motivation for quadratic layers?*\n\n**->** The composition of multiple quadratic layers can approximate any desired order of a Taylor series and the composition of multiple linear layers is still only a linear function. Thus, if the input to a linear network is the basis that the reviewer mentions, the output is still quadratic not a higher order polynomial as we desire. This is discussed in the first paragraph of section 4.2.\n\n**Q1** *How important is the annealing strategy on Laplacian normal regularisation? In DiGS, the motivation for the annealing strategy was that the divergence loss, while very \"stabilising\", caused oversmoothing, hence the need to anneal the loss. Your analysis explains that the oversmoothing is due to the mean curvature being penalised everywhere which is not ideal, and instead introduce Laplacian normal regularisation to mitigate this, so it sounds as if annealing would not be necessary?*\n\n**->** The second order loss corresponds to 4th order PDE in the gradient optimization, which involves a diffusion term that requires small learning rates for stability. The relatively large regularization at the start allows the optimization to enforce the constraint, which must be greater than the Lagrange multiplier, and avoid local optimizers.  The smaller regularization towards the end of the evolution allows for larger learning rates that speeds the convergence without being trapped by local optimizers if one employed a uniformly small regularization. Thus, while the annealing strategy may not be necessary to obtain accurate results, it does speed the convergence of the method.  Moreover, the directional Laplacian minimization does still retain some minor remnant of shape regularization only due to the finite parameterization network representation of the shape where a small amount of diffusion can occur away from the normal direction.  Thus, the annealing helps with recovering finer shape details (although much less so than needed for the original Laplacian regularization).\n\n**Q2** *In equation 15/16, it should be a matrix-vector product not a dot product, i.e. $D^2u(x)\\nabla u(x)$ rather than $D^2u(x)\\cdot \\nabla u(x)$.*\n\n**->** We have used operator theoretic notation where the dot notation means \"the operator $D^2u(x)$ applied to $\\nabla u$\", which corresponds to matrix vector multiplication in this case. We will add a comment.\n\n**Limitation1** *Some discussion of limitations w.r.t. DiGS (I guess they assume the reader understands they inherit the limitations of DiGS, e.g. the extra time required for computing and backpropagating through second derivatives, which makes sense)*\n\n**->** No comment necessary.\n\n**Limitation2** *Would like some discussion of whether the author believes that this new loss fixes the issues with thin features*\n\n**->** Penalizing the mean curvature as in the divergence loss can result in pinching thin structures and necks in the gradient flow, which is a well-known property of mean curvature. Since we are not penalizing the mean curvature of the surface, our approach has less of a problem with thin structures as we have seen in visual results.  As discussed in Reviewer biQ8 Q1, the flow can still have a small amount of diffusion in other (tangent) directions, so it may not fully \"fix\" the issue, though works well in practice.  We will add this into discussion."}, {"review_id": "zN3opWwAsw", "reviewer": "Reviewer_bsBS", "summary": "The paper provides a theoretical framework to study the optimization of neural SDFs. In this framework, the eikonal loss - used as regularization to obtain a SDF - results unstable. Existing regularizers are proven to stabilize the eikonal loss, but they empirically lead to oversmoothing. To overcome this issue, the authors propose a novel regularization, promoting a small second derivative in the normal direction, to be used in combination with a more expressive network (i.e., quadratic layers). The experimental section provides comparison with sota in the SDF reconstruction task on three datasets.", "questions": "It would be interesting to investigate how the proposed regularizer performs with different neural network configurations that do not require quadratic layers. \n\nFor instance, could increasing the number of layers and neurons provide a similar effect without significantly increasing computational costs?\n\nIs the proposed framework utilizing sinusoidal activation functions? \n\nCould positional encoding techniques be employed in combination with the proposed regularizer to enhance the representation capabilities of the network? This could potentially improve the network's ability to capture high-frequency detail and produce more accurate SDF reconstructions.", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In my opinion, the papers have several strengths.\n\n*Importance of the task:* The optimization of neural SDFs is a significant task that finds applications in various pipelines. By addressing the stability and oversmoothing challenges, this paper contributes to improving the quality of SDF reconstructions, making it relevant to a wide range of domains.\n\n*Practical Applicability:* The proposed regularizer can be easily integrated into existing SDF reconstruction pipelines, making it convenient for others to adopt in their work.\n\n*Experimental Results:* The authors provide comprehensive experimental comparisons with state-of-the-art methods on three commonly used benchmark datasets. These results demonstrate the effectiveness of the proposed approach and support the assumptions made in the paper.\n\n*Theoretical Justification:* The paper goes beyond empirical observations by providing a theoretical justification for the proposed regularization approach. This strengthens the credibility of the method and enhances the understanding of its underlying principles.", "weaknesses": "I think the main weaknesses to consider are the following.\n\n*Increased Training Time:* The major weakness is the increased training time associated with the proposed method due to the use of quadratic layers, as noticed by the authors. The ablation study suggests that the regularizer alone does not improve performance, necessitating the use of a more expressive network. However, the more expressing network alone would lead to instability. This drawback should be taken into account when considering the practical implementation of the proposed approach.\n\n*Limited Exploration of Neural Architectures:* It would have been valuable to explore alternative neural architectures that could potentially preserve high-frequency details without resorting to computationally expensive quadratic layers. Investigating other network settings, such as increasing the number of layers and neurons, could provide insights into potential trade-offs between computational efficiency and performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness1** *Increased Training Time: The major weakness is the increased training time associated with the proposed method due to the use of quadratic layers, as noticed by the authors. The ablation study suggests that the regularizer alone does not improve performance, necessitating the use of a more expressive network. However, the more expressing network alone would lead to instability. This drawback should be taken into account when considering the practical implementation of the proposed approach.*\n\n**->** As mentioned in Section 5.5, we believe a large part of the computational cost increase is due to the lack of a native Pytorch implementation of quadratic layers, which we currently implement with stacking linear layers.  We hope this work and other recent work with quadratic layers will motivate developers to produce a dedicated Pytorch implementation, which would address this current drawback.\n\n**Weakness2** *Limited Exploration of Neural Architectures: It would have been valuable to explore alternative neural architectures that could potentially preserve high-frequency details without resorting to computationally expensive quadratic layers. Investigating other network settings, such as increasing the number of layers and neurons, could provide insights into potential trade-offs between computational efficiency and performance*\n\n**->** The goal of this paper was to show the instability of the eikonal loss, how to mitigate it without over regularization, and show how this opens up the possibility to new, finer shape representations. Quadratic networks are one possibility that illustrates the latter contribution.  We hope that this paper simulates more thought in the community on networks structures to better represent shape.\n\nHaving said that, a reason that we replace linear neurons with a quadratic neurons is that in FFN [17], it is shown that a standard MLP fails to learn high frequencies, despite increasing MLP parameters. One motivation for quadratic networks was to design a new network structure with more finer shape representation and the same or even fewer parameters. As the comment to Reviewer urXf Q2 indicates, we get a order of magnitude fewer parameters compared to FF(1/4 are due to quadratic and the rest are due to the use of sine activations).\n\nWith regards to increasing number of layers and neurons, that has been explored extensively in the literature for quadratic [23] and extensively for linear MLPs, so the trade-off is well known. Again, we re-iterate our focus, which is to illustrate the possibility of finer shape representation network structures given that the key limitation, the instability of the eikonal loss, has been addressed in this work.\n\n**Q1** *It would be interesting to investigate how the proposed regularizer performs with different neural network configurations that do not require quadratic layers. For instance, could increasing the number of layers and neurons provide a similar effect without significantly increasing computational costs?*\n\n**->** See the previous response. We think the cost of quadratic networks will improve with customized GPU implementations in Pytorch. Increasing layers/neurons will increase the number of parameters, which is detrimental in size/power limited applications, e.g., edge applications.\n\n**Q2** *Is the proposed framework utilizing sinusoidal activation functions?*\n\n**->** Yes, we mentioned it in section 4.2.\n\n**Q3** *Could positional encoding techniques be employed in combination with the proposed regularizer to enhance the representation capabilities of the network? This could potentially improve the network's ability to capture high-frequency detail and produce more accurate SDF reconstructions.*\n\n**->** SIREN demonstrates that the sinusoidal activations are better than ReLU + P.E. when fitting the high-frequency signals. Typically one does not use sinusoidal activation and P.E. together. The combination of sinusoidal activation and P.E. is an interesting future topic, tangential to our focus in the present paper."}, {"review_id": "t7UWrVsXkV", "reviewer": "Reviewer_Nfwa", "summary": "The paper first utilizes Von Neumann analysis to put forward a plausible explanation for instability of neural SDFs as the function space enlarges (as network parameters increase). Along the way, they show that some of the commonly used regularizers can combat this instability if they are weighted heavily enough. They develop a new regularizer based on the fact that the gradient is in the null space of the Hessian for an SDF (almost everywhere), and demonstrate its superior performance on three benchmark datasets. Lastly, they utilize quadratic layers instead of linear ones in their neural SDFs, a novel network structure for neural implicit representations, which seems to increase the expressivity of their SDFs.", "questions": "See the two points above under weaknesses.", "limitations": "Yes, the authors have adequately noted the limitations of their method. The main one is the increased training time for quadratic layers.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Brings in theory and analysis on gradient descent of functionals (on reduced basis function spaces), to shed light on the stability of the Eikonal loss and various regularizers already in use. \n- Development of a novel regularizer that is more stable with respect to increasing parameter count.\n- Demonstration of improvement in quality with use of quadratic layers, a novel architecture.\n- The writing and presentation is quite clear and easy to follow.\n- The insights from this article will help to evaluate the stability of future regularizers and inform their development.", "weaknesses": "- The paper could have provided a bit more context for the Von Neumann analysis. They provide a reference, but it still might be helpful to give a bit of an introduction to the analysis (or give a more specific chapter/section/theorem reference).\n- The ablation study could have also been done on ShapeNet. In particular, I'd be curious to see the quadratic vs linear results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness1** *The paper could have provided a bit more context for the Von Neumann analysis. They provide a reference, but it still might be helpful to give a bit of an introduction to the analysis (or give a more specific chapter/section/theorem reference).*\n\n**->** We will add more related references in the final version, e.g., see L. Evans/PDEs Chapter 4.3 on spectral methods, and Trefethen Chapter 4. We added a reference [33] that provides some tutorial of Von Neumann analysis in the context of DL; we will specify this more explicitly in the final version. Unfortunately, its a collection of techniques and not a single theorem.\n\n**Weakness2** *The ablation study could have also been done on ShapeNet. In particular, I'd be curious to see the quadratic vs linear results.*\n\n->See Global Response."}, {"review_id": "raZljIVkFf", "reviewer": "Reviewer_urXf", "summary": "The authors demonstrate that the eikonal equation which is often used for representing 3D shapes leads to training instabilities. Based on this finding the authors propose a new regularizer for representing 3D shapes based on the eikonal equation. This regularizer is based on the hessian of the SDF function. Moreover, the authors propose to use quadratic layers instead of linear layers. ", "questions": "see weaknesses", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The eikonal equation indeed leads to training instability. The solution based on a second order regularizer seems sensible. The empirical results seem to support the regularizer and architecture choice.", "weaknesses": "- In table 3, why does the runtime decrease when adding the new regularizer which is based on the hessian? I would assume that this impacts the runtime negatively. Can the authors elaborate on this?\n- How does the number of parameters of the methods in tab 1 compare to each other?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1** *In table 3, why does the runtime decrease when adding the new regularizer which is based on the hessian? I would assume that this impacts the runtime negatively. Can the authors elaborate on this?*\n\n**->** The major savings comes from the number of back-propagation calls.  In our approach, we only need to compute derivatives of scalar functions in comparison to the divergence term. Since $D^2(x)\\nabla u(x)$ can be computed conveniently as half the derivative of $\\langle \\nabla  u(x), \\nabla  u(x) \\rangle$ with respect to $x$, we can compute $L_{L.n} = \\langle D^2(x)\\nabla u(x), \\nabla u(x)\\rangle$ by a single back-propagation call. In contrast, there is no simpler expression to compute Laplacian, thereby necessitating multiple back-propagation calls to compute each double derivative. Please refer to the supplementary code/models/losses.py line 37-53.\n\n**Q2** *How does the number of parameters of the methods in tab 1 compare to each other?*\n\n**->** See table below.\n\n\n| Model    | Parameters |\n|----------|------------|\n| IGR      | 2.1M       |\n| SAL      | 2.1M       |\n| IGR+FF   | 2.1M       |\n| PHASE+FF | 2.1M       |\n| SIREN    | 0.26M      |\n| DiGS     | 0.26M      |\n| Ours     | 0.20M      |"}]

Author rebuttal:
We thank the reviewers for a thorough review. There are some concerns mentioned by several reviewers so we will address them here.

**[Slight Correction on ShapeNet Results]**: We apologize that there are some minor errors in the evaluation of ShapeNet in the initial submission. We mistakenly put the result of one category on the table, not the overall result. We have fixed them and updated the new result for ShapeNet here. It does not affect the theory and conclusion of our paper.

**[Ablation Study on ShapeNet]**: In addition, some reviewers requested an ablation study of ShapeNet, so we include it here.  Results are reported on the table below. They show that on 5 out of 6 metrics, our normal Laplacian regularization using linear networks outperforms DiGs (optimized hyper-parameter), showing the utility of the new regularization. On 4 out of 6 metrics, our normal Laplacian regularization out-performs the standard Laplacian regularization using quadratic networks, again showing the utility of our new regularization separately. Note that in both cases the metrics where the normal Laplacian normal performs worse are just slightly worse compared to the amount of increase in the other metrics.
On all 6 metrics, the quadratic network using the same Laplacian regularization as DiGs out-performs DiGs, showing the utility of quadratic networks alone. Note that each of our contributions, i.e., Laplacian normal regularization and quadratic layers, each separately show increase performance against DiGs (except one metric in the linear case) even though the hyper-parameters were not optimized in the approaches compared against DiGs. 


|                       |             | IoU $\uparrow$ |             |              | squared |Chamfer $\downarrow$              |
|-----------------------|-------------|----------------|-------------|--------------|------------------------------|--------------|
| Method                | mean        | median         | std         | mean         | median                       | std          |
| DiGS                  | 0.9390      | 0.9764         | 0.1262      | 1.32e-4      | 2.55e-5                      | 4.73e-4      |
| Lin+$L_{\text{L. n.}}$ | 0.9586      | 0.9809         | 0.0993      | 1.71e-4      | 1.23e-5                      | 1.20e-3      |
| Qua+$L_{\text{div}}$  | 0.9593      | 0.9852    | 0.1130      | 5.45e-5 | 1.05e-5                      | 3.60e-4      |
| Ours                  | 0.9671 | 0.9841         | 0.0878 | 6.86e-5      | 6.33e-6                 | 3.34e-4 |
Table: Ablation study on ShapeNet.

We also named our method StEik(Stable Eikonal) for future convenient reference.

Meta-review: This paper presents an analysis of the popular Eikonal loss in terms of evolution PDE stability. After identifying a potential instability in the standard formulation this paper suggests a novel regularization improving upon this aspect. The paper presents experiments testing this new stabilization technique and other architecture changes (e.g., quadratic layers), comparing also to relevant previous works. The paper is well written, presents a novel and useful analysis and is able to justify its theoretical observations empirically. The paper demonstrates a rather mild improvement qualitatively and quantitatively over baselines. Please incorporate the new ablations in the camera ready version of the paper and improve exposition as requested by reviewers.
Decision: Accept (poster)