SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Huizong Yang $^{1*}$  Yuxin Sun $^{1*}$  Ganesh Sundaramoorthi $^{2}$  Anthony Yezzi $^{1}$
<sup>1</sup>Georgia Institute of Technology <sup>2</sup>Raytheon Technologies <sup>*Equal contribution
{huizong.yang, syuxin3, ayezzi}  $@$  gatech.edu, ganesh.sundaramoorthi@rx.com
We present new insights and a novel paradigm (StEik) for learning implicit neural representations (INR) of shapes. In particular, we shed light on the popular eikonal loss used for imposing a signed distance function constraint in INR. We show analytically that as the representation power of the network increases, the optimization approaches a partial differential equation (PDE) in the continuum limit that is unstable. We show that this instability can manifest in existing network optimization, leading to irregularities in the reconstructed surface and/or convergence to sub-optimal local minima, and thus fails to capture fine geometric and topological structure. We show analytically how other terms added to the loss, currently used in the literature for other purposes, can actually eliminate these instabilities. However, such terms can over-regularize the surface, preventing the representation of fine shape detail. Based on a similar PDE theory for the continuum limit, we introduce a new regularization term that still counteracts the eikonal instability but without over-regularizing. Furthermore, since stability is now guaranteed in the continuum limit, this stabilization also allows for considering new network structures that are able to represent finer shape detail. We introduce such a structure based on quadratic layers. Experiments on multiple benchmark data sets show that our new regularization and network are able to capture more precise shape details and more accurate topology than existing state-of-the-art. $^{1}$
Implicit neural representations (INR) [1]–[17], which are neural network representations for implicit representations of signals (e.g., shape, images), have recently become a powerful tool for modeling shape in learning based frameworks for surface reconstruction tasks [1]–[3], [10]–[17] in computer vision and graphics. INRs typically represent a shape as the zero level set of its corresponding signed distance function (SDF), which is represented with a neural network (e.g., a multi-layer perceptron - MLP). To learn an INR, one minimizes a loss consisting of a data fidelity term (e.g., fidelity to known points on the surface, i.e., a point cloud, for the point cloud to surface reconstruction task [1]–[3], [10]–[17]) and regularization terms. A regularization term used is the eikonal loss [11], which constrains the neural representation to be an SDF. While existing methods have shown the ability to recover complex scenes and objects, in many cases as datasets become more complex, finer scale geometric and topological structures may not always be recovered.
In this paper, we show that the continuum limit of the optimization of neural SDFs as the network representation power increases (to recover finer scale shape features) can be unstable due to the eikonal loss. This limits recovery of fine shape details and/or can lead to convergence to sub-optimal
$^{1}$ Code: https://github.com/sunyx523/StEik
local minima resulting in gross errors. We provide a theoretical framework based on geometric PDEs and PDE stability analysis [18]–[21], which has recently been proven to be a powerful tool for analyzing deep learning optimization [22]–[24], to study the optimization of the eikonal loss. Within this framework, we show how other terms (such as the normal constraint [10] and divergence loss [14]) in state-of-the-art proposed for various end goals can also have a stabilizing effect, giving new justification for such terms. However, we show (both in theory and empirically) that such terms, which we show are penalties on surface area and mean curvature, can over-regularize the resulting surface, thus failing to capture fine details of the shape (e.g., thin structures). Based on geometric PDEs, we show how to construct a regularization term that stabilizes the PDE, but does not over-smooth the surface.
Furthermore, stabilizing the eikonal loss optimization with our new regularization enables the use of new neural networks with higher representation power that can capture finer scale details of shape, without suffering from the destabilizing effects of the continuum PDE. We demonstrate this point by proposing a new network structure for neural SDFs, based on quadratic layers [25]–[31], which have thus far been applied mainly to classification tasks. Unlike compositions of linear layers that preserve linearity, compositions of quadratic functions are not quadratic, but can be a higher-order polynomial. Thus, we can represent shape with finer piece-wise Taylor approximations, resulting in finer shape representation with fewer network parameters than state-of-the-art [14].
Our contributions are: 1. We provide a theoretical framework, based on geometric PDEs, to study the optimization of neural SDFs. In particular, we use PDEs to analyze the eikonal loss, and show it can be unstable. This theory sheds light on the design of neural SDFs and serves as a framework to design new methods. 2. We use this framework to analyze existing terms proposed in the literature, and show how some can provide a stabilizing effect, even though they were motivated for other end goals. This provides new theoretical justification for these methods while also providing a rigorous analysis of their limitations. 3. We use geometric PDEs to propose a new loss regularization, i.e., second order derivative in the normal direction, that avoids over-regularization while stabilizing the eikonal loss. 4. We propose the use of quadratic layers for neural SDFs, which provide an arbitrary order piece-wise polynomial approximation of the shape. This provides a finer shape representation than existing art, without being subject to the instabilities of the eikonal loss. 5. We provide an extensive benchmark comparison to state of the art on three datasets: the Surface Reconstruction Benchmark [32], ShapeNet [33], and large scene reconstruction [12]. We demonstrate that our method consistently improves state-of-the-art, especially on challenging geometries and topologies.
We call our method  $StEik$  for stabilizing the eikonal equation. Note that while we benchmark our methods on the task of surface recovery from point cloud data, our theory/methods can be applied to other problems that aim to recover neural SDFs.
Shape implicit neural representations (INRs): Traditional approaches for representing 3D shapes such as meshes pose difficulties in integrating them with deep learning methods. Deep learning approaches operating directly on traditional representations have limited quality and flexibility. Recently, neural networks have been proposed to represent shapes and scenes using signed distance functions (SDF) [1], [10]–[16] or an occupancy functions [2], [3], [17]. They have proven more convenient and accurate than traditional representations within deep learning-based solutions. DeepSDF [13] was the first to introduce the use of SDFs in INRs, and was used to represent a collection of shapes. It regresses on ground truth SDFs. In many applications, such SDF ground truth is difficult to obtain. Thus, some methods learn INRs directly from raw data, e.g., point clouds (from range scanners) or 2D images (e.g., [3], [16]) in multi-view reconstruction applications.
SAL [1] aims to recover SDFs from point clouds. SALD [10] further improves SAL by incorporating surface normal data. IGR [11] proposes a loss function based on the eikonal equation, which helps regularize the learned function towards an SDF. FFN [17] and SIREN [12] introduce high frequencies into their architecture to avoid bias towards low-frequency solutions in different ways. FFN uses a ReLU MLP that is paired with a Fourier feature layer. SIREN uses the sine activation. Recently, DIGs[14] improves the performance of SIREN on shape representation tasks by proposing a soft constraint on the divergence of the gradient field and proposing a new initialization method. DiGS is motivated by avoiding the use of normal data, which is not available for many applications.
Approaches above recovering SDFs use the eikonal constraint in training, which we show limits to an unstable PDE, causing artifacts that limit recovery of fine details or convergence to sub-optimal local minima. We provide a theoretical framework to understand this instability and explain how some existing approaches can unknowingly mitigate this instability, however, producing an undesirable over-regularizing effect. Our framework enables us to design a new regularizer that stabilizes the eikonal term while recovering finer geometric details (without over-regularizing).
Quadratic Deep Neural Networks: Our new regularization enables us to use new neural networks for SDFs to represent finer shape details without suffering from destabilizing effects of the eikonal term that become more prominent in higher capacity networks. We use quadratic layers in INRs, which is novel, to illustrate this point. Quadratic Deep Neural Networks (QDNNs), proposed back in 1990s [25], [26], have been recently used to enhance the learning capability of Deep Neural Networks (DNNs) [27]–[31]. Rather than linear functions used in conventional linear layers, a quadratic function is used. Since compositions of quadratic functions can be higher-order polynomials, such QDNNs can represent piece-wise polynomial functions. Thus, QDNNs have better model efficiency because they can approximate polynomial decision boundaries using smaller network depth/width. However, the improvement is limited when it is applied to Convolution Neural Networks (CNN) [34]–[39]. In contrast, we demonstrate that using quadratic neurons in MLPs for representing shapes as implicit functions is highly effective.
3 Theory and Analysis of the Stability of Neural SDF Optimization
In this section, we present geometric PDEs as a framework for analyzing the continuum limit of neural SDF optimization, show the instability in the eikonal loss, and show how existing neural SDF approaches can mitigate the instability. This serves as a framework for our new methods in Section 4. To analyze stability of the optimization/PDE, we use spectral methods from numerical PDEs and Von Neumann analysis. Our exposition is self-contained, however, for a more detailed introduction, see Ch. 4 of [40] and Ch. 4.3 of [41], and for a tutorial on stability analysis in the context of deep networks, see [24].
3.1 PDE as the Continuum Limit of Neural SDF Optimization
Let  $u: \Omega \subset \mathbb{R}^n \to \mathbb{R}$  be the function that is evolving in the continuum (e.g., level set representation; the hyper-surface of interest is the zero level set, i.e.,  $\{x \in \Omega : u(x) = 0\}$ ). This is the continuum limit of the typical neural SDF evolutions. Suppose the loss of interest (defined on  $u$ ) is  $L$ . Then the gradient descent is given by the PDE:
$$
\frac {\partial u}{\partial t} = - \nabla L (u), \tag {1}
$$
where  $t$  is the artificial parameter of the evolution (the continuum equivalent of the iteration index),  $\nabla L$  satisfies the relation  $\delta L\cdot \delta u = \langle \delta u,\nabla L(u)\rangle_{\mathbb{L}^2}$ , and the latter expression is the  $\mathbb{L}^2$  inner product between the gradient and the (infinite dimensional) perturbation of  $u$ ,  $\delta u$ . Note that while we analyze stability of gradient descent, our analysis also applies to second order optimizers (e.g., Nesterov momentum) as such optimizers do not change stability properties [18], [24]. Suppose now that  $u$  is parameterized by  $\theta$ , denoted  $u_{\theta}$ , as in neural SDFs. We compute the projected gradient descent of the loss with respect to the parameters  $\theta$ . Note that if we wish to perturb  $u$  according to a perturbation  $\delta \theta$ , then  $\delta u = \frac{\partial u}{\partial\theta}\cdot \delta \theta$ . Therefore,
$$
\delta L \cdot \delta \theta = \int_ {\Omega} \nabla L (u) (x) \frac {\partial u}{\partial \theta} (x) \cdot \delta \theta d x = \int_ {\Omega} \nabla L (u) (x) \frac {\partial u}{\partial \theta} (x) d x \cdot \delta \theta .
$$
Thus, the projected gradient descent in parameter-space is
$$
\frac {\mathrm {d} \theta}{\mathrm {d} t} = - \int_ {\Omega} \nabla L (u) (x) \frac {\partial u}{\partial \theta} (x) \mathrm {d} x. \tag {2}
$$
The corresponding PDE evolution of the neural representation (in function space) is
$$
\frac {\partial u}{\partial t} = \frac {\partial u}{\partial \theta} \frac {\mathrm {d} \theta}{\mathrm {d} t} = - \sum_ {i} \frac {\partial u}{\partial \theta_ {i}} \left\langle \nabla L (u), \frac {\partial u}{\partial \theta_ {i}} \right\rangle_ {\mathbb {L} ^ {2}}, \tag {3}
$$
where  $\mathcal{B} = \left\{\frac{\partial u}{\partial\theta_i}\right\}_i$  is a basis for the sub-space of the tangent space of function representations that is spanned by the parameterization of the network (e.g., neural SDF). The evolution above is simply a projection of the continuum gradient of the loss onto the basis of the tangent space formed by the neural representation.
Note that as the neural network representation gains more representational power (more capacity to represent finer scale and more divers shapes), the basis  $\mathcal{B}$  approaches spanning the entire tangent space of functions, i.e., in  $\mathbb{R}^n$ , and hence the projected PDE approaches the full PDE (1). Therefore, analyzing the unconstrained PDE (1) gives insight into the neural representation. In the next subsections, we will focus on the notion of stability of the PDEs, which impacts the accuracy of the neural representation.
3.2 PDE Stability Analysis: Theoretical Framework for Analysis of Neural SDF Optimization
Current approaches for learning a neural signed distance function minimize a loss that consists of a data fidelity term and regularization. Regularization aims to keep the representation close to a signed distance function, and can also include terms that regularize the underlying shape (e.g., to keep the shape smooth). In this sub-section, we will focus on the eikonal loss that is part of the regularization. A necessary condition for a signed distance function is that it satisfies the eikonal PDE and thus the eikonal loss penalizes deviation from that constraint:
$$
| \nabla u (x) | = 1, \quad x \in \Omega , \quad \Longrightarrow L _ {\mathrm {e i k}} (u) = \frac {1}{2} \int_ {\Omega} | | \nabla u (x) | - 1 | ^ {p} \mathrm {d} x, \tag {4}
$$
where  $p = 1$  or  $p = 2$  for a  $L^1$  or  $L^2$  loss, respectively and  $\nabla u$  is the spatial gradient.
We claim that the gradient descent PDE for the eikonal loss maybe unstable at some space-time locations. By stability, we mean that the solution of the PDE converges as  $t \to \infty$ . By Von Neumann analysis [40], if the homogeneous component of the linearization is non-zero, and the evolution in the frequency (Fourier) domain has an unbounded amplifier, the PDE is unstable. We use Von Neumann analysis to show that the gradient descent PDE of the eikonal loss is unstable. By arguments in the previous section, this means that as the representation of the power of the neural SDF increases, the optimization can become unstable. The gradient descent PDE for the Eikonal loss is
$$
\frac {\partial u}{\partial t} = \nabla \cdot \left(\kappa_ {e} \nabla u\right), \quad \kappa_ {e} (x) = \left\{ \begin{array}{l l} \operatorname {s g n} [ 1 - | \nabla u (x) | ] / | \nabla u (x) | & p = 1 \\ | \nabla u | ^ {- 1} - 1 & p = 2 \end{array} , \right. \tag {5}
$$
where  $\mathrm{sgn}$  is the sign function. The local linearization of this equation is obtained by treating  $\kappa_{e}$  as constant, which is true locally; this results in the linearization:
$$
\frac {\partial u}{\partial t} = \kappa_ {e} \Delta u, \tag {6}
$$
where  $\Delta$  denotes Laplacian, and note  $\kappa_{e}$  can be positive or negative. When  $\kappa_{e} < 0$ , the process is a backward diffusion, which is ill-posed and therefore fundamentally unstable, regardless of the numerical implementation scheme to be used. To see this, we may compute the spatial Fourier transform of the above equation, which yields:
$$
\frac {\partial \hat {u}}{\partial t} (t, \omega) = - \kappa_ {e} | \omega | ^ {2} \hat {u} (t, \omega) \Rightarrow \hat {u} (t, \omega) \propto e ^ {- \kappa_ {e} | \omega | ^ {2} t}, \tag {7}
$$
where  $\omega = (\omega_{1},\dots ,\omega_{n})$  is the frequency variable, and  $\hat{u}$  is the Fourier transform of  $u$ . Notice that when  $\kappa_{e} < 0$ , the process diverges and so is unstable. Therefore, the projected gradient descent PDE of the Eikonal loss when  $u$  is represented with a (parametric) neural representation can become unstable as the representational power of the neural SDF increases (approaching the continuum limit).
One may wonder, if the optimization of the Eikonal loss is unstable, why the network optimization seems to converge. There may be several reasons for this. Firstly, since  $\kappa_{e}$  can be positive or negative at certain locations, the PDE could go from unstable to stable and even oscillate between these two states without fully blowing up. However, this can cause irregularities in the evolution and recovered shape (see Figure 1). Second, due to the finite parameterization of neural representations, networks with less capacity may project to a flow that annihilates some of the unstable components. Lastly, as we will see in the next sub-section through analysis, various regularization terms introduced (for other purposes) can have a stabilizing effect. Nevertheless, these approaches can limit the representational power of the network to represent fine-scale shape details. Our approaches in the next section, built upon our theory, stabilize while allowing more complex networks to have finer shape representation.
Figure 1: Visual demonstration of the eikonal instability in the INR. (a) shows the level set at the iteration,  $m$ , before an instability of the non-regularized SDF optimization. [Top row]: shows the level set at various subsequent iterations (b)-(d) of the continued non-regularized SDF evolution. [Bottom row]: shows the level set at various iterations of the SDF evolution when adding our proposed regularization (directional divergence) after (a).
3.3 PDE Stability Analysis of Existing Neural SDF Representations
We now use our theory to analyze existing methods. In [14], a regularization term is added to the loss function for training neural SDFs; the loss (called the divergence loss) is as follows:
$$
L _ {\mathrm {d i v}} (u) = \int_ {\Omega \backslash \Omega_ {0}} | \Delta u (x) | ^ {p} \mathrm {d} x, \tag {8}
$$
where  $\Omega_0$  are points on the ground truth surface (e.g., points of a point cloud or the zero level set of the ground truth). The authors observe empirically that the Laplacian of a SDF is close to zero and thus this is added as a constraint. Although we show in the next section that this is not always or only partially true, we will now show that this term has another beneficial property, i.e., that it stabilizes the instability of the eikonal loss gradient descent. The gradient descent PDE for the sum of the above divergence loss and the eikonal loss ( $\alpha_{e}L_{\mathrm{eik}} + \alpha_{d}L_{\mathrm{div}}$ , where  $\alpha_{e}, \alpha_{d} > 0$  are weights) is
$$
\frac {\partial u}{\partial t} = \alpha_ {e} \nabla \cdot \left[ \kappa_ {e} \nabla u \right] - \alpha_ {d} \left\{ \begin{array}{l l} \Delta [ \operatorname {s g n} (\Delta u) ] & p = 1 \\ \Delta [ \Delta u ] & p = 2 \end{array} \right. \tag {9}
$$
which is a fourth-order PDE. Note that in implementations, one would have to approximate the sign function with a differentiable approximation. We will assume  $\operatorname{sgn}(x) = 2\sigma(x) - 1$ , where  $\sigma$  is the sigmoid function, i.e., the key property is that the approximation is positively sloped near the origin, and close to a constant away from the origin on either side. Note that the stability of the PDE is typically dominated by the highest-order terms, which in the above case is stable. To see this, we linearize the first term as done previously (assuming  $\kappa_{e}$  is constant, and approximating sign as linear near the origin and constant elsewhere). In this case,
$$
\Delta [ \operatorname {s g n} (\Delta u) ] (x) \approx \left\{ \begin{array}{l l} \kappa_ {d} \Delta [ \Delta u ] (x) & \Delta u (x) \approx 0 \\ 0 & | \Delta u (x) | \gg 0 \end{array} \right.,
$$
where  $\kappa_{d} > 0$  is the slope of the sign approximation at zero. Therefore, in both  $p = 1$  and  $p = 2$ , the linearization of the PDE (near  $\Delta u = 0$  for  $p = 1$  and everywhere for  $p = 2$ ) is given by
$$
\frac {\partial u}{\partial t} = \alpha_ {e} \kappa_ {e} \Delta u - \alpha_ {d} \kappa_ {d} \Delta [ \Delta u ] = \alpha_ {e} \kappa_ {e} \sum_ {j = 1} ^ {n} \frac {\partial^ {2} u}{\partial x _ {i} ^ {2}} - \alpha_ {d} \kappa_ {d} \sum_ {j, k = 1} ^ {n} \frac {\partial^ {4} u}{\partial x _ {j} ^ {2} \partial x _ {k} ^ {2}}. \tag {10}
$$
Computing the spatial Fourier transform of the above linearized equation yields:
$$
\frac {\partial \hat {u}}{\partial t} (t, \omega) = - \left[ \alpha_ {e} \kappa_ {e} | \omega | ^ {2} + \alpha_ {d} \kappa_ {d} | \omega | ^ {4} \right] \hat {u} (t, \omega) = A (w) \hat {u} (t, \omega) \Rightarrow \hat {u} (t, \omega) \propto e ^ {A (\omega) t}. \tag {11}
$$
Note that in any local approximation of  $\kappa_{d}$  with a constant,  $\kappa_{d} > 0$ . Thus, regardless of the sign of  $\kappa_{e}$ , so long as  $\alpha_{d}$  is chosen large enough, the set in which  $A(\omega)$  is positive can be minimized, and so the process is stable. Thus, besides aiming to enforce the empirically observed property that the Laplacian of the neural SDF is close to zero, that term also adds stability to the neural SDF optimization, adding a regularizing effect.
In several works, a term is included to penalize the deviation between the normal to the SDF and the ground truth normal direction to the surface (or point cloud), which provides further constraints on the recovered SDF. In some problems this ground truth data is available. In addition to serving as an additional constraint, for particular forms of that constraint [11], we show that term can stabilize the eikonal term. The normal constraint is given by the loss:
$$
L _ {\text {n o r m .}} (u) = \int_ {\Omega_ {o}} | \nabla u (x) - N _ {g t} | ^ {p} \mathrm {d} x, \tag {12}
$$
where  $\Omega_o$  are points on the ground truth surface. The gradient descent of this term is given by
$$
\frac {\partial u}{\partial t} = \nabla \cdot \left[ \kappa_ {n} \left(\nabla u - N _ {g t}\right) \right], \quad \kappa_ {n} = \left\{ \begin{array}{l l} | \nabla u - N _ {g t} | ^ {- 1} & p = 1 \\ 1 & p = 2 \end{array} , \right. \tag {13}
$$
which includes a forward diffusion, which if the weight on this term is chosen larger than  $-\alpha_{e}\kappa_{e}$ , would stabilize the (unstable) backward diffusion of the eikonal loss.
4 New Shape Regularization and Representation for Finer Neural SDFs
4.1 A New Stabilizing Term Without Over-Regularization
We introduce a new stabilizing term for eikonal loss. To motivate our approach, we first shed some more insight into the divergence loss (penalty on the Laplacian of  $u$ , the SDF representation). We first recall a fact from differential geometry. For a hyper-surface in  $\mathbb{R}^n$ , the mean curvature  $H$  of the hyper-surface measures the average turning of the unit normal with respect to  $n$  principal directions of the surface [42]. We avoid precisely defining the mean curvature due to the lengthy technical details needed, and refer the reader to [42]. Of particular interest is expressing the mean curvature of a surface in terms of its level set embedding. If  $u$  is a level set function, then the mean curvature of the level sets can be written as the divergence of the normal vector field to the level sets [43], i.e.,
$$
H = \operatorname {d i v} \left(\frac {\nabla u}{| \nabla u |}\right). \tag {14}
$$