SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruian Wang*
Shandong University
wra.time@gmail.com
Zixiong Wang*
zixiong_wang@outlook.com
Yunxiao Zhang
zhangyunxiaox@gmail.com
Shuangmin Chen†
Qingdao University of
csmqq@163.com
Shiqing Xin
ShanDong University
xinshiqing@sdu.edu.cn
Changhe Tu
chtu@sdu.edu.cn
Wenping Wang
TEXAS A&M UNIVERSITY
wenping@cs.hku.hk
The Signed Distance Function (SDF), as an implicit surface representation, provides a crucial method for reconstructing a watertight surface from unorganized point clouds. The SDF has a fundamental relationship with the principles of surface vector calculus. Given a smooth surface, there exists a thin-shell space in which the SDF is differentiable everywhere such that the gradient of the SDF is an eigenvector of its Hessian matrix, with a corresponding eigenvalue of zero. In this paper, we introduce a method to directly learn the SDF from point clouds in the absence of normals. Our motivation is grounded in a fundamental observation: aligning the gradient and the Hessian of the SDF provides a more efficient mechanism to govern gradient directions. This, in turn, ensures that gradient changes more accurately reflect the true underlying variations in shape. Extensive experimental results demonstrate its ability to accurately recover the underlying shape while effectively suppressing the presence of ghost geometry.
In recent years, the neural signed distance function (SDF) has demonstrated its capability to represent high-fidelity geometry [37, 16, 8, 53, 1, 19]. Existing approaches primarily employ neural networks to map coordinates to their corresponding signed distance values. Depending on whether or not supervision is used, these approaches can be classified into two categories: learning-based methods and optimization-based methods, where the former fits data samples to their corresponding ground-truth implicit representations [37, 16, 8, 22, 21] and the latter directly infer the underlying SDF from point clouds [1, 19, 5, 3, 48] or multi-image [45, 29, 35]. Despite significant advancements in SDF-based surface reconstruction, both types of methods have their drawbacks. The supervised reconstruction methods may not generalize well [42] on shapes or point distributions that are not present in the training data. The optimizing-based methods, on the other hand, struggle to resolve the ambiguity of the input point cloud, particularly when normal information is absent. In this paper, our
*These authors contributed equally.
$\dagger$  S. Chen is the corresponding author.
focus is primarily on optimization-based reconstruction techniques for point clouds without normal information.
The majority of optimization-based approaches utilize first-order constraints to regulate the variation of the SDF. For example, the Eikonal term [19, 40, 53] is commonly employed to ensure that the gradients are unit vectors. However, as pointed out in [6], the Eikonal term is weak in its ability to regulate the direction of the gradients. As a result, it is challenging to prevent the emergence of ghost geometry (the level sets of the SDF are highly disordered) and unnecessary shape variations even when the Eikonal term is enforced. To address these issues, various second-order smoothness energy formulations [54, 6] are proposed to steer the direction of the gradients to change toward a desirable configuration. However, it entails considerable difficulty to regulate the extent to which the smoothness term is enforced. A commonly seen artifact is that the resulting surface may be excessively smooth.
In this research, we re-examine the problem of surface reconstruction based on surface vector calculus, and introduce a novel loss to facilitate the inference of a faithful SDF directly from raw point clouds without oriented normals. An interesting observation is that within the narrow thin-shell space of the underlying surface, where the real SDF is differentiable everywhere in the narrow thin-shell space, the gradient of the SDF is an eigenvector of its Hessian matrix and the corresponding eigenvalue is zero. Specially, when a point is situated on a surface, the normal vector transforms into an eigenvector of the Hessian of the SDF, with the corresponding eigenvalue being zero. In essence, the gradient and Hessian of the SDF must be aligned within a thin-shell region surrounding the underlying surface. This alignment allows for more effective control over the direction of the gradients. Building on the key observation, we develop a new loss function that promotes the alignment of the gradient and Hessian of the SDF, rather than relying on smoothness energy. We have conducted a comprehensive evaluation of our proposed approach on a variety of benchmarks and compared it to recently proposed reconstruction methods. Extensive experimental results show that our approach outperforms the state-of-the-art techniques, whether overfitting a single shape or learning a shape space. It can not only accurately recover the underlying shape but also suppress the occurrence of ghost geometry, which verifies the effectiveness of the gradient-Hessian alignment.
2.1 Traditional Methods
Traditional reconstruction methods can be classified into two categories: explicit and implicit. Explicit methods focus on establishing direct connections between input points, while implicit methods aim to fit an implicit field that conforms to the given points and normals. Roughly speaking, popular explicit reconstruction techniques utilize computational geometry methods, such as Delaunay triangulation or Voronoi diagrams, to infer connections between points. These methods can produce well-tessellated triangle mesh surfaces [26, 14, 47]. However, they may struggle to ensure manifoldness, particularly when the input point cloud contains defects such as irregular point distribution, noise, or missing parts. In contrast, implicit reconstruction techniques [25, 23, 39] are capable of generating manifold and watertight surfaces. However, the majority of these methods necessitate that the input point cloud is equipped with normal information.
In the event that the provided point cloud is devoid of normals, it is necessary to either estimate the normals and orientations prior [52] to surface reconstruction or devise a novel reconstruction framework that can operate in the absence of normal information [20, 27]. In essence, all of them have to estimate normals either prior to or during the process of surface reconstruction. As a result, these approaches remain heavily reliant on the quality of the estimated normals. In cases where the provided point cloud is of poor quality, it becomes challenging for these methods to overcome the inherent ambiguity introduced by the absence of normal information.
2.2 Supervision-based Learning Methods
Learning-based methodologies have demonstrated a superior capacity for reconstruction, particularly in the context of neural implicit function [51]. They optimize the network to implicitly encode a signed distance field or occupancy field with the supervision of ground truth. Preceding methodologies primarily employ object-level priors [37, 34, 18] to encode shapes, which are referred to as global
prriors. However, the utilization of global priors restricts the generation ability as the network is unable to deal with shapes that are absent from the training set. Subsequently, numerous methodologies concentrate on local data priors [22, 16, 8, 21] to enhance generalization capacity by utilizing small receptive fields. The approaches for acquiring local data priors encompass regular grid [22], KNN [17, 16, 8], and octree [46, 44] and so on. Although local data priors enhance the generation ability of learning-based methodologies, the priors derived from limited data may result in excessively smoothed surfaces. At the same time, the reconstruction performance may further deteriorate if the distribution of the test point cloud significantly deviates from the training data [42].
2.3 Optimization-based Learning Methods
To enhance the generalization ability, the direct fitting of 3D representations from raw point clouds without supervision has been extensively investigated in recent years, enabling end-to-end prediction of the target surface. The majority of optimization-based reconstructions necessitate the fitting of individual shapes with specific network parameters, where the parameters are acquired by imposing additional constraints. SAL/SALD [1, 2] employs unsigned distances to facilitate sign-agnostic learning. IGR [19] and Neural-Pull [5] primarily utilize Eikonal terms to constrain the field to be an SDF. Additionally, several methodologies [3, 4, 55] have been modified from Neural-Pull in pursuit of enhanced quality. DiGS [6] incorporates Laplacian energy into neural implicit representation learning for unoriented point clouds. In summary, the preponderance of existing methodologies constrain the norm of gradients or minimize smoothness energy to attain equilibrium between geometric details and overall simplicity, but this inevitably results in ghost geometry or over-smoothed reconstruction outcomes. In this paper, our emphasis is on the alignment of the gradients and the Hessian of the SDF to more effectively regulate the direction of the gradients.
3.1 Neural signed distance function
Given an unoriented point cloud  $\pmb{P}$  restricted in the range  $\Omega : [-1, 1] \times [-1, 1] \times [-1, 1]$ , our task is to find a neural signed distance function (SDF)  $f_{\theta} : \mathbb{R}^3 \mapsto \mathbb{R}$ , such that  $f_{\theta}$  predicts a signed distance value for an arbitrary query point  $\pmb{q} \in \Omega$ , where the neural function  $f_{\theta}$  is parameterized by  $\theta$  and can be assumed to be  $C^2$ -continuous everywhere in the domain. We use  $S_l$  to denote the level-set surface at the value of  $l$ , i.e.,
$$
\mathcal {S} _ {l} = \left\{\boldsymbol {q} \in \mathbb {R} ^ {3} \mid f _ {\theta} (\boldsymbol {q}) = l \right\}. \tag {1}
$$
The underlying surface can be naturally obtained by extracting the zero level-set surface  $S_0$ , through the use of contouring algorithms, such as Marching cubes [30]. The task of this paper is to fit the high-fidelity implicit representation without supervision.
3.2 Surface vector calculus related to SDF
In order to fit the SDF, there are typically three types of boundary conditions for constraining  $f_{\theta}$ : (1) Dirichlet condition  $f_{\theta}(p) = 0$  that requires each point  $p \in P$  to be situated on  $S_0$  as far as possible, (2) Eikonal condition  $\| \nabla f_{\theta} \|_2 = 1$  that enforces  $f_{\theta}$  be a distance field with unit gradients, and (3) Neumann condition  $\nabla f_{\theta} = \mathcal{N}$  that aims to align the gradients with the normal field  $\mathcal{N}$ . Existing neural implicit representations incorporate the aforementioned boundary conditions as constraints, either explicitly [19, 40] or implicitly [5, 1]. However, it is important to note that the Neumann condition cannot be enforced due to the absence of oriented normals.
According to the surface vector calculus [32], the SDF has a fundamental relationship with the
Figure 1: A 2D visual for differential property of SDF for circle. From left to right: SDF with gradient, minimum (0) and maximum eigenvalues of the Hessian matrix with their corresponding eigenvector.
differential properties of a smooth surface. Let  $\mathbf{J}_{\nabla f_{\theta}}$  denote the Jacobian matrix of the gradient  $\nabla f_{\theta}$
or equivalently the Hessian matrix  $\mathbf{H}_{f_{\theta}}$  of  $f_{\theta}$ . Since this matrix is symmetric and diagonalizable, we can find the eigenvectors and eigenvalues of  $\mathbf{H}_{f_{\theta}}$  for any point where  $f_{\theta}$  is differentiable (a narrow thin-shell space surrounding the base surface). By differentiating both sides of the identity  $\| \nabla f_{\theta}\|_{2} = 1$ , we get:
$$
\mathbf {H} _ {f _ {\theta}} \nabla f _ {\theta} (\boldsymbol {q}) = \mathbf {0}, \tag {2}
$$
which implies that  $\nabla f_{\theta}(\pmb{q})$  is exactly an eigenvector of the Hessian  $\mathbf{H}_{f_{\theta}}(\pmb{q})$ , with a corresponding eigenvalue of 0. To be more detailed, when  $\pmb{q}$  lies on the surface, the gradient represents the normal vector at  $\pmb{q}$ . Consequently, we have
$$
\mathbf {H} _ {f _ {\theta}} \mathcal {N} (\boldsymbol {q}) = \mathbf {0}. \tag {3}
$$
Additionally, the other two eigenvectors of  $\mathbf{H}_{f_{\theta}}(\pmb{q})$  define the principal directions at  $\pmb{q}$  [32, 36], with the eigenvalues respectively being the opposite of the corresponding principal curvatures. If  $\pmb{q}$  is off the surface but in the differentiable region, the three eigenvectors reflect the differential properties of the closest surface point of  $\pmb{q}$ , i.e., the projection of  $\pmb{q}$  onto the surface. In summary, the eigenvectors of the Hessian can disclose the fundamental properties of the underlying surface.
A similar property holds in the 2D setting. In Fig. 1, we provide a 2D circle as a toy example to illustrate this property. It's evident that in the vicinity of the base surface, the Hessian matrix of the SDF consistently exhibits two eigenvectors: one aligned with the gradient, and the other orthogonal to the gradient. Moreover, we visualize the distribution of the eigenvalues using a color-coded style, with the eigenvalue corresponding to the gradient being 0.
3.3 Gradient-Hessian Alignment
It should be noted that even if the underlying surface is infinitely smooth, the SDF may not necessarily be smooth everywhere. From a geometric perspective, the SDF is non-differentiable at medial-axis points with at least two nearest projections onto the surface. The neural implicit function  $f_{\theta}$ , being at least  $C^2$ -smooth, is unlikely to be identical to the real SDF. We can make a reasonable assumption that  $f_{\theta}$  resembles the real SDF in the differentiable region that in the non-differentiable region. In light of this, it is reasonable to enforce the alignment between the gradients and the Hessian for points within a thin-shell space of appropriate width.
Based on the aforementioned analysis, we propose a loss function to promote the alignment of the Gradient-Hessian:
$$
L _ {\text {a l i g n}} (\boldsymbol {q}) = \left\| \mathbf {H} _ {f _ {\theta}} \boldsymbol {g} (\boldsymbol {q}) \right\| _ {2} ^ {2} \tag {4}
$$
$$
\boldsymbol {g} (\boldsymbol {q}) = \frac {\nabla f _ {\theta} (\boldsymbol {q})}{\| \nabla f _ {\theta} (\boldsymbol {q}) \| _ {2}}. \tag {5}
$$
Figure 2: The level-sets, from left to right, show the distance fields learned by SIREN [40], SIREN with dirichlet energy [28], laplican energy [6], and hessian energy [54], DiGS [6] and ours with 100 points (black) as input. The black lines represent zero-isosurface. Our methods effectively suppress the ghost geometry with the concern of the gradient directions.
Figure 3: Visual comparisons on SRB [49].
Chamfer	F-Score
mean ↓	std. ↓	mean ↑	std. ↓
SPSR* [23]	4.36	1.56	75.87	18.57
SIREN [40]	18.24	17.09	38.74	31.26
SAP [38]	6.19	1.75	57.21	11.66
iPSR [20]	4.54	1.78	75.07	19.18
PCP [3]	6.53	1.75	47.97	14.50
CAP-UDF [55]	4.54	1.82	74.75	18.84
DiGS [6]	4.16	1.44	76.69	18.15
Ours (SIREN)	3.86	1.10	78.80	16.01
Table 1: Quantitative comparison on SRB [49]. The methods marked with \* require point normals.
Difference from the Eikonal term. By enforcing the Eikonal term, we can prevent the SDF from degenerating into a trivial field, such as  $f_{\theta} = 0$  almost everywhere. But for a distance field rooted at any surface,  $\| \nabla f_{\theta}\| = 1$  always holds except at the non-differentiable points. In practice,  $\| \nabla f_{\theta}\| = 1$  is evaluated at discrete points, and the number of network parameters may significantly exceed the number of sample points. Consequently, specifying only  $\| \nabla f_{\theta}\| = 1$  can result in ghost artifacts in the reconstructed surface, such as bulging effects, redundant parts, or excessive surface variations. For instance, consider the last row of Fig. 2. The reconstructed surfaces may differ significantly from the desired surface, even when  $\| \nabla f_{\theta}\| = 1$  holds almost everywhere. This observation motivates us to seek more effective control over the gradient direction.
Why not smoothness energy. In previous literature, various forms of smoothness energy, such as Dirichlet Energy [28], Hessian energy [9, 54], and Laplacian energy [6], have been employed. Generally, the inclusion of smoothness terms can encourage smoothness in the SDF, reducing variations, and thereby mitigating the emergence of ghost geometries.
However, for these smoothing-based approaches, while they can reduce the occurrence of ghost geometry, they often sacrifice the ability to accurately represent geometric details. In contrast, our regularization term enforces the alignment between the gradient and the Hessian, which significantly differs from enforcing simple smoothness. As shown in Fig. 2, we created a 'L' shape with 100 boundary points, following the same experimental setup as DiGS [6], to illustrate the contrast between various approaches. Smoothing-based approaches tend not to reduce the smoothness energy to zero, even at termination. In contrast, our regularization term can approach zero more closely. Moreover, aligning the gradient and the Hessian follows an inherent property of the SDF, independent of the specific case. This sets our approach apart from smoothing-based methods that require case-by-case tuning of smoothing weights. As a result, our approach exhibits superior feature-preserving ability.
We conducted two ablation studies in Section 4.3, which clearly demonstrate that our alignment loss term not only more effectively suppresses the occurrence of ghost geometry but also yields more faithful reconstruction results than traditional smoothing-based approaches.
3.4 Loss function
Based on the aforementioned discussion, our neural function  $f_{\theta}$  has to be at least  $C^2$  continuous. In this paper, we consider two types of activation functions. The first activation function is Sine, which facilitates learning high-frequency information as pointed out in SIREN [40]. Obviously, Sine is  $C^{\mathrm{inf}}$  continuous and satisfies our requirements. Additionally, Neural-Pull [5] leverages the SoftPlus activation function, which is a smooth version of ReLU and can also serve our purpose.
To this end, we establish the entire loss function by integrating the alignment loss into the existing loss configuration such that the network optimization can be formulated as
$$
\underset {\theta} {\arg \min } \left\{E _ {\text {o l d}} \left(f _ {\theta}\right) + \alpha E _ {\text {a l i g n}} \left(f _ {\theta}\right) \right\}, \tag {6}
$$
Normal C.	Chamfer	F-Score	IOU
mean↑	std.↓	mean↓	std.↓	mean↑	std.↓	mean↑	std.↓
SPSR* [23]	90.58	3.30	4.66	4.64	75.28	25.76	90.58	6.77
NSP* [50]	90.74	5.48	8.85	6.96	52.42	28.55	79.08	12.41
SAL [1]	86.69	9.66	29.98	31.86	25.76	22.43	60.62	18.61
IGR [19]	80.85	11.88	62.54	48.44	26.28	35.91	33.33	24.51
SIREN [40]	83.79	10.20	34.19	46.77	32.34	30.13	41.14	15.82
DiGS [6]	95.82	4.44	4.59	4.94	78.87	27.34	89.54	8.83
OSP [4]	94.73	3.94	6.80	6.61	59.12	25.82	50.84	12.56
iPSR [20]	93.22	5.26	5.95	5.97	68.42	26.36	85.01	9.34
PGR [27]	91.90	4.93	7.34	4.81	51.44	23.12	78.34	11.17
POCO+ [8]	96.41	3.53	3.62	4.21	85.42	23.13	94.40	5.55
Ours (SIREN)	96.52	3.31	3.38	3.64	88.71	18.28	89.80	6.47
Table 2: Quantitative comparison of surface reconstruction under ShapeNet [13] with 3K points. The methods marked with \*\* require normals, and the methods marked with  $^{+ + }$  are supervision based.
where  $E_{\mathrm{align}}(f_{\theta})$  is given by
$$
\sum_ {\boldsymbol {q} \in \mathcal {Q}} \beta_ {\boldsymbol {q}} L _ {\text {a l i g n}} (\boldsymbol {q}), \tag {7}
$$
and  $E_{\mathrm{old}}(f_{\theta})$  is the original loss function of existing works (the details will be discussed in the experiments). We leverage SIREN [40] without the normal version and Neural-Pull [5] in our experiment. Since the alignment effect between gradients and Hessian should be more emphasized near the underlying surface, we leverage an adaptive weighting scheme inspired by [15]:
$$
\beta_ {\boldsymbol {q}} = \exp (- \delta * | f _ {\theta} (\boldsymbol {q}) |), \tag {8}
$$
we set  $\delta = 10$  by default. At the same time, we set the default value of  $\alpha$  to 6. More details can be checked in our supplementary material.
Metrics. The indicators for comparison include normal consistency, chamfer distances, and F-Score, where normal consistency (\%, abbreviated as 'Normal C.) reflects the degree to which the normals of the reconstructed surface agree with the normals of the ground-truth surface, chamfer distance (scaled by  $10^{3}$ , using  $L_{1}$ -norm) measures the fitting tightness between the two surfaces, and F-Score (\%) indicates the harmonic mean of precision and recall (completeness). We set the default threshold of F-Score to 0.005. All meshes are uniformly scaled to  $[-0.5, 0.5]$ , and 100K points are sampled from each mesh for evaluation.
4.1 Optimization-based Surface Reconstruction
Surface Reconstruction Benchmark (SRB). The Surface Reconstruction Benchmark (SRB) [49] contains five shapes, each of which has challenging features, e.g., missing parts and rich details. The approaches for comparison include screened Poisson surface reconstruction (SPSR) [23], SIREN [40], Shape as points (SAP) [38], iPSR [20], Predictive Context Priors (PCP) [3], CAP-UDF [55] and DiGS [6]. Note that SPSR leverages normals given by the input scans. As shown in Tab. 1, our method outperforms the existing methods, as evidenced by superior results in both Chamfer distance and F-score metrics. In particular, the visual comparison presented in Fig. 3 demonstrates that our method is capable of accurately recovering the hole feature of the Anchor model, despite the absence of points on the inner wall. Furthermore, it successfully preserves the adjacent gaps of the Lord Quas model. In addition, we compared our approach with DiGS using SRB, following the evaluation settings of DiGS. Detailed performance statistics can be found in the supplementary material.
ShapeNet. The ShapeNet dataset [13] comprises a diverse collection of CAD models. We follow the splitting of [50] for the 13 categories of shapes with total of 260 shapes. Our comparison is performed using 3K points. The baseline methods include Screened Poisson Surface Reconstruction (SPSR) [23], NSP [50], SAL [1], IGR [19], SIREN [40], DiGS [6], OSP [4], iPSR [20] and PGR [27]. Note that SPSR and NSP require normal inputs. To ensure the validity of our experiments, we provide
Figure 4: Comparison of our method to other methods under ShapeNet [13], ABC [24], and Thingi10K [57].
ground-truth normals to those that require normal information. Additionally, we include the learnable baselines Shape as Points (SAP) [38] and POCO [8] for comparison and retrain them from scratch on ShapeNet with 3K points. Based on the quantitative results presented in Tab. 2 and the visual comparison in Fig. 4. Our method effectively suppresses unnecessary surface variations and adapts the implicit representation to the inherent complexity encoded by the point cloud. Furthermore, our reconstruction accuracy is comparable to supervised methods. Specifically, our method outperforms POCO in Normal Correctness, Chamfer, and F-Score, but lags slightly behind POCO in terms of IOU.