SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qijian Zhang $^{1}$ , Junhui Hou $^{1*}$ , Yohanes Yudhi Adikusuma $^{2}$ , Wenping Wang $^{3}$ , Ying He $^{2}$
$^{1}$ Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China
$^{2}$ School of Computer Science and Engineering, Nanyang Technological University, Singapore
$^{3}$ Department of Computer Science and Engineering, Texas A&M University, Texas, USA
qijizhang3-c@my.cityu.edu.hk, jh.hou@cityu.edu.hk
Yohanes.adikusuma@ntu.edu.sg, wenping@tamu.edu, yhe@ntu.edu.sg
Geodesics play a critical role in many geometry processing applications. Traditional algorithms for computing geodesics on 3D mesh models are often inefficient and slow, which make them impractical for scenarios requiring extensive querying of arbitrary point-to-point geodesics. Recently, deep implicit functions have gained popularity for 3D geometry representation, yet there is still no research on neural implicit representation of geodesics. To bridge this gap, we make the first attempt to represent geodesics using implicit learning frameworks. Specifically, we propose neural geodesic field (NeuroGF), which can be learned to encode all-pairs geodesics of a given 3D mesh model, enabling to efficiently and accurately answer queries of arbitrary point-to-point geodesic distances and paths. Evaluations on common 3D object models and real-captured scene-level meshes demonstrate our exceptional performances in terms of representation accuracy and querying efficiency. Besides, NeuroGF also provides a convenient way of jointly encoding both 3D geometry and geodesics in a unified representation. Moreover, the working mode of per-model overfitting is further extended to generalizable learning frameworks that can work on various input formats such as unstructured point clouds, which also show satisfactory performances for unseen shapes and categories. Our code and data are available at https://github.com/keeganhk/NeuroGF.
The computation of geodesic distances and paths on 3D mesh models has been extensively studied over the past few decades and plays a critical role in many geometry processing tasks, including texture mapping, shape description, correspondence, and deformation. Traditional computational geometry methods, such as discrete wavefront propagation [25, 8, 39, 41, 32] and geodesic graphs [46, 2], excel at computing exact or high-quality approximate geodesic distances and paths on meshes with arbitrary triangulation. However, these methods often suffer from computational inefficiency or require a pre-computation step. On the other hand, partial differential equation (PDE) methods [17, 10, 36] are renowned for their efficiency, flexibility and ease of implementation, but the accuracy of their results is sensitive to the quality of mesh triangulation.
It is also worth mentioning that existing geodesic algorithms are primarily designed for computing single-source all-destination (SSAD) or multiple-sources all-destination (MSAD) geodesics on meshes. While there are some works focused on all-pairs geodesic computations, such as [40, 4, 12], these methods are built upon relatively complex computational and optimization processes.
Dealing with the trade-off among accuracy, speed, and complexity for geodesic computation remains technically non-trivial.
As 3D deep learning continues to advance, integrating geodesic information into deep learning frameworks becomes increasingly important for a wide range of applications, enabling more accurate and efficient analysis and manipulation of complex 3D shapes. However, until now, none of the existing geodesic algorithms have been seamlessly integrated into deep neural networks.
In this paper, we make the first attempt to utilize deep neural networks for accurately and efficiently answering geodesic distance and path queries. Towards this goal, we propose neural geodesic fields (NeuroGFs), an overfitting paradigm for representing geodesic distances and paths of a given 3D shape in a single neural model in an implicit fashion. Our approach is highly desirable to applications that require extensive and frequent online queries of point-to-point geodesic computations. Evaluations on common 3D models demonstrate that NeuroGFs excel in solving both SSAD and point-to-point geodesic problems. They achieve exceptional performance and strike a satisfactory balance between representation accuracy, model compactness, and query efficiency. Specifically, NeuroGFs are capable of achieving computation times of less than  $1\mathrm{ms}$  for solving SSAD on 3D meshes consisting of 400K vertices, surpassing traditional methods, such as the heat method and discrete geodesic graphs, by two orders of magnitude. Besides, NeuroGFs offer the unique advantage of encoding both 3D geometry and geodesics in a unified representation. It is important to note that despite its compact size, with only 259K hyperparameters, the NeuroGF network consistently delivers high accuracy, with a relative mean error of less than  $0.5\%$ . Moreover, to facilitate more diverse applications, we further investigate generalizable learning frameworks of NeuroGFs by introducing different types of 3D shape feature encoders, which can well generalize to unseen shapes and novel categories. These results strongly demonstrate that NeuroGF is an effective and efficient tool for geodesic queries on 3D meshes, opening up exciting possibilities for a wide range of potential applications in the field of 3D deep learning. The main contributions of this work can be summarized as follows:
- We make the first attempt to explore neural implicit representations for answering geodesic queries on 3D meshes.
- In addition to geodesic distances, we model shortest paths as discrete sequences of ordered 3D points and customize a learning process of feature-guided curve deformation, which can flexibly approximate the shortest path between any pair of source and target query points.
- We construct a unified representation structure for both geometric and geodesic information of given 3D shapes. Compared with traditional computational and optimization-based approaches, the proposed NeuroGF framework is easy to implement, highly efficient for online query, and naturally enjoys the powerful parallelism of modern GPUs.
- We evaluated the performances of our NeuroGF representation paradigms in both overfitted and generalizable working modes, demonstrating the great potential and technical extensibility of our approach.
Discrete Geodesics. The existing techniques for computing geodesics on 3D surfaces can be broadly classified into six categories: computational geometry methods, graph methods, PDE methods, surface partitioning methods, and iterative methods.
Traditional computational geometry methods [25, 8, 39, 47, 41, 32] are capable of computing exact geodesic distances and paths on arbitrary meshes. However, these methods are computationally expensive, especially when dealing with large meshes. Additionally, while these methods are well-suited to solving the single-source/multiple-sources and all-destinations problem, they are not practical for computing all-pairs geodesic distances due to their high computational costs.
Graph-based methods [46, 2] exploit the local nature of the discrete geodesic problem by constructing a sparse graph, where each edge represents a short geodesic path. These paths are computed locally by using exact geodesic algorithms. By transforming the mesh-based geodesic problem into a graph-based shortest-path problem, graph-based methods can use efficient algorithms such as Dijkstra's algorithm to compute geodesic distances and paths. Furthermore, their results are guaranteed to be a true distance metric and have controllable accuracy. However, achieving high-accuracy geodesic
computation often requires significant memory for storing the graphs, as the complexity of the graph increases with the desired accuracy.
PDE methods are highly flexible and can work for a wide range of discrete domains, including point clouds, implicit functions, polygonal meshes, regular grids and even broken meshes. They solve the Eikonal equation either directly [17, 44, 5, 37] or indirectly [10, 3, 36]. PDE methods are efficient and easy to implement. However, since they compute only a first-order approximation of geodesic distances, they typically require meshes with fairly good tessellation. To obtain geodesic paths using PDE methods, the negative gradient of the computed geodesic distances must be back-traced. However, the accuracy of the resulting paths also depends on the mesh quality, as the gradient may not be continuous or may have sharp discontinuities at the mesh edges. Nevertheless, PDE methods remain popular for their efficiency and versatility in computing geodesics on various discrete domains, including those beyond polygonal meshes.
Surface partition methods aim to break down the discrete geodesic problem into small sub-problems. The geodesic tracks (GT) method [23] places evenly-spaced, source-independent Steiner points on edges. Given a source vertex, it constructs a Steiner-point graph that partitions the surface into mutually exclusive tracks. Within each triangle, the tracks form sub-regions with approximately linear change in the distance field, enabling efficient approximation of geodesic distances. While the GT method supports high-quality geodesic path and isoline tracing, it remains inefficient for point-to-point distance and path queries due to the need for constructing the Steiner point graph using a Dijkstra-like sweep. The separator-based method (SEP) [12] performs a nested bisection of the input mesh using separator curves. It approximates the distances between each mesh vertex and a small relevant subset of these curves using polynomials. Consequently, the geodesic distance between any two mesh vertices can be approximated by solving a small number of simple univariate minimization problems. The SEP method efficiently handles point-to-point distance queries but does not provide geodesic paths. Furthermore, it is limited to query points located on mesh vertices, and it may not perform well on anisotropic meshes.
Additionally, there are iterative approaches like the edge flipping method [33] and the curve shortening method [38] that specially target the computation of geodesic paths and loops. These methods typically rely on an initial path and proceed in an iterative manner. Their time complexities depend on the area of the region the path sweeps. In contrast, our method does not require an initial path and can efficiently compute geodesics with a single forward pass through the network. It provides flexibility in solving both SSAD/MSAD and point-to-point geodesics. Furthermore, our method takes advantage of the high parallelism offered by modern graphics hardware, resulting in impressive runtime performance.
3D Deep Learning with Geodesics. Geodesics provide valuable structural information that can be seamlessly integrated into deep learning frameworks for diverse tasks in shape analysis. Several existing works have leveraged geodesic information within deep learning models. For example, GCNN [22] introduces manifold-adapted convolutional operators, constructing local geodesic coordinate systems for patch extraction. MeshCNN [14] combines specialized convolution and pooling layers operating on mesh edges by leveraging intrinsic geodesic connections for performing feature extraction on triangular meshes. Geodesic-Former [28] proposes geodesic-guided transformer model for few-shot instance segmentation of 3D point clouds, where geodesic distance information is leveraged to tackle the issue of imbalanced point density. GSA [18] designs geodesic self-attention to facilitate capturing long-range geometrical dependencies of point cloud objects. In the above studies, geodesics are exploited as additional input information to boost the subsequent learning models. Differently, GeoNet [15] explicitly learns geodesics-informed representations from unstructured point clouds in a supervised manner, which can be considered as a pre-training process of the target backbone networks. More recently, GraphWalks [31] proposes a differentiable geodesic path estimator learned in a supervised manner, but its prediction accuracy is still unsatisfactory. In terms of working modes, [15, 31] targeted at learning generalizable deep models for geodesics prediction, while in this work we tend to explore both overfitted and generalizable learning frameworks. Concurrently, [49] proposes a graph-based geodesic learning network that works on mesh models, while our implicit field-based modeling paradigm shows better flexibility for various input formats such as unstructured point cloud data.
Neural Implicit Representation for 3D Shape Geometry. The working mechanisms of existing neural implicit representations can be categorized as generalizable [29, 24, 9, 30, 6, 16] and overfitted
Figure 1: Overall workflow of querying point-to-point geodesic distances and shortest paths as well as extracting signed distance fields for surface geometry representation from NeuroGFs.
paradigms [34, 35, 27, 21, 26, 45, 19]. Essentially, the former category of approaches tends to learn generalizable shape priors from a large collection of shapes at the cost of a lower-fidelity representation, while the latter category of approaches aims at reproducing a single shape with high-fidelity by accordingly overfitting a single neural network model. In this paper, we start by following the representation paradigm of the latter overfitted neural implicit models, i.e., separately overfitting a single neural network on a single target mesh model, to validate the representation power of our NeuroGFs. Then, we further make efforts to extend generalizable schemes to learn geodesics for unseen shapes and novel categories. Generally, despite the proliferation of research on deep implicit geometric information representation, to the best of our knowledge, there is still no such effort on implicit geodesics representation.
Denote by  $S$  a given geometric shape represented in the form of a 3D mesh model  $\mathcal{M}_S = (\mathcal{V},\mathcal{E},\mathcal{F})$  where  $\mathcal{V},\mathcal{E},\mathcal{F}$  are the sets of vertices, edges, and faces, respectively. In an overfitted representation paradigm, our ultimate goal is to construct a neural model  $\mathcal{N}_{\Theta}$  parameterized by network weights  $\Theta$  which are particularly optimized on the given shape  $S$  for fitting geodesic information. Specifically, given an arbitrary pair of geodesic queries, we deduce their corresponding point-to-point geodesic distance and shortest path through  $\mathcal{N}_{\Theta}$ , which can be formulated as:
$$
\{d, \mathbf {c} _ {s \rightarrow t} \} = \mathcal {N} _ {\Theta} (\mathbf {q} _ {s}; \mathbf {q} _ {t}), \tag {1}
$$
where  $\mathbf{q}_s$  and  $\mathbf{q}_t$  denote the source and target 3D points located on the underlying surface of shape  $S$ ,  $d \in \mathbb{R}^+$  is a scalar value representing the geodesic distance between the two geodesic query points, and  $\mathbf{c}_{s \to t} \in \mathbb{R}^{M \times 3}$  represents the shortest path from source  $\mathbf{q}_s$  to target  $\mathbf{q}_t$  modeled as a discrete sequence of  $M$  ordered 3D points, i.e.,  $\mathbf{c}_{s \to t} = \{\mathbf{p}_m\}_{m=0}^{M-1}$ , where the subscript  $m$  indicates the order index of each point.
In the offline training phase, we employ a supervised learning approach to overfit the neural network  $\mathcal{N}_{\Theta}$  on a specific shape  $S$ . The training is performed using a collection of ground-truth geodesics pre-computed from its corresponding 3D mesh  $\mathcal{M}$ . Once the overfitting process is completed, we can efficiently compute point-to-point geodesics for arbitrary online queries via a single forward pass through the trained network  $\mathcal{N}_{\Theta}$ .
3.2 Architectural Design
In principle, we stick to the most concise technical implementation of the overall learning pipeline, without highly introducing sophisticated modules or mechanisms, for exploring the viability and potential of our proposed NeuroGF representation paradigm. As illustrated in Figure 1, all required learnable components of  $\mathcal{N}_{\Theta}$  are simply built upon the stack of either fully-connected (FC) layers or point-wisely shared multi-layer perceptrons (MLPs). The input query points are individually lifted to high-dimensional feature vectors using FC layers, enabling subsequent manipulation of the data. For ease of representation, the feature embeddings of a pair of source and target queries,  $\mathbf{q}_s,\mathbf{q}_t\in \mathbb{R}^3$  are denoted as  $\mathbf{e}_s,\mathbf{e}_t\in \mathbb{R}^{D_e}$ . These embeddings are obtained by passing the inputs through three consecutive FC layers, progressively increasing the output channels. After obtaining the source and
target feature embeddings  $\mathbf{e}_s$  and  $\mathbf{e}_t$ , we proceed by deploying two separate learning branches to predict the geodesic distance  $d$  and the shortest path  $\mathbf{c}_{s\rightarrow t}$ .
In addition to the geodesic distance and shortest path learning branches, we also incorporate an auxiliary learning branch for surface geometry representation by predicting the corresponding signed distance field (SDF). Given a query point  $\mathbf{q}' \in \mathbb{R}^3$  and its lifted  $D_e$ -dimensional feature embedding  $\mathbf{e}'$ , we predict its signed distance  $s' \in \mathbb{R}$ . This allows us to obtain a unified representation that encodes both geodesics and geometry. It is worth emphasizing that, during training,  $\mathbf{q}'$  is sampled from the entire bounding sphere of the given shape  $S$ , while  $\mathbf{q}_s$  and  $\mathbf{q}_t$  are selected from the mesh vertex set  $\mathcal{V}$ . During testing,  $\mathbf{q}_s$  and  $\mathbf{q}_t$  can be arbitrary points on the underlying shape surface, not limited to the vertices of the given mesh.
Overall, after the initial query point embedding, our neural network architecture consists of three parallel learning branches for predicting geodesic distances, shortest paths, and signed distance fields. While the three types of outputs are closely correlated, we deliberately design the intermediate learning processes in an inter-independent manner. This means that there is no explicit interaction (such as feature fusion or propagation) among the branches. We make this design choice to avoid redundant computations in scenarios where only one specific property is required for online query.
Neural Fitting of Geodesic Distances. The geodesic distance regression is modeled as a learnable function of the "feature-space difference" between source and target queries. We compute an absolute difference vector between query embeddings  $\mathbf{e}_s$  and  $\mathbf{e}_t$ , which is further mapped to a scalar value of the geodesic distance through a stack of FC layers, which can be formulated as:
$$
d = \mathcal {B} _ {\mathrm {g d i s t}} \left(\left| \mathbf {e} _ {s} - \mathbf {e} _ {t} \right|\right), \tag {2}
$$
where  $|\ast|$  means element-wisely taking the absolute value of each vector entry, and the geodesic distance learning branch  $\mathcal{B}_{\mathrm{gdist}}: \mathbb{R}^{D_e} \to \mathbb{R}^+$  is implemented as three consecutive FC layers, where the output dimension of the last layer is set to 1.
Neural Fitting of Shortest Paths. The shortest path generation is achieved as a feature-guided curve deformation process by adapting previous "folding-style" 3D shape reconstruction decoders [43, 13], where pre-defined 2D regular grid primitives conditioned on unique shape signatures are smoothly deformed to the target 3D shape surface. Inheriting the same working mechanism, we tend to deform the straight line segment between a pair of source and target geodesic query points to the 3D curve of their shortest path, which is conditioned on a learnable deformation guidance vector exported from source and target point embeddings.
Formally, we start by sequentially sampling  $M$  discrete 3D points with uniform intervals along the directed straight line segment from source  $\mathbf{q}_s$  to target  $\mathbf{q}_t$ , as given by:
$$
\mathbf {l} _ {s \rightarrow t} = \left\{\mathbf {q} _ {s} + \left(\mathbf {q} _ {t} - \mathbf {q} _ {s}\right) / (M - 1) \cdot m \right\} _ {m = 0} ^ {M - 1}, \tag {3}
$$
where  $\mathbf{l}_{s\rightarrow t}\in \mathbb{R}^{M\times 3}$  serves as the initial line primitive to be deformed. In the online query phase, one can flexibly adjust the density of curve points according to the actual requirements by simply sampling the initial straight line segment with different uniform intervals. Next, the required curve deformation guidance vector can be deduced by:
$$
\mathbf {g} _ {s \rightarrow t} = \hbar \left(\left[ \mathbf {e} _ {s}; \mathbf {e} _ {t} \right]\right), \tag {4}
$$
where  $\mathbf{g}_{s\rightarrow t}\in \mathbb{R}^{D_e}$  uniquely determines the deformation result of the corresponding  $\mathbf{l}_{s\rightarrow t}$ ,  $[*\ast]$  denotes channel concatenation,  $\hbar :\mathbb{R}^{D_e + D_e}\to \mathbb{R}^{D_e}$  represents a learnable mapping function that can be simply implemented as a single FC layer.
To perform feature-guided curve deformation, we append  $\mathbf{g}_{s\rightarrow t}$  to each point in the initial line primitive  $\mathbf{l}_{s\rightarrow t}$ . The resulting  $M$ -by- $(D_e + 3)$  feature matrix is row-wisely embedded for shortest path generation, which can be formulated as:
$$
\mathbf {c} _ {s \rightarrow t} = \mathcal {B} _ {\text {s p a t h}} \left(\left[ \mathbf {g} _ {s \rightarrow t}; \mathbf {l} _ {s \rightarrow t} \right]\right), \tag {5}
$$
where  $\mathcal{B}_{\mathrm{spath}}$  comprises four point-wise MLPs with the output dimension of the last layer set to 3.
(Auxiliary) Neural Fitting of Signed Distances. As discussed before, there is an auxiliary learning branch that consumes the feature embedding  $\mathbf{e}^{\prime}$  of the given geometric query point  $\mathbf{q}^{\prime}$  as input and regresses a scalar value of its signed distance, which can be formulated as:
$$
s ^ {\prime} = \mathcal {B} _ {\mathrm {s d i s t}} ^ {\prime} \left(\mathbf {e} ^ {\prime}\right), \tag {6}
$$
where  $\mathcal{B}_{\mathrm{sdist}}^{\prime}:\mathbb{R}^{D_e}\to \mathbb{R}$  is implemented as three consecutive FC layers with the output dimension of the last layer set to 1.
3.3 Learning Objective
Corresponding to the three types of outputs, the overall learning objective of NeuroGF representations comprises three loss function terms for the supervision of predicted geodesic distances  $d$ , curve points of shortest paths  $\mathbf{c}_{s\rightarrow t}$ , and signed distances  $s^{\prime}$ . Additionally, we also impose two auxiliary constraint terms on the generated shortest paths to enhance the consistency of the three output shape properties. Specific mathematical formulations are given as follows.
1) Supervision of Geodesic Distances. We compute  $L_{1}$  loss between the predicted and ground-truth geodesic distances  $d$  and  $\tilde{d}$  as:
$$
\ell_ {\mathrm {g d i s t}} = \left\| d - \tilde {d} \right\| _ {1}. \tag {7}
$$
2) Supervision of Shortest Paths. We point-wisely compute  $L_{1}$  losses between the generated and ground-truth curve points of shortest paths  $\mathbf{c}_{s\rightarrow t}$  and  $\tilde{\mathbf{c}}_{s\rightarrow t}$  as:
$$
\ell_ {\text {s p a t h}} = \left\| \mathbf {c} _ {s \rightarrow t} - \tilde {\mathbf {c}} _ {s \rightarrow t} \right\| _ {1}. \tag {8}
$$
Note that since we uniformly use  $M$  curve points for approximating shortest paths during training, the raw ground-truth geodesic paths typically contain different numbers of points. For the convenience of supervision, during data pre-processing, points in the raw ground-truth geodesic paths are densely interpolated and then resampled to the desired number of  $M$  points.
3) Supervision of Signed Distances. We compute  $L_{1}$  loss between the predicted and ground-truth signed distances  $s'$  and  $\tilde{s}'$  as:
$$
\ell_ {\mathrm {s d i s t}} = \left\| s ^ {\prime} - \tilde {s} ^ {\prime} \right\| _ {1}. \tag {9}
$$