$\Delta$-PINNs: physics-informed neural networks on complex geometries
Authors: Francisco Sahli Costabal
Simone Pezzuto
Paris Perdikaris
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning
Laplace-Beltrami
physics-informed neural networks
partial differential equations

Abstract: Physics-informed neural networks (PINNs) have demonstrated promise in solving forward and inverse problems involving partial differential equations. Despite recent progress on expanding  the class of problems that can be tackled by PINNs, most of existing use-cases involve simple geometric domains. To date, there is no clear way to inform PINNs about the topology of the domain where the problem is being solved. In this work, we propose a novel positional encoding mechanism for PINNs based on the eigenfunctions of the Laplace-Beltrami operator. This technique allows to create an input space for the neural network that represents the geometry of a given object. We approximate the eigenfunctions as well as the operators involved in the partial differential equations with finite elements. We extensively test and compare the proposed methodology against traditional PINNs in complex shapes, such as a coil, a heat sink and a bunny, with different physics, such as the Eikonal equation and heat transfer. We also study the sensitivity of our method to the number of eigenfunctions used, as well as the discretization used for the eigenfunctions and the underlying operators. Our results show excellent agreement with the ground truth data in cases where traditional PINNs fail to produce a meaningful solution. We envision this new technique will expand the effectiveness of PINNs to more realistic applications.

Under review as a conference paper at ICLR 2023
A A-PINNS: PHYSICS-INFORMED NEURAL NETWORKS ON COMPLEX GEOMETRIES
A Anonymous authors
Paper under double-blind review
ABSTRACT
Physics-informed neural networks (PINNs) have demonstrated promise in solving forward and inverse problems involving partial differential equations. Despite recent progress on expanding the class of problems that can be tackled by PINNs, most of existing use-cases involve simple geometric domains. To date, there is no clear way to inform PINNs about the topology of the domain where the problem is being solved. In this work, we propose novel positional encoding mechanism for PINNs based on the eigenfunctions of the Suplace-Beltree technique allows to create an input space for the neural network that represents the geometry of a given object. We approximate the eigenfunctions as well as the operators involved in the partial differential equations with finite elements. We extensively test and compare the proposed methodology against traditional PINNs in complex shapes, such such as the Eikonal equation and heat transfer. We also study the sensitivity of our method to the number of eigenfunctions used, as well as the discretization used for the eigenfunctions and the underlying operators. Our results show excellent agreement with the ground truth data cases where traditional PINNs fail produce a meaningful solution. We envision this new technique will expand the effectiveness of PINNs to more realistic applications.
operator. This
coil, a heat sink and a bunny, with different physics,
MOTIVATION
Physics-ifformed neural networks (PINNs) Raissi al. (2019) are an exciting new tool for blending data and known physical laws in the form of differential equations. They have been successfully applied to multiple physical domains such fluid mechanics Raissi et al. (2020). solid mechanics Haghighat et al. (2021). heat transfer Cai et al.] (2021) and biomedical engineering Kissas et al. (2020): Ruiz Herrera al.) (2022). to name a few. Even though this technique can be used solve forward problems, they tend to excel when performing inverse problems. Since their inception, there have been multiple attempts to improve PINNs, in areas such as training strategies Nabian et al. (2021); Wang et al. (2022) and activation functions Jagtap et al. (2020).
Despite recent progress, many works still consider simple geometric domains to solve either forward or inverse problems, hindering the applicability of PINNs to real world problems, where the objects of study may have complicated shapes and topologies. In this area, there have been multiple attempts to introduce complexity to the input domain of the neural networks. One approach is describe the boundary of the domain with signed distance function Sukumar & Srivastaval (2022): Berg & Nyström (2018); McFall & Mahan 2009). In this way, the boundary conditions can be satisfied exactly instead of relying on penalty term in the loss function. Another approach is to use domain decomposition to model smaller but simpler domains Dagtap & Karniadakis (222)) Coordinate mappings between simple domain have also been proposed for convolutional Gao et al. (202T) and fully connected networks Li al. (2022). Nonetheless, all these works demonstrate examples of 2-dimensional shapes. When using PINNs in 3-dimensional surfaces there have been attempts to ensure that the vector fields that may appear in the partial differential equations remain tangent to the surface by introducing additional terms in the loss function Fang et al. (2021) 1): Sahli Costabal et al. 2020).. This approach works well when the surface relatively simple and smooth, specifically when the Euclidean distance of the embedding 3-D space between two points in the domain is similar to the intrinsic geodesic distance on the manifold. However, in several applications the two distances may sensibly differ, exemplified in Figure

---

Under review as a conference paper at ICLR 2023
In this work, propose to represent the coordinates of the input geometry with positional encod- ing based on the eigenfunctions of the Laplace-Beltrami Beltrami operator of the manifold, or the Laplacian in the case of bounded open domain in the Euclidean space. In this way, points that are close in the geometry will remain close in the positional encoding space. Next, the input of the neural network will be the value of finite number of the eigenfunctions evaluated a point in the domain and the oulput will remain the physical quantity that we are modeling with PINNs. Positional en- coding have shown great success in improving the capabilities of neural networks and are used in transformers Vaswani et al. (017). neural radiance fields Mildenhall et al. (2020) and PINNs Wang Beltrami eigenfunctions can be approximated numerically for any shape with the finite element method. By approximating the eigenfunctions on a mesh we lose the ability to use automatic differentiation to compute the operators of the partial differential equations within existing library codes. However, automatic dif- ferentiation could be preserved whenever the finite element library offers such capability, as many existing libraries do. We show that common operators such the gradient and the Laplacian can be efficiently computed with finite elements applied to the output of the neural network. We demon- strate the capabilities of the proposed method by testing different geometries, such coil, heat sink and a bunny and different physics, such as the Eikonal equation and heat transfer.
al. (2021a) to name a few. For our positional encoding. the Laplace-Belrrami
Laplace-Beltrami eigenfunctions
V
V10
V50
Vi00
exact
A-PINNs
PINNs
GCN
geodesic distance
18
Figure 1: Learning the Eikonal equation on a coil. Top row: the 1st 10th 50th and 100 Laplace- Beltrami eigenfunction of the geometry. Bottom row, first: the ground truth solution of the Eikonal equation, which represents the geodesic distance, and the data points used for training shown gray spheres. Second, the solution of A-PINNs, our proposed method trained with 50 eigenfunctions, Third, the traditional PINNs approximation. Last, the approximate solution of physics-ifformed graph-convolution network.
METHODS
2.1
PHYSICS-INFORMET NEURAL NETWORKS
We consider the problem where we have partial observations of an unknown function u(æ). with an input domain x E B, where B is an open and bounded domain in Rdd d = 2, or d-dimensional smooth manifold (typically surface). We also assume that u(x) satisfies partial differential equation the form_N [u, λ] = O, where A/I, >] is a potentially non-linear operator parametrized by X. The partial observations u; are located in B at positions æ;, i = .. which may therefore fall on the boundary ØB. Boundary conditions such as Neumann boundary conditions Vu n = 1, 1..... B. We proceed by approximating
It may also be enforced boundary points x/. the unknown function with a neural network u z = NN f), parametrized with trainable parameters A. In order to learn function that satisfies the observed data, the boundary conditions
2

---

Under review as a conference paper at ICLR 2023
and the model, we postulate a loss function composed of three terms
C(fu, ) (g ) 0, A) MSEEataa (( ). 0) MSEPPE (M 0). (1)
The first term favours the network learn the available data
N 2(u ((u=)(x;;))) -(xx))))²
EdtEaata(uu 2z}, N
(2)
The second term ensures that the underlying partial differential equation is approximately satisfied
R MSEPDE (0, T [((ā(ar, 0;
1
(3)
Here, we will evaluate the model at locations x. that can be generated randomly within the domain at every optimization iteration or can have predefined values. Minimizing this term will ensure that the model is approximately satisfied in the entire domain. We can also define the parameters of the operator NT. A] as trainable variables that will be learned during the optimization procedure. The final term in the loss function ensures that the boundary conditions are satisfied
B Σ (Vi(a) B
1
(SFF 2/),0)
-
(4)
We note that Dirichlet boundary conditions can be included as data in {u;, This is the basic structure of -nnfore-i neural networks that can be accommodated for large class of prob- lems Raissi et al. (2019). One issue this formulation is that the neural network that represents the solution is parametrized with vector æ Rs, which is Euclidean space. In this way, it is difficult to inform the network about the geometry of the domain or, in the case of a complex domain in Rd where the boundaries lie. This is important in the computation of the differential operators. Furthermore, points in the manifold B that are close in the Euclidean space are not necessarily close in the intrinsic distance, illustrated in Figure
2.2 EIGENFUNCTIONS
OPERATOR
In this work, we propose to encode the manifold of interest B with the eigenfunctions of the Laplace- Beltrami operator - A = V Vi-,, instead of using the Euclidean coordinates æ. These functions vx(x) can be computed by solving the eigenvalue problem: - Ar; = Aon, where A; the corre- spondent eigenvalue of u; (z)) J. In the case of domain with open boundaries, this problem is usually accompanied with either homogeneous Dirichlet or Neumann boundary conditions. In this work, will use the latter. The set of cigenvalues provide spectral signature of the domain, albeit not unique Gordon et al.] (1992). operator, endowed with homogeneous boundary conditions, infinity. When ordered by their magnitude, the first eigenvalue Ag is always zero, whereas the sec- ond one is strictly positive. We note that the physical interpretation that small eigenvalues are associated with low frequency functions, represented by the corresponding eigenfunction. The set of eigenfunctions form an orthogonal basis for the Hilbert space LB( hence every function in (B3) can be expressed as a linear combination of the eigenfunctions Evans (2010). Although these functions may be computed analytically for few shapes (e.g., equilateral triangle, rectangle, circle), their numerical approximation is possible for an arbitrary manifold. Here, we represent the domain using a mesh with nodes and elements and solve the eigenvalue problem using finite element shape functions. As such, we can obtain the discrete Laplacian matrix A and mass matrix M as
possible to show that the cigenvalues of the Suplace-Beltrane
real, non-negative, and tending to
Pel A
Thl A
/ N,N,dB,
Agj
VN. VN,dB, M.,
B
where A represents the assembly of the local element matrices, and N are the finite element shape functions. Then, solve the eigenvalue problem: Ar AMe, see Appendix A for more details.
3

---

Under review as a conference paper at ICLR 2023
2.3 PHYSICS-INFORMET NEURAL NETWORKS ON MANIFOLDS
We propose to represent the position of a point in the manifold by N eigenfunctions of the Laplace-Beltrane Beltrami operator associated with the N lowest eigenfunctions, such that v, [v) (2(;) ..., UNx (x:)). Now, instead of parametrizing the neural network with the Fuclidean position, we use: u á ( f). Since e varies smoothly in the manifold, intrinsically close points will be also close in the space V.
Another interpretation of the proposed method is that we are constructing features or positional encodings for points in the manifold. Then, the approximation of the solution û can be seen as a non- linear combination of these features. our case, the eigenfunctions represent features of increasing frequency, seen in Figure I A similar concept has been proposed for Cartesian domains with Fourier feature mappings Tancik et al. Wang et al. (202 Here, the Cartesian coordinates transformed with sine and cosine functions, which have random frequencies that can be tuned for the problem of interest. The main difference is that in our method the frequencies are given by the eigenfunctions, and are defined in the manifold where want to solve the problem. We can select different eigenfunctions depending on the nature of problem, where high frequency problems might need: a larger amount of eigenfunctions.
The main challenge of our methodology is that the differential operators cannot be computed directly from the neural network via automatic differentiation. Since the eigenfunctions cannot be computed in closed form for an arbitrary manifold, we have a map between æ and v that is defined using finite elements. The only way to differentiate the eigenfunctions with respect to the position æ is again by using finite elements. Therefore, we use this numerical approximation of the differential operators that are required to compute the PDE operator [|:]. Commonly used operators, such as the gradient and the Laplacian can be easily computed with linear finite elements. We also note that one could employ the chain rule to compute, for example, the gradient, such that Vz = V Vrv. The term V û can be computed with automatic differentiation since only depends on the neural network, and the second term Vxy needs to be computed with finite elements. We found small advantages with this approach, and it becomes cumbersome for second order operators. Instead, we take the simplest route: we use the neural network predictions at the nodal locations and use finite elements to evaluate the operator. For instance, to compute V at the centroid of linear triangular element, we need to evaluate the neural network û; at every U;. which corresponds to the selected eigenfunctions evaluated at the nodal locations x,, Then, the gradient can be computed as V Re B [ū, üuz úg]" T where the matrix B defines the gradient operator for a linear triangular element in particular, see Appendix [A] One of the advantages of this approach is that the gradient is guaranteed to be tangent to the manifold, which is not the case for the traditional physics-informed neural network formulation [Sahli Costabal et al. (2020). Similarly,
compute the Laplacian Ad; at any given nodal location by predicting ú; at all the neighboring nodes, including the node of interest. Then, we can use the definition of the discrete Laplacian presented in equation 5 and approximate Aú, RJ A:jūy.
The advantage of being able to compute the operators locally the element node level is that we still use mini-batch techniques to estimate the loss function and its gradient. In this way, we avoid predicting the operator for the entire manifold, which could slow the training process for large meshes. In general, we will randomly select nodes or elements for each training iteration to evaluate the loss term MSEPDE defined in equation
3 NUMERICAL EXPERIMENTS
3.1 EIKONAL EQUATION ON COII.
In our first numerical experiment we compute geodesic distances using the Fikonal equation
Vu Vu = 1, u(xs)) = 0,
This equation has multiple applications, for instance, in cardiac electrophysiology Sahli Costabal et al. 22200) and seismology Smith et al. (2020), because it can be used to model the arrival times of a traveling wave. In the particular form shown in equation66, the solution u ()) can be interpreted as the geodesic distance from the point a6 any point the manifold.
4

---

Under review as a conference paper at ICLR 2023
In this example, we will solve this equation the surface of coil. which is generated by an helix with 30 mm of diameter and 12 mm per revolution of pitch. This curve is extruded with circular cross-section with 5 mm of radius. The resulting geometry can be seen in Figure IT which is discretized with 1546 points and 3044 triangles. We can generate a ground truth solution of the Fikonal equation by randomly selecting point the mesh for as and then applying the exact geodesic algorithm |Mtchhell et al. (1987) as implemented in libigl Jacobson et al. (2018). We randomly select 40 points of this solution use data ui. equation loss defined in 3 takes the form
In this case, the partial differential
(V(((BBB) (BAU.) (Brū.)- MSErDE () = R E
1
where B is the approximation of the gradient for particular triangle Here, the vector úf the prediction of the neural network at the nodes of the triangle r. which depend on the train- able parameters 0. We use 50 eigenfuctions of the Aaplace-Beltrami operator imput of the neural network, which is then followed by a single hidden layer with 100 neurons and hyperbolic tan- gent activations. We train the neural network with the ADAM optimizer Kingma & Ba (2014) for 40,000 iterations with default parameters. We use a batch size of 10 samples to evaluate both the M S Edsta and M MEEE term. For comparison, we also train the same neural network by removing the physics, that is not considering the partial differential equation loss shown equation| 7 Addition- ally, we train regular ahysics-informed neural network, where we use the Cartesian coordinates as the input of the neural network. We implemented the Eikonal equation in the MS computing Vu through automatic differentiation. This network has 2 hidden layers with and 100 neurons, respectively. We also tested against a physics detailed in Appendix [..]] with matched number of parameters. We train all methods for the same number of iterations and learning rate.
graph neural network Gao et al. (2022),
A-PINN
D-PIN
RINN
GCN
y
test points train points
ground Buth
ground 9uth
18
grouns
16
ground Both
Figure 2: Accuracy the coil example with the Eikonal equation. Correlation between predicted and ground truth values of geodesic distance for A-PINNs (first), A-PINNs trained without the loss functions that includes the Eikonal equation (second), traditional PINNs (third) and a graph convolutional network (last).
Figures T and 2 show the results of these experiments. For A-PINNs and the graph neural network we select the best out of 5 random initializations (see Appendix B.II. In the bottom row of Figure []] see the exact solution to the geodesic problem, the physics-informed neural network learned so- lution, the graph convolutional network and the proposed method. On one hand, we see that PINNs fail to capture the fast variations in the Cartesian space and then predict nearly constant inter- mediate value. On the other hand, we see that APPINNS can represent the exact solution with high accuracy. The graph convolutional network performs better than PINNs but struggle correctly learn the level-sets of the solution. This is reflected in Figure 2 where we show the correlation between the predicted and ground truth data for train and test points. We see that A-PINN achieves a high correlation in both training and testing data, while PINNs fail to fit both. When the physics are removed from A-PINNs, the method can only overfit the training data with a poor performance in the test data. Qualitatively, the normalized mean squared errors on the combined train and test dataset correspond to 3. 10-3 for APIINS, 0.99 for for A-PINNS without physics, 1.01 for PINNs, and 2.95 10-2 These results show more than 2 orders of magnitude improvement for the proposed method over the traditional formulation, at the expense of small pre- pre-processing step of computing the eigenfunctions of the Laplace-Beltrami Beltrami operator for that geometry (see Appendix E) is worth noting that generating the solution of the Eikonal equation with the exact geodesic algo- rithm does not lead to zero M FYDE loss defined in equation due to the underlying numerical
5

---

Under review as a conference paper at ICLR 2023
discretization. If evaluate this term using all the points in the mesh obtain residual value of 2.11. -00 Hence, the proposed method works even in the presence of imperfect physics, when the model is not exactly satisfied.
3.2 HEAT TRANSFER HEAT SINK
In this experiment, we simulate heat sink with convection losses in 2 dimensions and steady state. This can be expressed as the following boundary value problem
Au(Φ) Vu(x)= n 0.1u(x), a E dBe; Vu(x) n = 0, dB
0, x B; u(x) 1, dBD.
SDF-PINN
Figure 3: Learning the temperature distribution of a heat sink from boundary measurements. First, the boundary conditions and the finite element mesh used to create the solution. Second, the ground truth, and in the learned solution of (third) A-PINNs, (forth) signed distance function PINNs and (last) traditional PINNs. The data points are shown on the ground truth panel in black.
We include three different types of boundary conditions: Dirichlet in dBp, Robin in dBc and Neu- mann yB The Dirichlet boundary condition is to represent another body with constant, higher than ambient, temperature, which the heat sink B is attempting condition represents a convective flux, which is proportional to the temperature u. The Neumann boundary condition is used represent the symmetry of the domain. The geometry and location of the boundaries are shown in Figure 3 This example could be interpreted as a simplified model for the heat sink that is typically placed above a central processing unit (CPU) of a computer. To generate a ground truth solution for this problem we performed finite element simulations using the FEniCS library Logg et al.] (2012) on a mesh with 1289 nodes and 2183 linear triangular elements. In this experiment we will attempt to solve the inverse problem of learning the temperature inside the domain B from proposed A PINNs with and without physics, well conventional PINNs to solve this problem. In all cases, we will not inform the type of boundary conditions applied, and instead we only provide temperature measurements u, at the boundary dB. Thus, will not consider the term MS E,, in the loss function defined in equation I] In practice, this is important because the true form of the convective fluxes is often unknown empirically approximated. For A-PINNs we will approximate the Laplacian operator Au with triangular finite elements using the definition of discrete Laplacian A shown in equation 5 Then, we can define the loss term
dissipate. The Robin boundary
information of the temperature the boundary dB. We will again use the
nt
MSEPDE = Σ
Σ JEN(1)
where the index i represents the nodes of the mesh inside the domain B, j neighbor nodes of the node i, including itself. The term ú, represents the output of the neural net- work at the nodal locations j that depends on trainable parameters 0. For traditional PINNs, we use automatic differentiation to compute Åû and construct the loss term A EPDE- We also compare to physics-informed neural network that exactly imposes the boundary data using signed distance functions Sukumar Srivastaval (2022), which can only work in the case of a flat surface or 3D solid, detailed in Appendix B.2 We use 50 eigenfunctions represent the domain B for APPINNs, and hidden layers of 100 neurons for all cases. We train the network with ADAM Aingaann & Ba 2014) for 50,000 iterations using batch size of 30 samples evaluate each of the terms of the loss
N()) represents the

---

Under review as a conference paper at ICLR 2023
function. We use 393 values of located the boundaries as training data, shown in Figure We note that we use the same operator, the Laplacian. for the PDE that governs this problem and to construct the imput of the neural network. However, the eigenfunctions are computed with homoge- neous Neumann boundary conditions, which guarantees that the eigenfunctions vy are not solutions to the boundary value problem shown in since they have different boundary conditions.
The results of this experiment are summarized in Figures and First, we see that the geometry of the heat sink gready influences the solution, as shown Figure E The narrow fins have more temperature reduction than the wide ones. Qualitatively, we observe that A-PINN produces the closest temperature distribution to the ground truth solution. The method is able produce different temperature distributions for the different fins, which may be close in Euclidean distance, but are far apart in the intrinsic distance. The signed distance function PINNs produce a reasonable approxima- tion everywhere except for the narrow fins, where there is an artificial curvature. Finally, traditional PINNs tend to favor satisfying the MSEPpE. producing effectively nearly linear solution for which trivially A O. However, in this process the data provided at the boundaries is not fitted. Also, this method is not able to produce different temperature profiles for the different fins, corre- lating points that close only in the cartesian space. Quantitatively, mean squared error on the combined train and test sets drops 2 orders of magnitude for A-PINNs (1.62 (1.62-10-3) when compared to traditional PINNs (1.48 10-1). The signed distance function PINN produces an error slightly lower than A-PINNs of 1.45 10-3, through the exact imposition of the data as boundary conditions. Also in Figure 4 we can see that for PINNs, the correlations reflect exactly structure of the different fins. With this example, we have shown a problem with simple physics, linear inverse problem, with some degree of complexity in the geometry of the domain may cause PINNs to fail while A-PINN may successfully work.
see that the normalized
A- PINN
SDF-PINN
PINN
1.0
y=
test points train points
predidted
0.4
ground truth
1.0
0.4
ground truth
1.0
0.4
ground truth
1.0
Figure 4: Accuracy the heat sink example. Correlation between predicted and ground truth values of temperature for A-PINN (left), signed distance function PINNs (center) and traditional PINNs
(right).
3.3 PIYYSICS-INFOREDD DEEP OPERATOR NETWORK
We finalize this section by presenting a deep operator network Lu et al. 2019); Wang et al. (202 bb that learns the geodesic distance between two points on a manifold, This type of networks learn to transform functions instead of single points. They are composed by branch network that encodes the input functions and a trunk net that encodes the input position. The input function in our case cor- responds to boundary condition of the Fikonal equation, shown in equation 6 and the input coordi- nate corresponds to points in the manifold. We note two propertics of the geodesic distance function d(-,)) :): first, it is symmetric, d(æ), x2) = d(I2, xi). since the distance from æi to 22 is the same as the distance from Z2 to æ1. Second, the distance from a point to itself is zero, that is d(x x)) = for every point in the manifold. As in the rest of the experiments,
will represent points in the operator v; = [vi (æ:;), U2(2)), uN (æ:))- With this is mind, we propose the following deep operator network architecture d to approximate d
manifold by the eigenfunctions of the Laplace-Beltranl
M(
NN 0r) NN 0n).
(10)
(1 (1- 01 02
d( v2) = 2
(d(v), U2) + d(v2, v1)),
(11)
1 122
7

---

Under review as a conference paper at ICLR 2023
where 0y and 0g represent the trainable parameters of the trunk and branch networks respectively. This architecture automatically satisfies the symmetry and zero distance properties of the geodesic distance function. The network will be informed by informed by the Eikonal equation, which should be satisfied at any point of the domain. In this case, we use the chain rule to compute the gradient Va, d Vv,d p. Va," V. The gradient of the eigenfuctions with respect to the input positions Va,, computed with finite elements and gradient of the predicted distance function with respect the input eigenfunctions Ve, d is computed with automatic differentiation. The loss function to train the neural network still consists of the one detailed in equation [ but in this case, we do not have boundary data and the partial differential equation loss takes the form:
1 R Σ (ViVo.d var
MSEPDE 0r)
(12)
which is evaluated at the R triangle centroids of the mesh using mini-batch strategy.
yound yuth
predicted
gendwiic distnnca
Figure 5: Geodesics on a bunny. We learn the geodesic distance between any two points in the bunny surface. We randomly select points and show the ground truth and predicted geodesic distance between these points and the rest of the surface, and geodesic paths between 300 randomly selected points and these origin points.
We test the proposed methodology in the Stanford bunny Turk & Levoy (994). obtained from the software MeshMixer. This mesh has 5,550 vertices and 10,996 triangles, which results in 15,404,025 pairs of distance to be learned. We randomly select 50,000 pairs (0.32% of the total pairs) as training data, for which we compute the exact geodesic distance Mitchell al. (987).. We the 50 eigenfunctions associated with the lowest eigenvalues of the Laplace-Beltran operator to define v(x;). Both for the trunk and branch network, train for 50,000 iterations with ADAM
10 hidden layers of 200 neurons each. We ADAM|Kingma Kingma & Ba (2014) with exponential decay of the learning rate. To demonstrate the potential of this method, we also compute geodesic paths between æ and 22 by solving the following initial value problem: Φ = -Vad(a ( Ø2), x(0) = II. Some representative results of this method are shown in Figure 5 where we display the geodesic distance from 5 randomly selected points to the rest of the manifold for our method and the ground truth. We also show geodesic paths between 300 randomly selected points to each the origin points. We see qualitatively very good agreement in both for the predicted geodesic distances and geodesic paths. Furthermore, the geodesic path are actually directed to the origin point, which shows that the learned geodesic distances continuously grow out of the origin point. Quantitatively, we evaluate the test error using 10,000 randomly selected pairs and obtained a mean absolute error normalized by the maximum geodesic distance of 0.57%.
8

---

Under review as a conference paper at ICLR 2023
4 DISCUSSION
In this work, we present a novel method to use physics- -nfoorse neural networks in complex ge- ometries by representing manifolds by the eigenfunctions of the Laplace-Beile Beltrami operator. Given a mesh representing the geometry, we can compute these functions with standard finite elements. We show that this method outperforms the original formulation of PINNs, especially in cases where there points in the input domain are close in Euclidean distance but far in the intrinsic distance. We test our methodology by solving an inverse problem with the Eikonal equation and the heat equation, forward Poisson problem, and operator learning problem for inferring the geodesic distance between two points on a complex manifold. We show better performance than a physics- informed graph convolutional network Gao al. (002)) and comparable performance PINNs with signed distance functions, which is specially tailored for 2D problems with boundary data SSkumar & Srivastava (2022). The Laplace-Beltrami machine learning, such as point signatures in shapes Rustamov et al. (2007); [un et al. (2009), shape descriptors ['ang et al. (2015), for neural fields Koestler et al. (2022) and to approximate kernels on manifolds for Gaussian process regression Borovitskiy et al. (2020) and classification Gander et al.
Beltrami eigenfunctions have been used in other applications of
(022). To our knowledge this is the first time they have been used in the context of thysics-informe
neural networks.
The proposed method can be seen as an extension of positional encoding methods Vaswani et al. (2017), such as Fourier features Tancik et al. (2020), to arbitrary geometries. Furthermore, com- bining the Laplacian eigenfunctions in R we can recover a series cosine functions when using Neumann boundary conditions and sine functions when using Dirichlet boundary conditions, which correspond to the encodings proposed in Vaswani et al. 2017). These techniques have significantly improved the performance of neural networks, including PINNs Wang et al. (2021a). Nonetheless, we observed in our experiments that only using the eigenfunctions from the Laplacian with Neu- mann boundary conditions was enough to obtain accurate results. We that including the physics of the problem greatly improves the results. Furthermore, we tested our method in a pure regres- sion tasks in D and compared it Gaussian process regression which can be naturally extended to work on manifolds. We see that neural networks with the eigenfunctions as input tend to overfit the data and the results depend heavily on the number of eigenfunctions used. This confirms the regularization effect of the physics, explaining the excellent performance in problems shown here. We also note that Gaussian process are an effective tool for regression on manifolds |Borovitskiy et al. (2020), but extending them to include physical knowledge expressed as non-linear constraints is cumbersome, as it requires approximate inference techniques and will inevitably suffer from the classical poor scaling of Gaussian processes Raissi et al. (2017).
In our experiment for the Poisson equation where we can directly compare to traditional PINNs (see Appendix C. we observe that is possible to obtain similar levels of errors, showing that our method can also be used for simple domains. Furthermore, we did not see a decrease in performance when we compared the numerically obtained eigenfunctions versus the exact ones. The number of eigenfunctions used did alter the accuracy of our method. This quantity is a hyper-parameter that needs to be tuned, for all common positional encodings (Tancik al. (2020). We also observed that the size of the discretization to compute the operators also has an impact on the accuracy in a mon-monotonic fashion. The way that changing the discretization affects the loss landscape and the training dynamics requires further studies which are beyond the scope of this work.
Our current approach has some limitations. We numerically approximate the operators involved in the partial differential equations using finite elements. Although the gradient and the Laplacian, which are the most commonly used operators, can be approximated with linear elements, more sophisticated elements might be needed for other operators. We also need to compute the eigen- functions of the Laplace-Beltee Beltrami operator as a pre-processing step. Nonetheless, as we show in Appendix E this task takes just few seconds even for complex meshes with more than 10a nodes.
As future work, we plan to expand this methodology to work on 3D solid geometries, for which we can compute the eigenfunctions of the Laplace operator. There are multiple opportunities in this area, as most partial differential equations are actually solved in this type of domains. We would also like to extend this approach to time dependent problems, where we include time as an additional feature of the neural network. We envision that A-PINNs will enlarge the possibilities for applications of chysics-informen neural networks in more complex and realistic geometries.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ivo Babuška and John Osborn. Figenvalue problems. In PG Ciarlet and JL Lions (eds.), Handbook of Numerical Analysis, volume 2, chapter 5. Elsevier, 1991.
Jens Berg and Kaj Nyström. A unified deep artificial neural network approach to partial differential equations in complex geometries. Neurocomputing, 117:28-41, 2018.
Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, et al. Matérn gaussian processes on riemannian manifolds. Advances in Neural Information Processing Systems, 33:12426-12437, 2020.
Annalisa Buffa, Giancarlo Sangalli, and Rafael Vázquez. Isogeometric analysis in clectromagnetics: B-splines approximation. Computer Methods in Applied Mechanics and Engineering, 197217) 1143 1143-1152,2011 2010.
Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris. and George Em Karniadakis. Physics- informed neural networks for heat transfer problems. Journal of Heat Transfer, 143(6), 2021.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29, 2016.
Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
Yi Fang, Jin Xie, Guoxian Dai, Meng Wang, Fan Zhu, Tiantian Xu, and Edward Wong. 3d deep shape descriptor. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 2319-2328, 2015.
Zhiwei Fang, Justin Zhang, and Xiu Yang. physics-informer neural network framework for partial differential equations on 3d surfaces: Time-dependent problems. arXiv preprint arXiv:2103. 13878, 2021.
Lia Gander, Simone Pezzuto, Ali Gharaviri. Rolf Krause, Paris Perdikaris, and Francisco Sahli Costabal. Fast characterization of inducible regions of atrial fibrillation models with multi- fidelity gaussian process classification. Frontiers Physiology, pp. 260, 2022.
Han Gao, Luning Sun, and Jian-Xun Wang. Phygeonet: physics-informed geometry geometry-adaptive convo- lutional neural networks for solving parameterized steady-state pdes on irregular domain. Journal of Computational Physics, 428: 10079, 2021.
Han Gao, Matthew Zahr, and Xun Wang. -hysics-informed graph neural galerkin networks:
A unified framework for solving pde-governed forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 390: 14502, 2022.
Carolyn Gordon, David Webb, and Scott Wolpert. One cannot hear the shape of a drum. Bulletin of the American Mathematical Society, 27(1):134-138 1992.
Ehsan Haghighat, Maziar Raissi, Adrian Moure, Hector Gomez, and Ruben Juanes. A physics- informed deep learning framework for inversion and surrogate modeling in solid mechanics. Computer Methods in Applied Mechanics and Engineering, 379:113741, 2021.
Vicente Hernandez, Jose E. Roman, and Vicente Vidal. SLEPc: A scalable and flexible toolkit for the solution of eigenvalue problems. ACM Trans. Math. Software, 31(3):351-36,, 2005.
Thomas JR Hughes. The finite element method: linear static and dynamic finite element analysis. Courier Corporation, 2012.
Alec Jacobson, Daniele Panozzo, al. libigl: A simple C++ geometry processing library, 2018. https://ibigll.iitub..
Ameya Jagtap and George Em Kariiadakis. Extended thysics-informen neural networks (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlin- ear partial differential equations. Communications in Computational Physics, 28(5):2002-2044 2020.
10

---

Under review as a conference paper at ICLR 2023
Ameya D Jagtap, Kenji Kawaguchi, and George Em Kariaaaaii. Adaptive activation functions accelerate convergence in deep and physics-informa neural networks. Journal of Computational Physics, 1004:091330 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Georgios Kissas, Yibo Yang. Fileen Hwuang, Walter R Witschey, John A Detre, and Paris Perdikaris. Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non- invasive 4d flow mri data using physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 358: 12623, 2020.
Lukas Koestler, Daniel Grittner, Michael Moeller, Daniel Cremers, and Zorah I Lähner. Intrinsic neural fields: Learning functions on manifolds. arXiv preprint arXiv:2203.07967 2022.
Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207005209, 2022.
Anders Logg, Kent-Andre Mardal, and Garth Wells. Automated solution of differential equations by the finite element method: The FEniCS book, volume 84. Springer Science & Business Media, 2012.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for iden- tifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv: /910.0999.03 093,, 2019.
Kevin Stanley McFall and James Robert Mahan. Artificial neural network method for solution of boundary value problems with exact satisfaction of arbitrary boundary conditions. IEEE Trans- actions on Neural Networks, 20(8):1221-1233, 2009.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Joseph SB Mitchell, David M Mount, and Christos H Papadimitriou. The discrete geodesic problem. SIAM Journal on Computing, 16(4):647-668, 1987.
Mohammad Amin Nabian, Rini Jasmine Gladstone, and Hadi Meidani. Efficient training of physics- informed neural networks via importance sampling. Computer-Aided Civil and Infrastructure Engineering, 36(8):662-97) 2021.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Machine learning of linear differential equations using gaussian processes. Journal of Computational Physics, 348:683-699 6933, 2017.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, (648)) 2020.
Carlos Ruiz Herrera, Thomas Grandits, Gernot Plank, Paris Perdikaris, Francisco Sahli Costabal, and Simone Pezzuto. Physics-informet neural networks to learn cardiac fiber orientation from multiple electroanatomical maps. Engineering with Computers, pp. 1-17, 2022.
Raif M Rustamov al. Laplace-beltrami eigenfunctions for deformation invariant shape represen- tation. In Symposium on geometry processing, volume 257, pp. 225-233, 2007.
Francisco Sahli Costabal, Yibo Yang, Paris Perdikaris, Daniel E Hurtado, and Ellen Kuhl. Physics- informed neural networks for cardiac activation mapping. Frontiers in Physics, 8:42, 2020.
Jonathan D Smith, Kamyar Azizzadenesheli, and Zachary E Ross. Eikonet: Solving the eikonal equation with deep neural networks. IEEE Transactions on Geoscience and Remote Sensing, 59 (125:1665511066, 2020.
11

---

Under review as a conference paper at ICLR 2023
N Sukumar and Ankit Srivastava. Exact imposition of boundary conditions with distance func- tions in physics-inte informed deep neural networks. Computer Methods Applied Mechanics and Engineering, 389:114333, 2022.
Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer forum, volume 28, pp. 1383-1392, Wiley Online Library, 2009.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Greg Turk and Marc Levoy. Zippered polygon meshes from range images. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, pp. 311-318, 1994.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature net- works: From regression to solving multi-scale pdes with physics-informed neural networks. Com- puter Methods in Applied Mechanics and Engineering, 384: 13938, 2021a.
Sifan Wang. Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric par- tial differential equations with physics-informed deeponets. Science advances, 7(40):eabi8605, 2021b.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail train: A neural tangent kernel perspective. Journal of Computational Physics, 449:1 107668 2022.
Max Welling and Thomas N Kipf. Semi-supervised classification with graph convolutional net- works. In J. International Conference on Learning Representations (ICLR 2017), 2016.
12

---

Under review as a conference paper at ICLR 2023
A THE USE OF FINITE ELEMENTS IN A-PINNS
In this section we provide a brief description of the finite element method in the context of the proposed methodology. In finite elements, we approximate the discretization of function u(x;) a mesh with nodes and elements as the sum of functions with compact support Hughes (2012)
u(x)
Σ Na(x)),,
(13)
where N; represent the so-called shape functions that are associated with node of the mesh and u; is the value of the function at that node. In this work we only consider Lagrangian shape functions, that is those that satisfy the following delta property
[1. i= N.(x,) = 10 i i
(14)
These functions also satisfy the partition of unity, such that 2: Mlasa N₂(x) 1 Vz. An example of the shapes functions in ID would be the hat (also called triangular) function defining linear finite element. For a surface in 3D, the linear case corresponds to a piece-wise linear function that is in node and 0 in all neighboring nodes. In our case, we would like compute the eigenfunctions Laplace- Beltrami Laplace-Beltrami operator using finite elements, which corresponds to solving the following boundary value problem
Au(( = Xu(x), a E B, Vu 0, dB.
(15)
(16)
We can rewrite this equation by multiplying by test function wxx) and integrating in the entire domain
w w(Au + Xu)dB O. -f
(17)
Using Green's 's first identity, we arrive at the variational formulation of the problem
/ (Vw- Vu Awr)dB -
w Vu ndB = 0,
(18)
J
for all possible choices of the test function in some function space. The second term vanishes due to the boundary conditions of the problem. Typical choice for the space is the Sobolev space H H²(B) for Neumann boundary conditions and H)) for Dirichlet conditions. The space H²(B) contains the square-integrable functions (in the sense of Lebesgue) with square-integrable gradient.
The Galerkin approach for the numerical approximation of the eigenvalue problem consists in selecting a finite dimensional subspace of H ( Here, we consider the classic finite element space of piecewise linear functions. Now, we approximate both functions with finite elements Replacing, we obtain
w(a (x) ~ Ex Ni( u(x)= = E Vizako
J VNi(x)w; Σ (N -
JB Σ
N((w) Σ dy 0.
(19)
and since the nodal values Ujj do not depend on æ we can rewrite
Tancee
Thascze
(L VN.Co) (N,(2) - > J ( 0.
Σ w;
Σ Uj
(20)
Since this equation must hold for any test function, thus any values of wi, we arrive system of equations
Reade:
Σu, uj (f(VN())) VN₁(2) - (2) 0.
(21)
This system can be written in matrix form by defining
7el
nel
A / N,N,dB,
Aij A
VN. VN,dB, Mij a=1
(22)
13

---

Under review as a conference paper at ICLR 2023
where A represents the assembly of the local element matrices. Then, solve the eigenvalue problem: Au- = XMu to obtain the eigenfunctions, where u = [ui..... "yy..., Unnstal
It is possible show that the discrete set of cigenfunctions, obtained by solving the problem Au XMu, converges to the analytical eigenfunctions as the mesh size goes to zero (under some uniformity assumptions in the way the limit is attained.) For linear basis functions and convex domains, the eigenvalues converge quadratically with respect to the mesh size [Babškka & Osborn (199I). The convergence of the eigenfunctions is linear in the energy norm and quadratic in the L- norm. For non-convex domains, the convergence rate might be limited by the regularity of the eigenfunctions (e.g., the L-shaped domain.) Furthermore, the geometrical error in approximating the domain or the surface with a piece-wise linear complex may introduce additional approximation errors, usually in regions where the curvature is large. this context, isoparametric or isogeometric finite elements are more indicated Bufff et al. (2010).
In this work, we use linear triangular elements, for which we will provide further detail. A important part of the proposed methodology is the computation of the gradient Vu. In linear finite elements, the gradient is constant inside the element. We compute the gradient in the element as (Vu) B u where
Ni,x
N2
N3,2
B =
(23)
Ni,
N2,
N3.91
here Ni, represents the shape function associated with node of the element in the direction j. The vector u² = [u,, u2 ua] is comprised of the u. function values at the nodal locations of the triangle. Here we note that {ar, y) is actually local coordinate system of the triangle that will depend on its orientation. In general, the gradient with respect to the global coordinates X = {X, Y, 2) can be obtained by changing the local global coordinate systems with rotation matrix R" (V "") = R B" However, in all the applications of this manuscript this rotation matrix cancels. In the Eikonal equation, we can write the norm of the gradient as V B R* but for any rotation matrix its inverse corresponds to its transpose, such that R I. From a physical perspective this makes sense: the norm system. The rotation matrix involved in the operation VN, VN, that defines the Laplacian also cancels following the same argument. We end this section by providing the specific form of the B matrix for triangular element. To make the description more concise we introduce the following notation: Fab = Ha Fn and yab = yo yb. where I; and yE refer to the local coordinates of the node i in the triangle. We first introduce the area of the triangle A (F138/23 1228/1)/2. Then, we can define
vector is the same regardless the coordinate
B
1
y23
y31
/12
(24)
2.
3322
221
B COMPARISON TO OTHER METHODS
Other than traditional PINNs, compare to two other methods that are designed to work on com- plex geometries: a graph convolutional network Gao et al. (2022) and a physics-informen neural network that exactly imposes boundary conditions with signed distance functions (Sukumar & Sri-
vastava 2022).
B.1 GRAPH CONVOLUTIONAL NEURAL NETWORK
The authors of this approach Gao et al. 2022) propose to use graph convolutional network to ap- proximate the solution of a partial differential equation. The input of this network will be locations of the nodes of the mesh. In general, convolution in a graph can be expressed as 90 = U Ugg UT, where 90 a filter in Fourier domain and U are the eigenvectors of the graph Laplacian Welling & operator, which we use here, in the
Kipf (2016). This operator is different from Laplace-Beltran sense that only encodes topological information about neighbouring nodes, but it completely lacks geometrical information. It is clear from the application of the graph convolution that it represents linear combination of features. The result is then passed through a non non-linear activation function and the convolution operation can be applied multiple times to create deep graph conoolhcoolltiioall net- work. To avoid performing the eigen-dees segeedddcomposition of the graph Laplacian, it has been proposed to Chebyshev polynomials Defferrard al. (2016). The degree of the polynomial K represents the
14

---

Under review as a conference paper at ICLR 2023
maximum separation of nodes that will be reached by the convolution. The authors set K 10. Our approach is fundamentally different from the graph convolution, even though they are both related to the Laplacian. The graph convolution acts locally and propagates information neighboring nodes Welling & Kipf) (2016). Similarly, partial differential equations (PDEs) consider local differential operators, certified by their sparsity when discretized with finite elements. However, the solution operator of such PDEs is global, and hence its finite dimensional counterpart is dense. Another way see this is that local changes in the boundary conditions of the PDE yield global changes in the solution. For this reason, graph convolutional networks require either high order filters multiple layers propagate information across all nodes in the graph. In our approach, we encode global topological and geometric information about the domain in the Laplace-Beltrami Beltrami eigenfunctions. These functions provide suitable basis to approximate the solution of partial differential equations, which we non-linearly combine with fully connected neural network.
In Gao et al. (0022). the authors also compute the operators used in the underlying partial differential equations using the finite element method, which makes it perfect candidate for comparison against our method, We compare both methods on the coil example with the Eikonal equation, which is detailed in Section 3.1 To approximately match the number of parameters used in A-PINN, use hidden layers with 16 filters each, with ReLU activations for the Chebyshev convolutions with K 10. We train both methods for the same number of iterations and verified that both methods reached stable value of the loss. It is worth mentioning that training the graph convolutional network takes significantly longer than A-PINNs because it does not allow for a mini-batch implementation: the output at one location potentially depends on every input node of the graph. In the coil example, with only 1,546 nodes, A-PINNS is 9x faster to train than the graph convolutional network. This speed-up would only grow in more complex meshes, as the size of the mini-bated can be fixed for A-PINNs and will grow for the graph convolutional network.
101
A-PINN
GCN
mean squared error
10-3
number of data points
120
2000
number of parameters
10500
Figure 6: Ablation study comparing to graph convolutional networks. We assess the test in the coil example varying the amount of data points provided with fixed architecture (left) and vary the amount of parameters each methodology when training with 60 data points. We repeat each run times with different random initialization of the parameters and show the minimum and maximum error with filled band and the median with a solid line.
We finalize this comparison with an ablation study, by comparing the performance the two meth- ods in the coil example. For all cases, we repeat the experiment with 5 different random initialization of the parameters. We compare the effect adding more data points (Figure 5 left) for fixed archi- tectures used in Section 3. We see that for all cases, A-PINNs outperforms the graph convolutional network, and we see a decrease in error as we add more data points. This behavior is lacking for the graph convolutional network, which increases the error when 120 data points used. We also assess the effect of the number of parameters in both methods when training with 60 data points (Figure right). For A-PINNs, we use only one hidden layer and 50 eigenfunctions, and vary the
15

---

Under review as a conference paper at ICLR 2023
amount of neurons in the hidden layer. For the graph convolutional networks, use 3 hidden layers and vary the size of these layers. In A-PINNs, we see very little influence in the error if we change see that is very sensitive to the archi- tecture choice non-trivial way. We also see than in general, the graph convolutional network is much more sensitive to changes in initialization than A-PINNS.
the number of parameters. For the graph neural network,
B.2 EXACT IMPOSITION OF BOUNDARY CONDITIONS WITH SIGNED DISTANCE FUNCTIONS
In Sukumar & Srivastava 2022), the authors focused on solving forward problems with PINNs. Typically, the boundary conditions are enforced through a term in the loss function, here shown in equation [ To avoid this approach, which may lead to errors, the authors propose the following structure to approximate the solution of the partial differential equation
i(x 0) (N 0))(2)) + g(x),
(25)
where NN 0) represents a fully connected neural network with trainable parameters 0, o(-) is signed distance function, which zero at the boundary, and g()) function that interpolates the boundary conditions. With this construct, û satisfies the boundary conditions for any parameters O. The authors propose efficient ways to estimate the signed distance function phi(-) and they propose to use transfinite interpolation for g()). which we implemented. We test this method in the heat sink example, detailed in Section 3.2 For this method, we treat the data at the boundary as boundary conditions, which are then interpolated. In order to match the amount of parameters of --PINNs we use hidden layers with 100 neurons each and we train for the same number of iterations. In Figures 3 and 4
that this approach improves significantly the performance of traditional PINNs, however is not able to match the performance of A-PINNS.
We also note that the scope of this method is much narrower than A A-PINNs. So far it has only been demonstrated to work on 2D examples and there is no direct path to make it work in surfaces in 3D, which is one of the strengths of A-PINNs. Also, it does not allow for noise in the boundary conditions, because the interpolation g(-) will exactly impose the noisy measurements.
C POISSON EQUATION WITH EXACT EIGENFUNCTIONS
In this example, we select case that should favor traditional PINNs: solving a Poisson equation in a simple square domain. Here, we fabricate the following solution
u(x, v) = (x² 1) (1² - 1) exp (-(x - y)3
(26)
which can be seen in the left panel of Figure 7 Then, we define the following boundary value problem to be tackled by PINNs.
Au(æ) = f(x), æ == ta, y} E B
(27)
u(x) 0, dB.
The domain B is defined by square in [-1,1] x 1. The expression for fx) can be obtained by applying the Laplacian to the manufactured solution u(x). In this domain, we can compute the eigenfunctions of the Laplace operator with homogeneous Neumann boundary conditions analyti- cally, which correspond to
01,m y) cos (
Lx
my π
cos
(28)
with m = 1,.., VN, where N is the total number of eigenfunctions. We note that the boundary value problem described in equation 28 has Dirichlet boundary conditions, such that these eigen- functions do not provide suitable basis to solve this problem. In this case, we only provide data at the boundary x E dB and the values of f(x: in the domain æ B, effectively solving the for- ward boundary value problem shown in 28 The partial differential equation loss in this example corresponds to
R Σ
SSEPDE(O)
=
Σ JEN(1)
Aujá, - ft
(29)
16

---

Under review as a conference paper at ICLR 2023
where the index represents the nodes of the mesh inside the domain j the neighbor nodes the node i, including itself. The term ü, represents the output of the neural network at the nodal locations j that depends trainable parameters 0, and f, corresponds to f(æ;), with zy the position of node i. For traditional PINNs, we use automatic differentiation to compute A and construct the loss term MSFPDE: We neural networks with 3 hidden layers of 100 neurons and train with ADAM for 50,000 iterations.
N(i) represents
ground truth
PINN
A-PINN exact algenfunctions
6-PINN numerical eigenfunctions
Figure 7: Poisson example. The ground truth manufactured solution, the approximation of tradi- tional PINNs, A-PINNs using the exact eigenfunctions input and A-PINNs using numerically approximated eigenfunctions by the finite element methods. For both cases A-PINNS, we use 100 eigenfunctions and an element size 0.066 to discretize the operators.
We perform a sensitivity study to understand the influence of the number of eigenfunctions and the size of the discretization used evaluate the operator A[-] and the eigenfunctions computed numerically. We vary the number of eigenfuctions computed analytically and numerically from 9 400, and the size of finite elements between 0.02 0.2. Figures and summarize the results of this experiment. First, we note that traditional PINNs perform very well as expected, obtaining mean squared error of 1. 1.72- 10-5, Then, we observe that the results of A-PINNs are sensitive to both the number of eigenfuctions and the discretization. When using the exact, analytical eigenfunctions, we are able to obtain low errors if we use more than eigenfunctions for certain discretizations. When using the numerically obtained eigenfunctions, the same is true, expect for the case of 400 eigenfunctions. In this case, we cannot use a bigger discretization than 0.1 because the number of nodes becomes less than 400. The discretization size has bigger impact on the errors for both cases. There is a sweet spot between 0,066 and 0. where most cases produce low error. The error when the discretization is big is expected, since the Laplace operator will not be accurately approximated. The error when the discretization is small is not straightforward to explain, but we hypothesize that in these cases the M EPDE term becomes less important during training as the differences for become smaller as the scale of the element decreases. In Figure 7 we can see qualitatively that both PINNs and A-PINNs can successfully solve this boundary value problem.
D REGRESSION WITH A --NN
In this section we test the capabilities of neural network that takes as input the Saplace-Beltrane eigenfunctions to perform a regression task on manifold. In this case, we do not include any physics that may act as regularization of the predicted function. For this benchmark, we compare against Gaussian process regression with a Matérn kernel, which can be approximated on the manifold using Laplace-Beltrami the Laplace- Beltrami eigenfunctions as well Borovitskiy et al.) (2020). We choose a dragon geometry 9,057 vertices and 18, 10 triangles. We create synthetic ground truth function by computing the geodesic distance from a vertex d(x:):
d(x)) (xx) sin T
(30)
ddmax
where dmaa corresponds to the maximum geodesic distance from the selected vertex. We train with 50, 100, 500 and 1000 data points, randomly selecting them. We repeat this process five times and report the median emror. For the neural network (A-NN), we train using 9, 25, 49, 100, 225 400 eigenfunctions. We one hidden layer of 100 neurons and hyperbolic tangent
17

---

Under review as a conference paper at ICLR 2023
exact eigenfunctions
numerical elgenfunctions
10-1
eigenfunctions
9
49
mean squared error
226
400
PINN
10-8
element size [-]
0.02
element size [-]
0.20
Figure 8: Poisson example sensitivity analysis. We run the Poisson example varying the two hyper parameters introduced by A-PINNs: the number of cigenfunctions used input and the element size used to discretize the operators. We test A-PINNs using the exact eigenfunctions and the ones approximated by finite elements to understand the impact of this approximation.
activations. For the Gaussian process, select =3/2 2 for the Matérn kernel and approximate it with 1000 eigenfunctions. The results are shown in Figure In the lower right panel, we can see that the accuracy of A -NNN depends on the number of eigenfunctions used input. When there few data points available,
preferible to have less eigenfunctions lower frequencies to avoid overfitting, As the number of data points increases, more capacity is needed approximate the data and the eтrоr decreases when more eigenfunctions are used. Nonetheless, using high number of eigenfunctions, such 225 and 400, never produces good results. On the other hand, Gaussian process regression excels in approximating this smooth function and always provides lower errors than NNNS. We also show in Figure 9 top row, comparison of the output produced by ANNN and Gaussian process when training with 100 points. In general, where there is data, both methods do good job approximating the function. However, in the absence of data points, such as the horns of the dragon, ANNN tends to predict more extreme values that increase the overall error. In this case, the overall mean squared error drops from 6.8 10-2 A-NN 1.8 10 for Gaussian process regression. We see that, in general, A-NNs tend to overfit the data, and the addition of known physics across the domain a regularization effect that improves prediction, as seen in Figure 2 and
E A-PINNS ON LARGE MESHES
In this section we show the time that it takes to compute the eigenfunctions of the Laplace- Beltrami operator of different sizes. We use the FEniCS library Logg al. (2012) to assemble the operators, and the SLEPc library Hermandez et al. (2005) to solve the eigen-problem. We take 14 geometries from the open souree repository httpt: /ttps:///ithbbs com/ hamm/alecjacobsoo. common- -eeooon------ CPUs (2.20GHz). The results are summarized Figure I0 For the largest mesh with 134,345 nodes, the assembly of the operators and solving the eigen-problem took less than 35 seconds. This amount nodes can be used to describe complex geometries. We also see excellent scaling with respect to the number of nodes. In this regard, training both traditional PINNs and A-PINNs mains the largest computational burden in the process, and the computational cost of computing the eigenfunctions be considered simple pre-processing step.
and compute 100 eigenfunctions. We use only 2 Intel(R) Xeon(R)
18

---

Under review as a conference paper at ICLR 2023
A-NN
Gaussian process
ground truth
10
eigertfunctions
mean squared error
10
points
1000
Figure 9: Regression on a dragon. We show the results when 100 data points for NN using 25 eigenfunctions and Gaussian process. The ground truth is shown in the lower left, along with the training points, shown in white. The error comparison for different levels of data availability and different number of eigenfunctions for A-NN are shown in the lower right panel.
150k
number of nodes [-]
time to compute 100 eigenfunctions [s] 40
Figure 10: Scaling of the eigenfunction computation. We compute 100 eigenfunctions for a variety of meshes.
We finalize this section solving partial differential equation on a large mesh with 99970 triangles and 49987 nodes from the model called Lucy. We set the following screened Poisson equation:
Au u f
(31)
f (sin(37xr) + cos(3ry) + sin(3m=))/3,
(32)
19

---

Under review as a conference paper at ICLR 2023
where {z, y, } the Cartesian coordinates of nodes. We note that this equation does not require boundary conditions to be solved. We generate the ground truth solution using finite elements, shown in Figure [T] left. The source term f is shown in Figure IT right. We train A-PINN using 100 eigenfunctions, hidden layers of 100 neurons. We train for 20,000 iterations with batch size 100. We reformulate the partial differential equation in energy form for the FYDE. We obtain excellent agreement with the ground truth solution as shown in Figure obtaining a mean squared error of 2.15 10-4. Thus, we can that A-PINNS works well for large meshes, and that the eigenfunctions can be computed quickly.
ground truth
A-PINN
source term f
0.5
Figure 11: Solving a partial differential equation on large mesh. We train A-PINN with 100 data points mesh with 99970 triangles and 49987 nodes.
20


Reviews:
[{"review_id": "dxwQbkQrqW", "reviewer": "Reviewer_ebMA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a PINN-style network where the physical quantities are computed on a manifold. The manifold geometry of the data is represented via a positional encoding based on the eigenfunctions of the Laplace-Beltrami operator of the manifold,  propose to represent the coordinates of the input geometry with a positional encoding. The Laplace-Beltrami operator as well as the eigenfunctions are estimated using a finite element solver.  \n\n", "summary_of_the_review": "At a high level the manuscript is well-principled and the authors explain conceptually how they incorporate geometry into the PINN. The paper does have drawbacks – one of the major concerns is the lack of comparisons to the other neural networks or geometric-informed PINNs that the authors mentioned. Additionally, the authors could provide more detail on how they numerically approximate the Laplace-Beltrami operator and the finite element scheme/package that was used. Overall, the idea is nice but the impact is not distinguished and its unclear how their method compares to others. ", "strength_and_weaknesses": "Strengths: \nThe authors’ clearly motivate the need to incorporate geometric/topological information into their method. A nice feature of encoding the NN with eigenfunctions of the Laplace-Beltrami Operator over the domain is being able to hand select the eigenfunctions based on the magnitude of the eigenvalue given prior knowledge about the nature of the problem\n\nWeaknesses:\nHowever, they don't justify their choice of computing eigenfunctions of a continuous Laplace-Beltrami  is their approach. As they say, there is no closed form available, and they could have gone with a graph approach with a graph laplacian. In this setting there are neural networks that directly compute the eigenvectors of graph laplacians, for instance PowerEmbedd from this paper https://arxiv.org/abs/2209.12054.  Thus this whole approach could have been done with a GCN. This is not included as a comparison. In general, numerical experiments aren’t extremely convincing – need more ablations/comparisons to other types of networks but PINN and non-PINN ones such as GNNs, transformers, etc.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "c0QgQm_UFs", "reviewer": "Reviewer_vdFL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel positional encoding mechanism to inform PINNs about the topology of the domain, expanding the effectiveness of PINNs to complex geometric domains. The proposed positional encoding mechanism represents the coordinates of the input geometry with the eigenfunctions of the Laplace-Beltrami operator of a manifold, creating an input space for PINNs that represents the input geometry. Extensive numerical experiments (mainly in 2D) were carried out to compare the proposed methodology against traditional PINNs, showing excellent agreement with ground truth for cases where traditional PINNs fail.", "summary_of_the_review": "The paper proposed a novel positional encoding mechanism to inform PINNs about the topology of the domain, expanding the effectiveness of PINNs to complex geometric domains. While the weakness of the proposed methodology is as distinguishable as its strength, the computation cost and the implementation complexity of the proposed methodology can be higher than the traditional PINNs, making it less promising.", "strength_and_weaknesses": "Strength: \nThe proposed methodology enables the PINNs to solve problems defined on complex geometric domains with good accuracy.\n\nWeakness:\n1. The differential operators in the PDEs cannot be computed directly from PINNs using AD after applying the positional encoding method. Instead, they are computed numerically on the linear finite elements.\n2. Computing the eigenfunctions of the Laplace-Beltrami operator on large meshes can be very expensive.\n3. The performance of the proposed method strongly depends on the number of eigenfunctions used for positional encoding, while this quantity is a hyper-parameter needs to be tuned case by case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ya9TRLkbQzi", "reviewer": "Reviewer_yokT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces ∆-PINNs, a novel form of physics-informed neural networks (PINNs) that leverages the eigenfunctions of the Laplace-Beltrami operator of the manifold to represent the geometry of a 3D solid. This advancement would expand the use of PINNs from simple geometric domains to more complex forms. The paper goes into great detail on how this method is implemented and presents multiple test cases comparing against baselines such as previous PINNs or ground-truth, showcasing that the proposed method outperforms prior PINNs and more closely matches ground-truth observations.", "summary_of_the_review": "This paper makes a good contribution towards physics-informed neural net, by leveraging the eigenfunctions of the Laplace-Beltrami operator of the manifold to represent the geometry of a 3D solid. The paper possesses good clarity of expression, a well-explained proposal, and does not stumble in convincing the reader and expanding knowledge. Other than a small suggestion to include an explanation of the finite element method and relevant applicational domain, I believe this paper is worthy of being accepted.", "strength_and_weaknesses": "The paper is overall strong; the language is clear, the problem being solved is made obvious, and the effectiveness of the proposed solution is presented in a convincing manner. Though I am not an expert on the mathematics being utilized, I was still able to follow the logical procession of the equations even if I could not grasp the details. The figures and visualizations were not confusing, and the captions served well in clarifying what little doubt the figures may have presented.\n\nHowever, the one potential weakness I see is the mention of the finite element method. I happen to know what that is, and I am sure an audience with a general physics background would know what that is, but given that this is a CS-focused paper, perhaps a paragraph explaining what the finite element method is and how it works would be helpful to offer further understanding for those readers that would not know what it is. Given that understanding the finite element method is important for understanding the methodology of the paper, I believe this would be a worthwhile addition.\n\nIn addition, a concrete example of how physics informed network is used in Section 2.1 will help the readers to understand the contribution better. I happen to work in this domain, so I know the meaning of the three MSEs. However, I doubt if people outside of our domain understand the need of using these three terms.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject