Accelerated Training of Physics-Informed Neural Networks (PINNs) using Meshless Discretizations
Authors: Ramansh Sharma
Varun Shankar
Venue: NeurIPS 2022 Accept (2022)

Abstract: Physics-informed neural networks (PINNs) are neural networks trained by using physical laws in the form of partial differential equations (PDEs) as soft constraints. We present a new technique for the accelerated training of PINNs that combines modern scientific computing techniques with machine learning: discretely-trained PINNs (DT-PINNs). The repeated computation of the partial derivative terms in the PINN loss functions via automatic differentiation during training is known to be computationally expensive, especially for higher-order derivatives. DT-PINNs are trained by replacing these exact spatial derivatives with high-order accurate numerical discretizations computed using meshless radial basis function-finite differences (RBF-FD) and applied via sparse-matrix vector multiplication. While in principle any high-order discretization may be used, the use of RBF-FD allows for DT-PINNs to be trained even on point cloud samples placed on irregular domain geometries. Additionally, though traditional PINNs (vanilla-PINNs) are typically stored and trained in 32-bit floating-point (fp32) on the GPU, we show that for DT-PINNs, using fp64 on the GPU leads to significantly faster training times than fp32 vanilla-PINNs with comparable accuracy. We demonstrate the efficiency and accuracy of DT-PINNs via a series of experiments. First, we explore the effect of network depth on both numerical and automatic differentiation of a neural network with random weights and show that RBF-FD approximations of third-order accuracy and above are more efficient while being sufficiently accurate. We then compare the DT-PINNs to vanilla-PINNs on both linear and nonlinear Poisson equations and show that DT-PINNs achieve similar losses with 2-4x faster training times on a consumer GPU. Finally, we also demonstrate that similar results can be obtained for the PINN solution to the heat equation (a space-time problem) by discretizing the spatial derivatives using RBF-FD and using automatic differentiation for the temporal derivative. Our results show that fp64 DT-PINNs offer a superior cost-accuracy profile to fp32 vanilla-PINNs, opening the door to a new paradigm of leveraging scientific computing techniques to support machine learning.

Accelerated Training of Physics-Informed Neural Networks (PINNs) using Meshless Discretizations
Ramansh Sharma
Department of Computer Science and Engineering, SRM Institute of Science and Technology, India rs71460srmist. edu. in
Varun Shankar
School of Computing, University of Utah, UT, USA shankar0cs. utah.edu
Abstract
Physics-informed neural networks (PINNs) are neural networks trained by using physical laws in the form of partial differential equations (PDEs) as soft constraints. We present a new technique for the accelerated training of PINNs that combines modern scientific computing techniques with machine learning: PINNs (DT-PINNs). The repeated computation of the partial derivative terms the PINN loss functions via automatic differentiation during training is known to be computationally expensive, especially for higher-order derivatives. DT--PINNs are trained by replacing these exact spatial derivatives with high-order accurate numerical discretizations computed using meshless radial basis function-fing finite differences (RBF-FD) and applied via sparse-matrix vector multiplication. While in principle any high-order discretization may be used, the of RBF-FD allows for D'T-PINNS to be trained even on point cloud samples placed on irregular domain geometries. Additionally, though traditional PINNs (vanillaPPINNs) are typically stored and trained in 32-bit floating-point (fp32) on the GPU, we show that for DT-PINNs, using fp64 on the GPU leads to significantly faster training times than fp32 vanilla-PINS efficiency and accuracy of DT-PINNs via series of experiments. First, we explore the effect of network depth on both numerical and automatic differentiation of a neural network with random weights and show that RBF-FD approximations third-order accuracy and above are more efficient while being sufficiently accurate. We then compare the D'T-PINNs to PINNNs on both linear and nonlinear Poisson equations and show that D'T-PINNs achieve similar losses with 2-4x faster training times on a consumer GPU. Finally, we also demonstrate that similar results can be obtained for the PINN solution to the heat equation (a space-time problem) by discretizing the spatial derivatives using RBF-FD and using automatic differentiation for the temporal derivative. Our results show that fp64 DT-PINNs offer a superior cost- cost-accuracy profile to fp32 vanilla-PINE opening the door to a paradigm of leveraging scientific computing techniques to support machine lcarning.
snecretely-trinne
with comparable accuracy. We demonstrate the
Introduction
Partial differential equations (PDEs) provide convenient framework to model large number of phenomena across science and engineering. In real-world scenarios, PDEs are typically challenging impossible solve using analytical techniques, and must instead be approximately solved using numerical method. A variety of numerical methods to solve these PDEs have been developed including but limited to finite difference (FD) methods (19) (which work primarily on rectangular domains
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

partitioned into Cartesian grids) and finite element (FE) methods (36) (which work on domains with curved boundaries but require partitioning the domain into multidimensional simplices). A modern class of numerical methods called meshless or meshfree methods generalizes finite difference methods in such a way as to remove the dependence on Cartesian grids, thereby allowing for the numerical solution of PDEs on point clouds. Of these, radial basis function-finie differences (RBF-FD) are among the most popular and widely-used (3 37 8 9 [2 13 TO] 25 26 14: 33 18), though a host of other such methods also exist. Much like FD or FE methods, these meshless methods can also approximate solutions to a desired order of accuracy.
More recently, PDE solvers based on machine learning (ML) have begun to gain in popularity due to the inherent ability of ML techniques such as neural networks (NNs) to recover highly complicated functions from data specified at arbitrary locations [1] 20). We focus on popular class of ML-based meshless methods called to discover/infer PDEs that govern a given data set, and as direct PDE solvers. Our focus in this work is on the latter problem, though our techniques extend straightforwardly to inferring PDEs as well. PINNs are typically multilayer feedforward deep NNs (DNNs) that are trained using PDEs and boundary conditions as soft constraints, leveraging automatic differentiation (autograd) for computing derivatives appearing in the PDE terms. The original PINNs, often referred to as vanilla- PINNs, are challenging to train, at least partly because PDE-based constraints lead to complicated loss landscapes (17) These issues are somewhat ameliorated by using domain decomposition (X- PINNs) (1) or gradient-enhanced training (G-PINNs) (38). Other approaches for ameliorating these issues involve curriculum training or sequence-to-sequee
informed neural networks (PINNs) (27). PINNs can be used both
learning (17). Many of these extensions can also help improve training and test accuracy. Much like other DNNs, PINNs are typically trained in 32-bit soating-point (i.e. fp32 or single precision).
In this work. we introduce a new technique for accelerating the training of vanilla-PINNs, Our technique relies on two key features: (a) using RBF-FD to compute highly accurate (nevertheless approximate) spatial derivatives in place of autograd, and (b) training the DNN in fp64 rather than fp32. These new discretely-trainet PINNs (DT-PINNs) can be trained significantly faster than fp32 vanilla-PINNG on consumer desktop GPUs with no loss in accuracy or change in DNN architecture. The use of RBF-FD allows DT-PINNs to retain the meshless nature of vanilla-PINNG thereby allowing for the solution of PDEs on domains with curved boundaries. As RBF-FD uses sparse- matrix vector multiplication (SpMV) to approximate the derivatives, DT-PINNS are also parallelizable on modern GPU architectures. is important to note that DTTPPNNS use autograd for the actual optimization of the PINN weights; only PDE derivatives are discretized using RBF-FD.
The NN literature does contain efforts to replace automatic differentiation with numerical differentia- tion. For instance, recent work showed that FD approximations be efficient for learning generative models via score matching (23). Another example is an NN architecture that involves learning FD-like filters for faster prediction of PDEs (35). In the PINN literature, fractional numerical differentiation as autograd cannot compute fractional derivatives 22)). Neverthless, to the best of our knowledge, ours is the first work on using meshless high-order accurate FD-like methods in conjunction with PINNs, allowing them to be trained without any loss in accuracy on domains with curved boundaries (just as autograd does). An alternative would involve eliminating autograd inefficiencies via Taylor-mode differentiation (4). However, we show that at least part of the speedups observed in DT-PINNs is because numerical differentiation results in training completing in fewer epochs than if autograd were to be used.
(F-PINNs) use
To alleviate concerns about replacing autograd with RBF-FD, we first compare fp64 RBF-FD approximation of different orders of accuracy against fp32 autograd for DNNs and show the cost benefits of using higher-order accurate RBF-FD. Then, to illustrate the features of DTT-PINNs, we focus for brevity on two purely spatial PDEs (the nonlinear and linear Poisson equations) and one space-time PDE (the heat equation). We use these settings to compare D'T-PINNs and vanilla-PINNs for relative errors, timings, and speedups on simple desktop GPU. We demonstrate through our experiments that DT-PINNS offer a superior cost-accuracy profile over Vanilla-PINNS.
The remainder of this paper is organized as follows. In Section 2 we review both vanilla-PINNS and RBF-FD. Next, in Section 3 we discuss how to train DT-PINNS Ns to solve both the Poisson and heat equations. Then, in Section we present experimental results comparing RBF-FD and autograd, and comparing DT-PINNs against vanilla-PINN on the Poisson and heat equations. We summarize

---

our results and discuss possible future work in Section 5 Finally, the appendix contains additional results, code snippets, and key implementation details.
Notation:
We use æ to refer to spatial coordinates in d dimensions. On the other hand, a bolded indicates vector with more than elements (an array). Finally, the symbol on top of quantity indicates that the quantity is an approximation.
quantity such
Broader Impacts: To the best of the authors" knowledge, there are no negative societal impacts of our work including potential malicious or unintended uses, environmental impact, security, or privacy concerns.
2 Review
We now provide a brief mathematical review of both vanilla-PINNY and RBF-FD discretizations. Unless we note otherwise, all derivatives in this section are spatial or temporal. We focus on three prototypical PDEs: the nonlinear Poisson equation, the linear Poisson equation, and the heat equation.
2.1 Physics-informed neural networks
Let Q c. Rd be a domain with boundary given by aQ; here, d is the spatial dimension. We will focus on the solution of the nonlinear Poisson equation on l using PINNs. Let e Rd and let R - R be the solution to
Au(x) = ea(x) f(x), x E EQ, (an(x) V + ()((x) = 9((), x E an,
(2)
where A the Laplacian in Rd, is the Rd gradient, n(xx) is the unit outward normal vector on the boundary 00, (x)) and g(r) are known functions, and a, 3 E IR are known coefficients. If the ex()) term is dropped from we obtain the simpler linear Poisson equation:
Au(x) = f(x), I E Ω
(3)
The vanilla-PINN technique for solving either Poisson problem involves approximating the unknown solution u(x) by a DNN ū(x, w) (where w is a vector of unknown NN weights), so that []ū(,, w) uxx)]l for some norm ||-|| and some tolerance e. In the absence of existing solution data, this accomplished by enforcing I and (2) as soft constraints on ü(x) to find the weights during training. Denote by X {axx j- the set of training points at which these constraints are enforced; the context of PDEs, these are also called collocation points. For convenience, we divide X into two sets: N₁ interior points in the set X; and N₁ boundary points in the set Xs; then, X X, U Xb- and N = N₁ + No. Further, let B = an(x)- +. The vanilla-PINN training loss e(x, w) can then be written as:
No
e(xr, w)
Aiiz, w) w-m,, - ((xx)) f(55))
+
Biil,, w)la=,, s(z:)) (4)
N
N
1=1
PDE
Boundary condition loss on boundary
where A and the V term in B are both applied through autograd. The tanh activation function is typically used, L-BFGS is used as the optimizer for finding the weights w, and training is typically done in fp32 (17). For the linear Poisson equation, one simply omits the e term from the loss above.
For time-dependent PDEs, the PINN becomes a function of space and time Ü(x t). We focus on the forced heat equation, given by
Ou(æ, t)
Øt Au(z,t)+ f(z,t), x E Ω
(5)
Bu(x, l) ,( (), E an,
(6)
u(x, 0) = un (z).
(7)
where (7 is an initial condition and ux(x)) is some known function. While the A term is handled
via autograd, there are two options to handle temporal derivatives: in a continuous fashion or time-discrete fashion. In the former, one samples the full space-time interval f2 x [0, T with collocation/trainng points, and then uses autograd to compute all spatial and temporal derivatives. The loss terms are also augmented with the initial condition (7). which is enforced on the full space- time solution. In the time-discrete approach, one typically discretizes the time derivative using an appropriate scheme (such as a Runge-Kurt Kutta method), and then proceeds in a step by step fashion. We focus on the continuous approach in this work.
3

---

2.2 Radial basis function-finite differences (RBF-FD)
We now briefly review RBF-FD methods. Given some function f: R + R, the goal of any FD formula is approximate the action of linear operator L on that function (i.e., to approximate f at some location T1. This is typically accomplished by using a weighted linear combination of f at Ij and its n nearest neighbors. Mathematically, this can be written as:
()()))
(xx()));
(8)
km
where the real numbers eh are called FD weights, and the set of points "1,..., In is called an FD stencil. In general, given set of samples X = { one can repeat the above procedure to find FD weights at every single point. These weights can be assembled into an N x N differentiation matrix L so that [f(x))x x se L fx))lx If n << N, L will be a sparse matrix with at most non-zero elements per row. If X lies on Cartesian grid, the entries of I. (i.e., the FD weights c&) are known in advance. However,
X is a more general point cloud, standard FD cannot be used to generate the entries of L (see Mairhuber- interpolatory combination of RBFs and polynomials instead. Without loss of generality, we describe the RBF-FD procedure for 21 and its -1 nearest neighbors. Let d(r) r" where is odd, be radial kernel (a polyharmonic spline), and (x)), j ...... (c++) be basis for polynomials of total degree E in d dimensions; we use tensor- pessor-product Legendre polynomials. The RBF-FD weights for the operator L at the point 21 computed by solving the following dense (block) linear system on this stencil:
theorem (7). The RBF-FD method involves using an
A PT
P
C
Ca Cq]
(9)
0
A
where
e
(
Au = & (1|2s - =2,1)), I,...,m,
Pu = (x,)), i= I,...,n,,j = I,...,
(10)
( d d
La= = L - x, (I) (D2z=zn Lq L2) .J =
(11)
where e is the (column) vector of n RBF-FD weights. The vector A is set of Lagrange multipliers enforcing the condition p'c-= = Lg, thereby ensuring that (a) the RBF-FD weights can exactly differentiate all polynomials up to total degree 6; and that (b) the error in the RBF-FD approximation to L when applied to all other functions is (hh+11))) where 0 < h S 1 is a measure of sample spacing in the stencil, and is the number of derivatives in the differential operator L (6). We set i = -=p+ 0 - 1 based on the desired order of convergence so that the error is O(h"). We then set the stencil size to if é is odd, and m = 1 if é is even (29). L becomes more dense for higher values of p and dimension d, as scales as O(N) for fixed n, with large speedups possible by computing multiple sets of weights using each stencil (28 [9] 34 32). For domains with fixed boundaries, the RBF-FD weights can be precomputed and reused during simulation. However, domains with moving boundaries require recomputation of RBF-FD weights proximal to the boundary every time-step; fortunately, this can be done quite efficiently (32)
( + 11 this ensures that (9) has a solution (2. and also set
Opr)). When this procedure is repeated for each point in the set X, the cost
Ghost points When tackling boundary conditions involving derivatives (such as in (2) using RBF. FD, it is common to include a set of N, ghost points outside the domain boundary Ol? into the set of samples to ensure that RBF-FD stencils at the boundary are less one-sided; this aids in numerical stability and accuracy. Ghost points allow us to also enforce the PDE at both the interior and boundary points. We therefore define and use the extended set X = X₁ U X, U X,, where X, is the set of ghost points. For the remainder of this article, let the RBF-FD differentiation matrix for A be L (dimensions (N₂ + No) x (N, + +2N))), and for B be B (dimensions N₂ x (N, + 2N2)).
3 EnccretelyTTrineee PINNS (DT-PINNs)
Having described both vanilla-PINN' and RBF-FD, we are now ready to describe DT-PINNS. In short, DT--PINN are PINNs that are trained using the sparse differentiation matrices L and B in place of the autograd operations used to compute the Laplacian and boundary operators in the loss function 4 (and its heat equation equivalent). All operations are carried out in fp64.
4

---

Poisson Equation Focusing first on the nonlinear Poisson equation T recall that ú(æ, w) is the PINN approximation to the true solution u(z). Let the evaluation of ü( w) on the set X be ü, iee, obtained by evaluating Ü(z, w) at interior, boundary, and ghost points. Further define the vector which is the loss function evaluated only the interior and boundary points, i.e. e(x, w)lx Then, the DT-PINN loss function can be written as:
N No. Lu - exp( u 2
Büü - gl
(12)
Nb.
PDE loss in interior and on boundary
Boundary condition loss on boundary
where L and B were defined previously, exp(ū) is the clement-wise exponential of the vector ü, and f= [(=)xx. and g - gxx) x,; here, has dimension (Nę + NN) x and g has dimension Ni. 1.
For efficiency, L and B can be precomputed using RBF-FD before the training process begins, and then simply multiplied with the vector ů to obtain its numerical derivatives. The loss function (12)
is then minimized over usual using autograd in conjunction with suitable optimizer. For the linear Poisson equation we simply drop the exp(i) (i) term.
Heat Equation When using DT-PINNS for the heat equation, we demonstrate the flexibility of our method by using mixed training technique where the time derivative is handled with autograd and the spatial derivatives are discretized with RBF-FD; this also allows us to bypass the Courant- Friedrichs-Leww Lewwy (CFL) constraint the time-step. We carefully order the evaluations of the network so that I. and B multiply the right quantities. Let ū(r f, w) be the PINN, and recall that we have N₁ time steps over the interval [0, 7]: in addition, also have the initial condition at time = making for a total of N, + steps. Define úg = -= === where A& is the timestep. This vector the evaluation of on all spatial locations (including ghost nodes) for the k-th time slice. This
definition in allows to define two vectors, ū. and úæ as follows:
Lüo Lüi
Buo Būn
üs
üg
(13)
LūN,
BüN
The vector ūt has dimensions (N, + ()
No) x 1, and üy has dimensions N,N, x 1. Next,
we define the data vectors and g as follows:
fo
fl
gi
g
(14)
[x.
LE
and gi: = g(x, x), Finally, we define two more vectors: uA = uo(x))x= the vector evaluating the initial condition on the set X (interior and boundary points); at spatial locations (interior and boundary) for each time
where f. = [(x
and ū. the vector of evaluations of 82 Ør
slice:
0
(15)
N
du - This vector is computed using autograd. With these different vectors defined, we can finally write the DT-PINN loss vector for the heat equation as
where (0x)
1
N₂ + N Uuo - ã ملا=X
(N₂ + 1)(N, + No) uA fll
(N₂ + 1)N jug
gi?
Initial condition
PDE loss in interior and houndaary
Boundary condition loss boundary
(16)
5

---

1.8
1.6
0.5
1.4
1.2
0
1
0.8
0.6
-0.5
-0.5
0.4
0.2
(a)
(b)
Figure 1: Quasi-uniform collocation points and the manufactured solution on the unit disk. The figure shows (a) N = 1663 interior and boundary collocation points on the unit disk and (b) the manufactured solution given by (19).
4 Results
We now present experimental results comparing DT-PINN and vanilla-PINN performance on the linear Poisson equation (3), the nonlinear Poisson equation ( and the forced heat equation (5)
for 5000 epochs are reproducible with the seeds we used in the experiments. We used the I.-BEGS optimizer with manually fine-tuned learning rates for both vanill--PINNs and DT-PINNs. Both DT-PINNs and vanilla-PINNS used a constant NN depth of = 4 layers with 50 nodes each across all runs. We use quasi-uniform collocation points generated using a node generator (30). Figure 16 shows the manufactured solution as specified in 19)). For the Poisson experiments, we report errors on test set of Ntet 21748 points. For the heat equation, we report results directly at the collocation points for convenience. For all experiments, the spatial domain Q is set to the unit disk
Setup All experiments
NVIDIA GeForce RTX 2070. All results
Ω = fx E Rd ||=:1 < 1}.
(17)
For illustration, show of the point sets the unit disk in Figure la (with N = 1663 points). In the 2D heat equation experiment, the space-time domain is chosen to be [0, 1]. The time interval [0, is evenly divided into 24 time steps that N, 24 (excluding t 0), and the time-step was set to Al Рí We measure all errors against a manufactured (specified) solution u, and specify / so that the solution holds true. The boundary condition term g computed by applying the operator B to u; we use a = 3 = for all tests. To compare DTTPINNS and mmilla-PiNNN to the manufactured solution u, we report the relative &g error
22
602
R4/12
(18)
where is the true solution vector, and ù is either the DT-PINN vanilla-PINN solution vector.
4.1 Effect of neural network depth
We first study the effect of PINN depth (fixing the number of nodes per layer) 8 on computing the Laplacian A of the output with respect to the spatial variable using either autograd RBF-FD. We compute errors against fp64 autograd for fp32 autograd and for RBF-FD with 2, 3, 4, and 5. N 19638 quasi-uniform collocation points. The results are in shown in Figure 2a We see p = 4 and p = 5 are more accurate than fp32 autograd, and that increasing increases the accuracy of RBF-FD by about two orders of magnitude. The errors are reasonably low for 3 also. In Figure 26 we report the time taken for the same test. It is immediately clear that fp64 autograd significantly more expensive than the fp32 variant, though both costs scale slowly with the network depth More importantly, the time taken for fp64 RBF-FD (for all orders) is both lower than both fp32 and fp64 autograd and independent of network depth 8, primarily since the RBF-FD weights can be precomputed and repeatedly reused during training.
All errors were computed

---

10
Relative error v
2.5 1e-1
Time vs.
lfp64)
2 (fp64)
p (fp64)
3 (fp64)
10-3
(fp64)
(fp64)
(fp64)
2.0
(fp64)
10-4
autograd (fp32)
autogred (fp32)
autograd (fp64)
10-3
Time (vec)
1.5
Relative
10-6
10-7
0.5
10-4
10-3
0.0
Network depth (s)
Network depth (s)
(a)
(b)
Figure 2: Autograd properties function of network depth s. The figure shows (a) effect of neural network depth 8 on the relative eπror (with respect to fp64 autograd) and (b) time taken for one application of autograd RBF-FD weights for = 19638 collocation points were precomputed using an efficient CPU code in approximately 0.1s. Error bars over 15 random runs shown.
fp32 and fp64, compared the time taken for SpMV using RBF-FD. The
4.2
Linear Poisson equation
1e-4
Relative error vs V
Time VS N
Speedup vs N
29
2 (fp64)
vanilla PINN (fo32)
27
vonilla PINN (Tp64)
(fp64)
p (fp64)
2 (1p64)
(fp64)
20
5 (1p44)
(p54)
(1po4)
venille-PiNN
(f664)
(fo04)
Relative error
venills PINN (1p64)
(1964)
15
Time (s)
Speedup
15
3
582
1663
2230
2196
4977
6114
8767
10638
582
828
1663
2238
3190
4977
6114
8767
19638
582
328
1663
2238
3198
6114
8767
18638
Training set size (N)
Training set size (N)
Training set size (N)
(b)
Figure 3: fp64 DT-PINNS and fp32 vanilla-PINN results on the linear Poisson equation 3 for different numbers of collocation points (N) and orders of accuracy (p). We show (a) the relative error in the PINN solution; (b) the time taken to converge to lowest relative error; and (c) the speedup attained by fp64 DT-PINNs relative fp32 vanilla-PINN for those times. Error bars over 5 random runs are shown.
Next, study the performance of fp64 DT-PINNS and fp32 vanilla-PINNe on the linear Poisson equation (3) on the domain [17] Letting x = [m1. x22, we specify the true solution u to be + sin(()11) cos (1T22)
2a (z) = u(z (2)
(19)
and enforce this by setting f = Au;; the true solution u is shown in Figure lb
We then solve for
as described Section The results of this experiment are shown in Figure 3 We present relative
emoor (Figure 3a), wall clock time (Figure 3b). and speedup (Figure 3c We also present results for
fp64 vanilla PINNs. is important to note that fp64 D'T-PINNs were completely stored and trained in fp64, a format widely known to be significantly slower the GPU than fp32.
7

---

Figure Baj shows the relative errors for DTT-PINNs as function of the number of collocation points N. DT-PINNs for p = 3, 4, 5 produce similar relative errors both fp32 and fp64 vanilla-PIND PINNNs for the same value of N. In contrast, the DT-PINN using p 2 is generally less accurate, showing that higher-order accuracy needed reach the same relative errors as vanilla-PINNs Examining Figures [3b and [c]
we also see that all fp64 DTT-PINNs can be trained much more rapidly than both fp32 and fp64 vanilla-PINNs. In fact, Figure 3c shows a maximum training speedup of 4x for DT-PINNS even if p 2 is ignored. In general, fp64 DT-PINNS for p > 2 are trained much more quickly than cnilla-PINNN without a significant loss in accuracy. We also note that using fp32 DT-PINNs did not lead to greater speedups over the fp64 DT-PPINNs, with a loss in accuracy. These results are shown in Appendix A.1.2
102
Training epochs N
1e2 Training epochs
vanille-PIMM PINN (fp32]
Vanille-PINI lfp32:
vanIlia-PINN (tp54)
9-
Va (Mp64)
= 2 (fp32)
p (fp64)
p 3 (tp32)
p 3 (tp64)
P (fp32)
p (fp64)
5 (fp32)
(fp64)
Epochs
Epochs
2
582
828
1863
2236
3196
4977
6114
8767
19638
582
828
1663
2236
3198
4977
6114
8767
19838
Training set size (N)
Training set size (N)
(a)
(b)
Figure 4: Number of training epochs to achieve the lowest relative error as function of number of collocation points N and order p for (a) fp32 DT-PINNs and (b) fp64 DT-PINNs, Error bars over 5 random runs are shown.
The superior performance of fp64 DT-PINNS becomes clearer when we examine the number of training epochs as function the number of collocation points N (Figure Figures 4ndd44/ illustrate that both fp32 and fp64 DT-PINNS reach their lowest relative errors in fewer epochs than vanilla-PINNs. These results provide evidence that DTT-PINNs have simpler loss function landscapes than their vanilla-PING PINN counterparts, also implying that loss functions involving linear combinations of NN values are easier to minimize than loss functions involving derivatives of NNs. Figure 46 also shows that only fp64 DT-PINNs take fewer epochs to train as N is increased. We also see that moving to fp64 does not appear to significantly speed up vanilla-PINNS. is therefore the combination of discrete training and fp64 that results in speedups for increasing N
4.3 Nonlinear Poisson equation
Next, to understand the influence of nonlincarities in terms not including the differential operator, we test the performance of DT-PINNS on the nonlinear Poisson equation (T). To measure errors, we use the manufactured solution given by (19). and set f = Au e". The results are shown in Figure 5 for simplicity, we omit p = and fp64 vanilla-PINNG as both these have poor cost-accuracy tradeofis. First, Figure 5a shows that despite some outliers, fp64 DT-PINNS achieve comparable relative errors to fp32 Vanilla-PINNNs. Further, Figure 55b shows that DT-PINNS are still trained faster than vanilla- PINNs. However, when comparing Figure 5c to Figure (linear Poisson equation). we see that the average speedup is higher for the linear Poisson equation. This shows that D'T-PINNs may not offer speedups over Manill-PINNS if terms not involving differential operators dominate training times.
'e also attempted to train fp32 vanilla-PIN PINNs using ghost points, but using ghost points offered no improvement (results not shown).
8

---

1e-4
Relative error vs.
1o2
Speedup
3,0
= (t:64)
vanila PINN (fp32)
3 (t644)
(fp64)
3 (tp64)
(f:64)
(fp64)
2.5
vanille-fttak (fp32)
p 5 (fp61)
2.50
Relative error
2.25
Time (x)
1.5
1.75
1.50
1.00
0.75
582
828
1665
2236
1196
4977
6114
5767
19838
582
828
1663
2238
119
4977
6114
8767
19636
582
828
1083
2230
3190
4977
6114
8707
19838
Training set size (N)
Training set size (v)
Training set size (v)
(a)
(b)
(c)
Figure 5: fp64 D'T-PINN' and fp32 vanilla-PINN results on the nonlinear Poisson equation ① for different numbers of collocation points (N) and orders of accuracy (p). We show (a) the relative error in the PINN solution; (b) the time taken to converge to lowest relative error; and (c) the speedup attained by fp64 DT-PINNs relative to fp32 vanilla- PINN for those times. Error bars over 5 random runs are shown.
4.4 Heat equation
1e-3
Relative error /
Speedup
1.0
DT-PINN (1p64)
DT-PINN (fp64)
(fa32)
5.0
0.8
Relative error
Speedup
4.0
3.5
3.0
0.4
2.5
2.0
1.5
Order of accuracy (p)
Order of accuracy (p)
(a)
(b)
Figure 6: 1p64 DT-PINN and fp32 vanilla-PINN results on the heat equation on N = 828 spatial points and N. 24 time-steps. The figure shows (a) the relative eиror for fp64 DT-PINNs as function of approximation order p; and (b) the speedup attained by fp64 DT-PINNs over fp32 vanilla-PINNs as a function of p. Error bars over 5 random runs are shown.
Next, compare fp64 DT-PPINN and fp32 vanilla-PINNs demonstrate the flexibility of our method, we adopt a mixed training approach where only spatial derivatives are discretized with RBF-FD. We specify the true solution u be
the 2D heat equation. In order to
u Jr, () u( 22, () = 1 + sin(TZ)) cos(TT22) sin(r!),
(20)
and specify f=
Au so that the solution u satisfies the heat equation for space-time. We
compute the initial condition as no(2, 0) u(x1, x2, 0) 1. We trained on N' = 828 spatial collocation points over 25 time slices (including time L 0) for total of 20, 700 spacetime collocation points; express all results function of p. These results are shown in Figure 6

---

First, Figure ba shows similar results to the 2D Poisson equation, with > 2 achieving relative errors similar fp32 DT-PINNs. (ojure that we achieve 2-4x speedups over vanilla-PINNs. function of the number
We observed in our experiments that the speedup appears to increase of time-steps N₂ (results not shown). It is likely that one could achieve further speedups by also discretizing the temporal derivatives, but we leave this exploration for future work.
Summary and future work
We presented a novel technique, DT-PINNs, that involves training PINNs by using RBF-FD for spatial derivatives, and using fp64 weights and training instead of fp32. This involved replacing all autograd operations (dense matrix-matrix multiplies) related to PDE loss terms with an SpMV operation. We showed that using RBF-FD approximation order of p > 2 resulted in DT--PINNs that were comparable in accuracy to vanilla-PINNS while offering 2-4x speedups in training times for both the linear and nonlinear Poisson equations. We also showed that DT-PINNS trained in a mixed fashion (autograd for time, RBF-FD for space) also achieved comparable accuracy and speedup on the heat equation. DT-PINNS therefore constitute a new paradigm for scientific machine learning that allow practitioners to leverage existing sophisticated scientific computing techniques to accelerate ML training times.
There are several possible extensions to our current work. It is likely that using DT-PINNs in conjunction with X-PINNs and G-PINNs will yield even greater speedups in training times. Further, DT-PINNs open the door to leveraging compute more efficiently. For instance, the SpMV operations could be parallelized using distributed memory systems in conjunction with GPUs, thereby allowing scaling to very large training sets; alternatively, the SpMV operation could be parallelized core CPUs while other operations are conducted on the GPU. It may also be profitable to explore mixed- peeecicinnionn training of DT-PINNs. Finally, DT-PINNS can be viewed partially linearized constraints; it may be profitable to explore other types of constraint linearization to accelerate training and simplify loss function landscapes.
many-
vanilla-PINNS with
Acknowledgments and Disclosure of Funding
VS was supported by National Science Foundation (NSF) grant CCF 1714844.
References
[1] Barnett, G. A. (2015). A Robust RBF-FD Formulation based on Polyharmonic Splines and Polynomials. PhD thesis, University of Colorado Boulder.
[2] Bayona, V., Flyer, N., Fomberg, B., and Barnett, G. A. (2017). On the role of polynomials in RBF-FD approximations: IL Numerical solution of elliptic PDEs. J. Comput. Phys., 332:257-273.
[3] Bayona, V., Moscoso, M., Carretero, M., and Kindelan, M. (2010). RBF-FD formulas and convergence properties. J. Comput. Phys., 229(22):2281-229
[4] Bettencourt, J., Johnson, M. J., and Duvenaud, D. (2019). faylor-mode automatic differentiation for higher-order derivatives in jax.
[5] Davydov, O. and Oanh, D. (2011). Adaptive meshless centres and RBF stencils for Poisson equation. J. Comput. Phys., 230(2):287-300
[6] Davydov, O. and Schaback, R. (2018). Minimal numerical differentiation formulas. Numerische Mathematik, 140(3):555-59) 592.
[7] Fasshauer, G. E. (2007). Meshfrre Approximation Methods with MATLAB. Interdisciplinaty Mathematical Sciences Vol. 6. World Scientific Publishers, Singapore.
[8] Flyer, N., Barnett, G. A., and Wicker, L. J. (2016a). Enhancing finite differences with radial basis functions: Experiments on the Navier-Stokes equations. J. Comput. Phys., 316:39-62.
[9] Flyer, N., Fornberg, B., Bayona, V., and Barnett, G. A. (2016b). On the role of polynomials in RBF-FD approximations: I. Interpolation and accuracy. J. Comput. Phys., 321:21-38.
10

---

[10] Flyer. N., Lehto, E., Blaise, S., Wright, G. B., and St-Cyr, A. (2012). A guide to RBF 2BF-ggenrattd finite differences for nonlinear transport: shallow water simulations on a sphere. Comput. Phys.. 231:4078-400
[11] Flyer, N. and Wright, G. B. (2007). Transport schemes sphere using radial basis functions. J. Comput. Phys., 226: 1089-44444
[12] Flyer, N. and Wright, G. B. (2009). A radial basis function method for the shallow water equations on a sphere. Proc. Roy. Soc. A, 465:1999-1976.
[13] Forbberg, B. and Lehto, E. (2011). Stabilization of 2BF-generated finite difference methods for convective PDEs. J. Comput. Phys., 230:2270-2285.
[14] Fuselier, E. J. and Wright, G. B. (2013). A high-order kernel method for diffusion and reaction- diffusion equations on surfaces. J. Sci. Comput., 56(3):535-565.
[15] Han, J., Jentzen, A., and E, W. (2018). Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34)850058855 8510.
[16] Jagtap, A. and Karniadakis, G. (2020). Extended physics-informed neural networks (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. Communications in Computational Physics, 28:5)220 2041.
[17] Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R., and Mahoney, M. W. (2021). Characterizing possible failure modes in innsice-informed neural networks. In Beygelzimer, A., Dauphin, Y., Liang. P., and Vaughan, W., editors, Advances in Neural Information Processing Systems.
[18] Lehto, E., Shankar, V., and Wright, G. B. (2017). A radial basis function (RBF) compact finite difference (FD) scheme for reaction-diffusion equations on surfaces. SIAM Sci. Comput., 39:A2129-A2151.
[19] Veque, R. J. (2007). Finite difference methods for ordinary and partial differential equations: steady-state and time-dependent problems. SIAM.
[20] Long, Z., Lu, Y., Ma, X., and Dong, B. (2017). Pde-net: Learning pdes from data.
[21] Okuta, R., Unno, Y.. Nishino, D., Hido, S., and Loomis, C. (2017). Cupy: A numpy-compatible library for nvidia gpu calculations. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirty-first Annual Conference on Neural Information Processing Systems (NIPS).
[22] Pang, G., Lu, I.., and Karniadakis, G. F. (2019). fpinns: Fractional physics-informed neural networks. SIAM Journal on Scientific Computing, 4164)226603A2226.
[23] Pang, T., Xu, K., LI, C., Song, Y., Ermon, S., and Zhu, J. (2020). Efficient learning of generative models via finitediifference score matching. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 19175-19188. Curran Associates, Inc.
[24] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.,, Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
[25] Piret, (2012). The orthogonal gradients method: radial basis functions method for solving partial differential equations on arbitrary surfaces. J. Comput. Phys., 231((0))466--4675
[26] Piret, C. and Dunn, J. (2016). Fast RBF OGr for solving pdes arbitrary surfaces. AIP Conference Proceedings, 1776(1).
[27] Raissi, M., Perdikaris, P., and Karniadakis, G. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707.

---

[28] Shankar, V. (2017). The overlapped radial basis function finite difference (RBF-FD) method: A generalization of RBF-FD. J. Comput. Phys., 342:211-228.
[29] Shankar, V. and Fogelson, function-finite difference (rbf-fd) discretizations of advection- diffusion equations. J. Comput. Phys., 372:616 639.
(2018). Hyperviscosity-batt based stabilization for radial basis
[30] Shankar, V., Kirby, R., and Fogelson, A. (2018a). Robust node generation for mesh-free discretizations on irregular domains and surfaces. SIAM Journal on Scientific Computing, 40(4):A2584-A261))
[31] Shankar, V., Narayan, A., and Kirby, R. M. (2018b). Rbf-loi: Augmenting radial basis functions (rbfs) with least orthogonal interpolation (loi) for solving pdes on surfaces. Journal of Computational Physics, 373:722-735.
[32] Shankar, V., Wright, G. B., and Fogelson, A. L. (2021). An efficient high-order meshless method for advection-diffusion equations on time-varying irregular domains, Journal of Computational Physics, 445:1 10633.
[33] Shankar, V., Wright, G. B., Kirby, R. M., and Fogelson, A. L. (2014). A radial basis function (RBF)-finite difference (FD) method for diffusion and reaction diffuusion equations on surfaces. J. Sci. Comput., 63(3):745-768.
[34] Shankar, V., Wright, G. B., and Narayan, A. (2020). A robust hyperviscosity formulation for stable RBF-FD discretizations of Advection-Difffsiinn Reaction equations on manifolds. SIAM Journal on Scientific Computing, 42(44::2371-A2200
[35] Shi, Z., Gulgec, N. S., Berahas, A. S., Pakzad, S. N., and Takáč, M. (2020). Finite difference neural networks: Fast prediction of partial differential equations. 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 130- 135. IEEE.
[36] Strang, G., Fix, G. J., and Griffin, D. (1974). An analysis of the finite-element method.
[37] Wright, G. B. and Fornberg, B. (2006). Scattered node compact finite difference-type formulas generated from radial basis functions. J. Comput. Phys., 219(1):99-123
[38] Yu, J., Lu, L., Meng, X., and Karniadakis, G. E. (2022). Gradient-enhanced physics-informed neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering, 393:1 14823.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] To the
best of the authors' knowledge, there are no negative societal impacts of our work including potential malicious or unintended uses, environmental impact, security, or privacy concerns.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes]
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
We release the datasets and codebase we used as part of the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "xRiBSqPLKJn", "reviewer": "Reviewer_8h43", "summary": "The paper uses a linear combination of radial bias functions  (RBFs) with additive odd poly-harmonics to discretise partial differential equations in a meshless manner. A deep neural network is used as trial function to obtain the approximate solutions through training over the data points with the equation to be solved as soft constraints.  In the  RBF-FD (finite differences) method the calculation of derivatives is performed with precomputed stencils. Restricting the approximation to nearest neighbors reduces the computational load significantly. The paper displays a solutions of Poisson and Heat dissipation equations as examples of the strengths of the method. It is found that the using a 4-ayer deep neural network that is trained to fulfill the equations using  GPU fp64 is very efficient in solving the problem compared to the state of the art.  ", "questions": "Question:\nIn the heat equation, the spatial data points are not in a regular mesh, but the mesh in time dimensions is. In the approximation (8) of the difference matrix in the implementation of the RBF-DT, the spatial stencil is based on nearest neighbors. How one is ensuring that the local Courant number is in the correct range (i.e CFL conditions are always fulfilled) despite the varying size of the spatial stencil? Is the global time step chosen in relation to the smallest spatial neighborhood in the irregular mesh? Or is the number of nodes in the spatial stencil made dependent of its local physical size? To me it seems that the RBF-DT with the full difference matrix is an implicit solver where this problem is not necessarily emerging, but making the approximation (8) makes the solver explicit as it will not depend on data further away of the nearest neighborhood.  Or is the fact that the temporal derivatives are calculated analytically with autograd using equation (15) automatically solving the issue?\nCould this be clarified in the text for the final version?\n\nSuggestion:\nThe reference to equations (3) and (2) on line 113 should be (2) and (1).", "limitations": "I do not think there are any potential negative societal impacts of this work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper goes trough, in a clear way, the theory of the  physics informed neural networks (PINN= and then builds upon that to introduce RBF-DF in a way that is pretty easy to follow.  It described system addresses many of the problems that arise in  the work of solving PDEs for complicated (and practical) geometries. The meshless nature  of the method makes its able to handle complex cases. Also, as it lacks the usual meshing of the geometry it is  much faster to set up ready for the computation, in addition to being efficient in its calculations.\n\nOf course, one would like that PyTorch would have all the  required autograd functions already implemented for even simpler scripting, but this is no way fault in the paper, but comment on paradigm shift where AI is providing new capabilities and better and better tools for problem solving. The appendix and submitted materials are valuable and greatly appreciated by those planning to use the solution method.\n\nThe figures do not have error bars, so one thinks the performance gains are reported from a single training run, albeit there is a good set of comparisons done to state of the art.  I would like to see results gathered from multiple runs for each of the points.                                                                                                                                   ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mlNyNkHcKiF", "reviewer": "Reviewer_6bVr", "summary": "The paper leverages modern scientific computing techniques, specifically the meshless radial basis function-finite difference (RBF-FD) method, to accelerate the training of PINNs. In detail, the paper replaces the computationally expensive automatic differentiation by RBF-FD with sparse-matrix vector multiplication, allowing a high efficiency and accuracy computation of the partial derivative terms, and a fp64 training on GPU. Several numerical experiments are carried out, including linear, non-linear and space-time problems, showing that the proposed method is significantly faster than fp32 vanilla-PINNs with comparable accuracy.", "questions": "1) The paper claims in line 77-78 that the proposed method “can be trained without any loss in accuracy on irregular domains (just as autograd does)”, while all the evaluation cases are carried out on regular domains, i.e. a unit disk in 2D. Extra evaluation cases with irregular domains are recommended.\n\n2) It is agreed that “the repeated computation of the partial derivative terms in the PINN loss functions via automatic differentiation during training is known to be computationally expensive, especially for higher-order derivatives”. Is the RBF-FD method capable to handle higher-order derivatives? If yes, it is recommended to add extra evaluation cases with higher order PDEs.\n\n3) In section 2.2 the RBF method is explained, while it is not quite clear the computation power required for pre-computation of the RBF-FD weights. For example, in Figure1(b), the time for pre-computing the RBF-FD weights should also be considered to achieve a fair comparison between autograd and SpMV with RBF-FD.\n\n4) Considering the RBF-FD weights, it is recommended to declare whether it is required to recompute and update during the training, especially for complex problems, e.g. strongly non-linear problem, shape deformation/optimization problem?", "limitations": "This method offers limited speed-up if the nonlinear terms not invoking differential operations dominates the calculation time.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe idea of the paper is straight forward - replacing the automatic differentiation by RBF-FD and sparse matrix-vector multiplication. The theoretical grounding is explained clearly, several empirical evaluations are used to validate the proposed method. The novelty of the contribution is fair, although both the PINNs and RBF-FD are well developed techniques, this is the first work combining them together to accelerate the PINNs training. The topic is closely related to the NeurIPS community.\n\nWeaknesses:\nThe weakness of the paper is mainly that the evaluation cases considered are somehow naive, which could not demonstrate the advantages of the proposed method sufficiently. The paper barely talks about the motivation of choosing fp64 over fp32 in the first place, and gives little reasons and evidences about why it would cause a training speed-up. And this leads to my suspicion on whether choosing fp64 would become a key point.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6r2jauq202C", "reviewer": "Reviewer_g9sM", "summary": "\nThe submitted work proposes the use of a discrete finite difference (FD) method for a faster computation of spatial derivatives (i.e. with respect to the signal domain) in Physics Informed Neural Networks (PINNs).\nMore specifically, the use of Radial-Basis-Function FD (RBF-FD) is proposed, which approximates an operator at a given position $x_i$ via a weighted linear combination of a position $x_i$ and its neighbors.\nThe weights can be learned by solving a dense block linear system, and an arbitrary point cloud may be used.\nThe underlying (implicit) interpolation between the points is a combination of RBFs and Legendre polynomials.\nThe linear operators are precomputed so that during the PINN training they can be directly evaluated on the collocation points.\nExperiments show the application of RBF-FD for the linear and non-linear Poisson equation, as well as the two-dimensional heat equation. For the heat equation, the temporal derivatives are still computed using auto-grad, and only the spatial derivatives are approximated via RBF-FD.\n\nResults show that the proposed method is able to accurately approximate the derivatives of Neural Networks with a significant speed-up compared to automatic differentiation. The speed-up depends on the complexity of the approximated operator.\nApplied to PINNs, this results in faster training of the networks with comparable accuracy to vanilla PINNs. Furthermore, the use of RBF-FD required fewer training epochs than vanilla PINNs, however, the exact causes for this are not known.\n\nContributions:\n- The use of RBF-FD for approximating differential operators in PINNs is proposed as an alternative to autograd\n- It is empirically shown that, with a sufficiently high order, RBF-FD provides accurate and fast approximations to the Laplacian of a Neural Network\n- On experiments for the (non-)linear Poisson equation and the Heat equation a  significant speed-up of PINNs is demonstrated. Fewer epochs were required for reaching a similar error as vanilla PINNs.", "questions": "- Going back to my point in the \"Quality\" section, why are only spatial derivatives approximated with RBF-FD? \n\n- Is there any existing understanding or intuition regarding **why** the RBF-FD method requires fewer training epochs? ", "limitations": "- The authors highlight that the speedup depends on the complexity of the approximated operator, i.e. they may not offer much speedup if nonlinear terms not involving differential operators dominate training times.\n- The approximations to the operators have to be precomputed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Significance:\n- [High] The authors provide an efficient alternative to autograd for computing PDE losses in PINNs. \n- [Low] The proposed method results in fewer required epochs for attaining a similar loss compared to vanilla PINNs.\n- [Low] The proposed method appears to be very general and orthogonal to existing methods for speeding up PINNs, allowing a seamless combination with existing work.\n\n### Originality:\n- The usage of RBF-FD as an alternative to autograd for PINNs is to the best of my knowledge novel.\n- As far as I can judge, RBF-FD methods seem to be well-established within the numerical PDE literature but have not yet been explored much within Machine Learning.\n\n### Quality:\n- The submission and experimental settings appear to be sound to the best of my knowledge.\n- While limited in their extent (in terms of different PDEs), the experiments serve as a valuable proof of concept.\n- It is not explained, why the approximation is only used for spatial derivatives whereas autograd is used for temporal derivatives.\nIt is unclear, whether this serves as an example for mixed usage of autograd and RBF-FD, or is due to some limitation of the underlying method. \n- Well maintained and readable code is provided; The appendix provides implementation details for the autograd implementation of sparse matrix operations, that are required for the proposed method.\n- No error bars are provided (Checklist 3.c). The authors argue that this is the norm for PINNs, which in my opinion does not warrant ignoring general best practices within the ML community. However, another argument made is the limited computational resources, which is reasonable for the PINN experiments. Experiments concerning Figure 1 (autograd vs RBF-FD) should not be very compute-intensive.\n\n\n### Clarity:\n- The paper is generally well written and argued for.\n- (minor) The plots are at times a bit confusing:  \n    - The x and y labels could be a bit bigger and more expressive (e.g. \"network depth s\" instead of just \"s\" in Figure 1, similar to N in Figures 2-4 and p in Figure 5)\n    - It's hard to (quickly) distinguish the vanilla PINN from the proposed method. Maybe a different style or color coding would help.\n\n\nTo summarize,  the paper provides an interesting and feasible alternative for computing spatial derivatives in PINNs, bridging the gap between well-established mesh-free methods for PDEs and PINNs.\nThe proposed method is novel and to the best of my knowledge technically sound, experiments are reasonable and reproducible, and the paper is well written. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept