Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks
Authors: Ghazal Farhani
Alexander Kazachek
Boyu Wang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs). However, even the simplest PDEs, often fail to converge to desirable solutions when the target function contains high-frequency modes, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under stochastic gradient descent with momentum (SGDM). This demonstrates SGDM significantly reduces the effect of spectral bias. We have also examined why training a model via the Adam optimizer can accelerate the convergence while reducing the spectral bias. Moreover, our numerical experiments have confirmed that wide-enough networks using SGDM or Adam still converge to desirable solutions, even in the presence of high-frequency features.

Under review as a conference paper at ICLR 2023
MOMENTUM DIMINISHES THE EFFECT OF SPECTRAL BIAS IN PHYSICS-INFORMED NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Physics-informed neural network (PINN) algorithms have shown promising re- sults in solving wide range of problems involving partial differential equations (PDEs). However, even the simplest PDEs, often fail to converge to desirable solutions when the target function contains high-frequency modes, due to phe- nomenon known spectral bias. In the present work, exploit neural tan- gent kernels (NTKs) to investigate the training dynamics of PINNs evolving under stochastic gradient descent with momentum (GDM). This demonstrates GDM sig- nificantly reduces the effect of spectral bias. We have also examined why training model via the Adam optimizer can accelerate the convergence while reducing the spectral bias. Moreover, our numerical experiments have confirmed that wide- enough networks using GDM Adam still converge to desirable solutions, even in the presence of high-frequent features.
INTRODUCTION
Physics-inforee neural networks (PINNs) have been proposed as alternatives to traditional numeri- cal partial differential equations (PDEs) solvers (Raissi et al. 2019 2020 SSriinnano & Spiliopoulos/ 2018 Tripathy& & Bilionis 2018). In PINNs, PDE which describes the physical domain knowledge of a problem is added as regularization term to an empirical loss function. Although PINNs has shown remarkable performance in solving a wide range of problems in science and engineering (Cai
al., 2022] Kharazmi et al.) 2019 Sun et al. 2020; Kissas et al.) 2020: Tartakovsky et al.) [018)), regardless of the simplicity of PDE itself, they often fail to converge to accurate solutions when
the target function contains high-frequency features (Krishnapriyan et al. 202T Wang al. 2021
This phenomenon known as the spectral bias exists in even the simplest linear PDES (Wang et al.
2021 Moseley et al. 2021 Krishnapriyan et al. 2021).
Spectral bias is not limited to PINNs. Rahaman al. 2019 empirically showed that all fully- connected feed-forward neural networks (NNs) are biased against learning complex components of target functions. Furthermore, Cao al. 2019) theoretically proved that in training infinitely-wind networks with squared loss, the corresponding eigenvalues of the neural tangent kernel (NTK) (Jacot ltt al. 2018) indicate the exact convergence rate for different components of the target functions. Thus, spectral bias happens when the absolute values of some of the eigenvalues of the NTK are large while others are small. Recently, utilizing the NTK of infinitely-wide PINNs, [Wang et al. 2022) examined the gradient flow of these networks during training. They proved that the training error decays based on e -xet where K; are the eigenvalues of the NTK. Thus, the components of the target function corresponding to the smaller eigenvalues have a slower rate of decay, which causes spectral bias. To tackle the issue of spectral bias, they proposed assign weight to each term of the loss function and dynamically update it. Although the results showed some improvements, as the frequency of the target function increased, their proposed PINN still failed to converge to solutions of PDEs. Moreover, assigning weights can result in indefinite kernels, the training process could become extremely unstable. Of note, compared to the typical NNs, analyzing the effect of spectral bias for PINNs is more challenging the loss function is regularized by means of adding the PDE equation. Thus, Wang et al. (2022) 's study was limited to training the model only based on GD.
Some studies proposed an alternative approach in which instead of modifying the loss function terms, a high-frequency PDE is solved in a few successive steps. In these methods, it is assumed that the optimal solution of low-frequency PDEs is close to the optimal solution of high-frequency

---

Under review as a conference paper at ICLR 2023
PDEs. Hence, instead of randomly initializing weights they are being initialized using the optimal solution of low--reqqennc PDEs. Moseley et al. (202T) implemented a finite element approach where PINNs were trained to learn basis functions over several small, overlapping subdomains. Similarly, Krishnapriyan et al. (2021) proposed learning method based on learning the solution over small successive chunks of time. Moreover, they proposed another sequential learning scheme in which the model was gradually trained on target functions with lower frequencies, and, at each step, the optimized weights were used as the warm initialization for higher-frequency target functions. In similar approach, Huang & Alkhalifah (2021) proposed to use the pre-trained models from low- frequeney functions and to increase the size of the network (neuron splitting) as the frequency of the target function is increased. Although these methods showed good performance on some PDEs, as the frequency terms became larger, the process became much slower as the required time steps would significantly grow.
In this work, we study the spectral bias of PINNS from an optimization perspective. Existing studies only have focused on effect of the vanilla GD (Wang et al. 2022) they are limited to some weak empirical evidence indicating that Adam might learn high feature faster Taylor et al. 2022). ) We prove that an infinitely-widd wide PINN under the vanilla GD optimization process will converge to the solution. However, for makes the convergence extremely slow, and hence not possible in practice. Moreover, we prove that for infinitely-wide networks, using the GD) with momentum (GDM) optimizer can reduce the effect of spectral bias in the networks, while significantly outperforming vanilla GD. We also investigate why the Adam optimizer can also accelerate the optimization process while decreases the effect of spectral bias in PINNs. To the best of our knowledge this is the first time that the gradient flow of the output of PINNs under the GDM, and Adam are being analyzed, and their relation to solving spectral bias discussed. Finally, our extensive numerical experiments on sufficiently wide networks confirm our theoretical findings.
modes, the learning rate needs to become very small, which
2
PRELIMINARIES
2.1
PINNS GENERAL FORM
The general form of well-posed PDE on a bounded domain (I R*) is defined as:
D uu] (x) fxx), x E Ω
u(x) = g(x), x an
where a differential operator and u(x) the solution (of the PDE), in which x ..g....aal ta) are viewed as ad- ditional coordinates within x. Hence, the initial condition is viewed as a special type of Dirichlet boundary condition and included in the second term.
(I), T2,..2 zz). Note that for time-dependent equations, 1 = (41, t2,.......
Using PINNs, the solution of Eq. can be approximated as u(x, w) by minimizing the following loss function:
No ((w) = No (u(x), w) gxx(g((())))
N₂ (D (u w) f(x:)))
Ca(w)
(w)
where [ and {xr/ |, are boundary and collocation points respectively, and w describes the neural network parameters. Lr(w) corresponds to the mean squared error of the boundary (and initial) condition data points, and L, (w) encapsulates the physics of the problem using the randomly selected collocation points. Similar to all other NNs, minimizing the loss function L(w) results in finding the optimal solutions **,
2.2 INFINITELY- WIDE NEURAL NETWORKS
fully-connected infinitely-wide NN with hidden layers can be written as Jacot al. (0018)
uh. (x)
N O2
Xn + b
X141 (w))
2

---

Under review as a conference paper at ICLR 2023
where O and by, respectively the weight matrices and the bias vectors in the layers h 1...., L, N the width of the layer, and (-)) ) is a B-smooth activation function (e.g. tanh(-)).
The final output of the NN is written as:
uxx, w) = N
o XL bb
where (, bo....; Or bt). At each time step can determine the change of the output with respect to the input, which defines a NTK:
K [xxx' (wu(x,((() T Vwu(x' w()).
is worth noting that the NTK associated with the model and is completely independent of
the choice of optimization algorithms the loss function. Liu et al. (2020) showed that for fully-connected NN with a linear output layer, the spectral norm of its Hessian satisfies [[H(w)] O V- Consequently,
the width become larger, the norm of the Hessian becomes smaller:
lim [ = O. Thus, one major consequence of dealing with infinitely-wide fully-connected NNs is that if the last layer of the network is linear, the NTK becomes static (not changing iterations), and the output of the network can be linearized (Liu et al. (020)):
ue (ww) =e w(w)) + (w Wo) Vu Vu(w)lmo
THEORETICAL RESULTS
3.1 CONVERGENCE OF GRADIENT DESCENT IN THE PRESENCE HIGH-FREQUENCY FEATURES
Generally, the optimization problems corresponding to over-parametrized systems, even on local
scale, are non-convex (Liu al. 2022 A loss funetion L(w) of y--uniformly conditioned NN
satisfies the p-PL, condition on a set S
C R if:
HC(w) for all w E S
(3)
where j is the lower bound of the tangent kernel K(w) of the NN. It has been shown that infinitely- wide networks satisfy the μ-PL, condition, variant of the Polyak-Lojasiewie condition, and as a result the (stochastic) gradient descent (SGD/GD) optimization algorithms will converge to the optimal solution (Liu et al. (022). The following proposition makes use of the μ-PL, condition to provide a convergence analy malyyis for an infinitely-wide PINN with loss function as in Eq. optimized with GD (Appendix(A) [A).
Proposition 1. Let Abmax and Armas respectively, be the largest eigenvalues of the Hessians V (w") and V2 (ww"). Consider an infinitely-wide PINN optimized with the following update rule:
W++1 W. r/WL(w))
where is a constant learning rate. Then, provided (11/(Nrmax + +rrmaa)) the J-PL, condition satisfied and the PINN will converge.
Although Proposition provides convergence guarantee for an infinitely wide-wide PINN, for PDEs exhibiting stiff dynamics (those with high frequency modes), the eigenvalues (of the Hes- sian) dictating convergence are often very large (Wang et al,, 2021). For example, take the one- dimensional Poisson equation:
8v.
Ox f(x), E
u(x) g(x), x an
with u(x) = sin) The norm of the Hessian H((w:) of order (C**) (Appendix Bb. Thus, as C grows does the bound the eigenvalues of (w,w,,, indicating that at least one of Abuae Xrmse will be large. Consequently, the learning rate must prohibitively small to guarantee convergence. Therefore GD is unusable in practice.
3

---

Under review as a conference paper at ICLR 2023
Ampitude
Time Stops (Epazhs)
Figure 1: Damped harmonic oscillators show three different characteristics.
3.2
WIDE PINN OPTIMIZATION USING GD WITH MOMENTUM
In order to accelerate learning, we investigate the training dynamics of PINNs under GD with mo- mentum (GDM) (Du 2019h:
W++1- W: + a(w: wt-1) v7V
(5)
where and 1) fixed rates. aates.Specificaat Specifically, analyze the gradient flow of the update rule in Eq. by leveraging the notion of the NTK. The following theorem (proof in Appendix reveals that GDM has different convergence behavior from GD:
Theorem 1. For infinitely-wide PINN, the gradient flow of GDM is:
i(x,, w()) [xū w()) =
ú(xo, (()) (x w(r))
u(x6, w()) gxx)) Diu](xr, w(r) h(xx))
K
that is analogous to point mass undergoing damped harmonic oscillation in viscous medium with a friction coefficient of p(a) that is function of a, Furthermore, is defined as:
Kbb Kbr
Kyb
K
where:
Kbbra..r) V v()(w, T Vwu((w,x')
Kbr(ar." V vw)(w,)) T V D(u((((
Kyrea.e") D(ul(((( T V., D(u((w((
three NTKs associated with the boundary and residual terms. Moreover, let 7 4/2m, K₁ be the i-th eigenvalue of K, and ni 6i, Then, the solutions to the gradient flow are of the form:
A + Agedust
(6)
A = + v22 1²
where A₁ and A2 are constants.
By examining the analogous gradient flow for the vanilla GD, where the training error decays the rate e e-xst (Wang et al. 2021) thus PINNs under vanilla GD suffer from spectral bias.
Once add momentum, the decay rate analysis becomes more involved as Eq. 6 yields three different cases of solutions. Each of the three cases is analogous to one the solutions of a damped harmonic oscillator Qian (1999); Arya (Fig.
Under-damped: Imaginary roots (72 < k())
drime: Real and equal roots (72 ki))
4

---

Under review as a conference paper at ICLR 2023
Over-damped: Real roots (7² > xt))
Under-damped case As v12² - nig has imaginary roots, Eq. 6 can be rewritten as:
At cos (wit ()
where W1 = Rv - and A and @ are two constants corresponding to the amplitude and the phase of the damped oscillation. In physics, this solution corresponds to an oscillatory motion in which the amplitude is decaying exponentially.
Critically-aamped case The general solution of the critically case can be written as:
(Bi - B2l)e
where B₁ and B2 are constants. As the oscillation motion is not present, the decaying rate for this case is much faster (Fig. D.
Over-damped case Lastly, in the over-damped case the general solution is simplified as:
e'1 (Cie + w2t)
where w2 222 NG and C1 and C2 are constants. Similar to the critically-damped case, the above equation states fast decay.
Thus, depending [x.. the absolute value of the eigenvalues of the kernel matrix K, the dynamics of the training error corresponding to different frequency components can differ. For larger eigen- values, the training error corresponds to under-damped solution, in which the amplitude of an oscillatory motion is decaying exponentially, whereas for smaller eigenvalues the correspondence is eritically-damper oscillation, with much faster decay. Thus, when using GDM instead of vanilla GD, the training process for the components of the target function that cor- respond the smaller cigenvalues will decay fast (undergoing the over-damped or critically-dampe motions), while the components that correspond to the larger eigenvalues will decay at a slower rate. As consequence, the effect of spectral bias will be less prominent (compared vanilla GD). To provide visualizations, in Appendix the decay dynamics of GDM and vanilla GD are plotted using small and large eigenvalue.
with over-damped
NUMERICAL EXPERIMENTS
In the previous Section, we proved that for infinitely-wide networks GDM and Adam can diminish the effect of spectral bias in theory. Here, we show that in practice, for sufficiently wide networks, Adam and GDM can significantly diminish the effect of spectral bias. Using Poisson's equation, the transport function, and the reaction-diffusie problem provide results from numerical experiments. Experiments for the reaction-diffusion problem are presented in Appendix H.2
4.1 POISSON'S EQUATION
Poisson's 's equation well-known elliptic PDE in physics. For example, in electromagnetism, the solution to Poisson's 's equation is the potential field of given electric charge. In Eq. 4 the general form of the mie-diensiionl Poisson's 's equation was presented. Here, we write Poisson's equation for specific source function and specific boundary condition:
f(x) ) = -C² sin(Cx:) x E [0,1] g(x) 0, 0,1
The defined loss function well as the analytical solution of Poisson's equation are shown in Appendix G We trained a network of two hidden layers of width 500, N, 100 and N, 100. Of note, Wang et al. (2022) had shown that during the training process, the NTK of networks with width of 500 practically stayed constant.
Our numerical experiments confirmed that for relatively small value of C 5л (where the effect of the spectral bias is not significant), after 55000 epochs, models trained via all three algorithms could accurately estimate the solution (Fig. top panel). The relative error [[ u)/ull for vanilla
5

---

Under review as a conference paper at ICLR 2023
GD was the order of 10 The relative error using GDM and Adam were respectively the order 10-3 and 10 (Fig. Ba). As the parameters of a PDE increase, its solution contains higher frequency modes, and as result, convergence becomes more challenging. When C' 10T, after 55000 epochs, the solution obtained from training the network with vanilla GD was far from the exact solution (Fig- [ bottom panel), and it exhibited relative enror larger than 10-1 (Fig, 3p). Meanwhile, the trained models with GDM and Adam had relative errors of 10 and 10 respectively (Fig. 3p).
It is also of interest to investigate the behavior of the training loss function and its decay rate. For the frefereuency case (C 5T)) the training loss via Adam had faster convergence, however, all three networks converged after about 30000 epochs (Fig. 4 left panel). For the high-frequenee case (C 10x), the training loss via GD had much slower decay rate, and after 35000 epochs did not converge. On the other hand, the training loss for both GDM and Adam could converge, and the model trained via Adam converged after about 25000 epochs (Fig. right panel).
(a)
(b)
(c)
1.0
1.0
0.5
0.0
0.5
005
0.5
-1.0
-.00
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
06
0.8
1.0
0.0
0.2
0.4
0.6
0.0
1.0
1.0
1.0
0.5
0.5
0.0
0.0
y -10
0.5
-0.5
3300
-1.0
-1.0
0.0
0.2
04
0.5
088
1.0
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.8
Figure 2: ID Poisson equation when C' 5m (top panel), and 10x (bottom panel). (a) Vanilla GD after (b) GDM (c) Adam.
(a)
(b)
Pointwise Error
10
Pointwise Error
10a
10r
10-1
10-2
104
W
10-3
10-
104
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 3: Estimated training error for ID Poisson equation. a) C' = 5T. b) = 10T.
Training Error
10a
10a
10
10a
Training
10
20
50
30
Epochs
*10
Epochs
Figure 4: Left panel: training loss when C 5T. Right panel: training loss when = 10T.
Of note, using vanilla GD and training the model for 180000 epochs resulted in a small relative error on the order of 10-2 (Fig. G.1). This confirmed our discussion presented in Section 3.1 that vanilla

---

Under review as a conference paper at ICLR 2023
GD will converge under large parameters, though due to the presence of high-frequency features it extremely slow.
4.2 EVOLUTION OF THE SOLUTION DURING LEARNING
Knowing the fundamental difference between GDM and GD. it is of interest to investigate how under these two algorithms solutions are evolved (in time). Thus, for the Poisson equation when C' 15x, we have plotted the solutions at epochs 2000, 10000, 20000, 30000, 40000, and 50000 (GD: Fig. 5 and GDM: Fig. 6 For GD: at epoch 10000, the solution has the correct sinusoidal shape, however it is vertically shifted and clearly cannot satisfy the boundary conditions. Plotting the eigenvalues of Kss (Fig. F.I. confirms that they are much smaller than the eigenvalues of Krr indicating that under GD the boundary and initial conditions are learnt much slower. As the training continues in time, the solution becomes less vertically shifted and closer to the boundary condition values. However, the learning process is slow, and the particular solution. Mcanwhile, GDM is much faster, and at epoch 10000 the solution already has the correct sinusoidal shape and satisfies one end of the initial conditions. By epoch 20000 the estimated solution has low error on the order of 10
epoch 50000 the estimation far from
(a)
(b)
www.
-10
0. 02 04
0.5
0.8
1.0
0.0
022
04 08
0.
1.0
0.0
02
04
08
0.
1.0
(d)
(9)
(1)
w
0.2 0.4 06 0.8 1.0
0.8
02
Figure 5: The solution in different stages of the learning process via GD. a) 2000 epochs, b) 10000 epochs, c) 20000 epochs, d) 30000 epochs, e) 40000 epochs, and f) 50000 epochs.
(b)
(6)
-10
45
15
1.0
-.5
25
2.00
0.0
02
04 08 088 1.0
0.0
0.2
0.4
0.0 1.0
0.0 02 0.4 0.0 0.0 1.0
0.0 0.2
0.4
0.8
1.0
0.0
02
0.4
0.6
1.0
0.0 02 0.4
0.6
0.8 1.0
Figure 6: The solution in different stages of the learning process via GDM. a) 2000 epochs. b) 10000 epochs, c) 20000 epochs, d) 30000 epochs, e) 40000 epochs, and f) 50000 epochs.
4.3 TRANSPORT EQUATION
The transport equation is hyperbolic PDE that models the concentration of substance flowing in a fluid. Here, we focus on a one-dimensional transport equation:
du
du
0, x EE E [o. 1]
a1
Or
g(z) = u(x, 0), ZEΩ
7

---

Under review as a conference paper at ICLR 2023
where 3 is control parameter (independent of and t). To facilitate later comparisons with Krishnapriyan et al. (202I), we used the same network architectures as their study: N, = 100, Nr 1500, and 4-layer network. We also chose the boundary and initial conditions to be u(x,0) = sin(x) and u(0,t) u(2m, t). The defined loss function as well as the analytical so- lution of the transport function shown in Appendix G
Similar to Poisson's 's equation, for small & values, the models trained via vanilla GD, GDM, and Adam could all easily converge to the solution, and had small relative errors. However, for B 20, after 125000 epochs the model trained with vanilla GD still failed to find the solution, and the averaged relative eтror stood a large value (on the order of 109). However, the model trained via GDM after 55000 epochs could converge to the solution and the estimated solution had the relative emoor on the order of 10-2, The estimated solution from training the model via Adam, after only 15000 epochs, had small relative error also on the order of 10-2 The exact solution, the estimated solutions (based on the three optišizers), and the absolute difference between the exact and estimated solutions are shown in Fig.
(a)
(b)
10
0.0
2
0.8
2
4.8
2
0.5
-10
0
-10
0
1.0
0.0
0.2
0.6
1.0
0.0
0.4
0.8
0.2
0.8
1.0
1.0
1,0
6
1.0
005
0.8
05
00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
02
0.4
0.6
0.8
1.0
-10
0.0
0.2
0.4
0.6
0.8
1.0
12
0.08
0,025
08
4-
0.00
4
2
04
2
0.04
2
0015
0,000
0.0
0.2
0.6
1.0
0.0
02
0.4
0.6
0.8
1.0
0.2
0.6
1.0
Figure 7: transport equation when 3 20: Top panels: The exact (analytical) solution. Middle panels: The estimated solutions. Bottom panels: The absolute difference between the exact and estimated solutions. (a) Vanilla GD, (b) GDM, (c) Adam.
4.4 COMPARISON WITH OTHER METHODS
Here, compare the solutions for Poisson's equation and the transport function (from wide net- work trained with Adam) with the solutions based on the curriculum learning (hereafter C-learning) Krishnapriyan et al. (202)) and the adaptive weight approach (hereafter AW) presented in Wang et al. (2022) Of note, C-learning used the Limited-memory
Fwetcher--ooelrfeeee Shanno (L-BFGS) algorithm that mimics the second-order optimization, as Krishnapriyan et al. (2021) re- ported that, for their approach, using other optimization algorithms under performed compared to L-BFGS. To reproduce the results we used the code respectively presented in KKrishnapriyan et al.
2021 and Wang et al. (2022). All networks were trained with the same architecture introduced in Section
We tested ID Poisson's 's equation with different values of C.. The comparison between the relative errors of the estimated solutions is plotted in Fig. For C 8n, both AW and C-learning exhibited errors of the order of 10-1 The errors using Adam were at most 10-3, For the 1D transport function with initial condition sin (x) and for different values of 3 E 22, 20j, the relative error the estimated solutions using the mentioned methods are plotted in Fig. DD. As the values of 3 become larger, the estimated solutions from C-learning and AW became less accurate. C-learning for 3 2 15 and AW for 3 10 had relative errors of the estimated solutions larger than 10 1_ Furthermore, using the C-learning methodology, order to solve the transport function for 2 5, we had to train models for 3 11, 13, 15, 17, 18, 19, 20 such that the optimized weights for the previous
8

---

Under review as a conference paper at ICLR 2023
(a)
Relative Error
10P
Relative Error
10
10
10"
10-
10
104
3t
47
6x
Sx
10tt
12x
10
15
20
(c)
(d)
Training Error
Relative Error
102
10a
101
10a
10-1.
10
15 Epochs
20
30
Epochs
x10P
Figure 8: Top Panel: The relative errors of the estimated solutions of ID transport and Poisson' 's equations for different values of B and using the C-learning, AW, and Adam, Bottom Panel: the training loss of different methods.
values were used warm initialize the model for the next beta value. Of note, further comparisons with other choices of initial condition presented in Appendix H.3
To further evaluate the effect that the optimization algorithms have on the spectral bias, we compared the rate of training loss decay for wide GDM, wide Adam, AW, and C-learning. An experiment with Poisson's 's equation with C' = 10x showed the training loss of the networks based on Adam had much faster decay than AW (see Fig. [d)).
For the transport function with = 20, the training loss under Adam converged much faster than the loss under AW, C-learning, and GDM Fig. Rp. Another experiment with Poisson's equation when C = 10z showed the training loss of the network based on Adam had a faster decay than AW (see Fig. (d). Although C-learning also showed fast training loss convergence, the relative етror of the estimate from C-learning was large (see Fig. (D). This is not surprising, as the loss landscape for PINNsof PDEs with high-frequency modes is compound KKrishappriaan et (202T) and they contain many saddle points which attract second-order optimization algorithms Dauphin al.] (2014). Our observation consistent with (Markidis) 202I), where they also observed this downside of L-BFGS and recommended using Adam for at least the first few epochs. It is also important to mention that, unlike the other methods, the weight initialization for C-learning was not random. Instead, it was based on the optimal solution achieved for lower frequency modes, which requires extra training.
5 CONCLUSION
In the present study, through the lens of NTKs, we examined the dynamics of training PINNs via GDM. We also showed that under GDM the convergence rate of low-frequency features becomes slower (analogous to under-damped oscillation) and many high-frequency features undergo much faster conversion dynamics (analogous to over-damped or critically-damped oscillations). Thus, the effect of spectral bias becomes less prominent. Moreover, discussed how training PINN via Adam can even further accelerate convergence, and showed it can be much faster than GDM. Although analyzed the theoretical dynamics of convergence by assuming infinitely-wide net- work, our experiments confirmed the estimated solutions obtained from the trained models via GDM and Adam had high accuracy, using finite and practical widths.
REFERENCES
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International

---

Under review as a conference paper at ICLR 2023
Conference Machine Learning, pp- 477-502. PMLR, 2019.
Atam Arya. Introduction to classical mechanics, 1998.
Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning, pp. 404-413. PMLR, 2018
Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In Advances in Neural Information Pro- cessing Systems, volume 32, pp. 476344434444 4772, 2019.
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics- informed neural networks (pinns) for fluid mechanics: A review. Acta Mechanica Sinica, pp- 1-12, 2022.
Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuun Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint arXiv: 1912.0119,. 2019.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex op- timization. Advances in neural information processing systems, 27, 2014.
Juan Du. The frontier of sgd and its variants in machine learning. In Journal of Physics: Conference Series, volume 1229, pp. 012046. IOP Publishing. 2019.
Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
Xinquan Huang and Tariq Alkhalifah. Pinnup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting. Journal of Geophysical Research: Solid Earth, pp- c2021JB023703, 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Ehsan Kharazmi, Zhongqiang Zhang, and George Em Karniadakis. Variational physics-informed neural networks for solving partial differential equations. arXiv preprint 191:::1912.000 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv::4/2.6900 69800, 2014.
Georgios Kissas, Yibo Yang, Eileen Hwuang, Walter R Witschey, John A Detre, and Paris Perdikaris. Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non- invasive 4d flow mri data using physics-ifformed neural networks. Computer Methods in Applied Mechanics and Engineering, 358: 12623, 2020.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics-informed neural networks. Advances Neural formation Processing Systems, 34, 2021.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. Advances in Neural Information Processing Systems, 32, 2019.
Chaoyue Liu, Libin Zhu, and Misha Belkin. On the linearity of large non-linear models: when and why the tangent kernel constant. Advances Neural Information Processing Systems, 33: 15954-15500 15964, 2020.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over- parameterized non-line non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 2022.
Stefano Markidis. The old and the new: Can physics- informed deep-learning replace traditional linear solvers? Frontiers in big Data, pp. 92, 2021.
10

---

Under review as a conference paper at ICLR 2023
Ben Moseley, Andrew Markham, and Tarje Nissen-Meyer. Finite basis physics- informe- neural networks (fbpinns): a scalable domain decomposition approach for solving differential equations. arXiv preprint arXhv:2107.07871, 2021.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural Networks, 12 (1):145-151, 1999.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning, pp. 5301-5310. PMLR, 2019.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physies-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, 367 2020.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339- 2018.
Luning Sun, Han Gao, Shaowu Pan, and Jian-Xun Wang. Surrogate modeling for fluid flows based on physicc-constrained deep learning without simulation data. Computer Methods in Applied Mechanics and Engineering, 361:1 121:2, 2020.
Alexandre M Tartakovsky, Carlos Ortiz Marrero, Paris Perdikaris, Guzel D Tartakovsky, and David Barajas-Solano. Learning parameters and constitutive relationships with physics informed deep neural networks. arXiv preprint arXiv:1808.03321 033998 2018.
John Taylor, Wenyi Wang, Biswajit Bala, and Tomasz Bednarz. Optimizing the optimizer for data driven deep neural networks and physics informed neural networks. arXiv preprint 270402205.004300 2022.
Rohit K Tripathy and Ilias Bilionis. Deep uq: Learning deep neural network surrogate models for high dimensional uncertainty quantification. Journal of Computational Physics, 375:565-588, 2018.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow patholo- gies in physics-informed neural networks. SIAM Journal on Scientific Computing, A3055 A3081, 2021.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:110768, 2022.
A CONVERGENCE OF WIDE PINNS
Proposition A. Let Aoyas and Armnet respectively, be the largest eigenvalues of the Hessians ('Co(w") and V2 (w²).. Consider an infinitely-wide PINN optimized with the following update rule:
W141 Wt v(W())
where is a constant learning rate. Then, provided n O11/(Abmne + drmaae)) the Ht-PL, condition satisfied and the PINN will converge.
Proof. The loss function at time + can be written as:
1 (wi+1 - w.) T TH(W))(w42) - w.)
L(w(41) = L(we) + (We+1 T T [(we) +
11

---

Under review as a conference paper at ICLR 2023
Following the approach of (Wang et al. 2021 let @ be an orthogonal matrix dhagonalizing ((w))) and ((we))l((wee)) With
where H(())) L (w,) (Wang al. 2021
y Vv have the following:
((w++1) [(we) V[ 1/2
|V
Σ Ao.Ve 2
Σ ArcU?
2/2 |VL(w,)]] (Abmax Armace)
[(ww) - V[W
where the Ab and Ar; the respective eigenvalues of the Hessians of La and fyr ordered decreasingly, and the summation taken over the components yf
From here, fixing any bound B such that Abmax Armax
S B, we obtain the following inequality:
((w(+1) & ((we) - /l (1- nB
Witness that 1f = 1/B, we can further simplify this to:
[(wc+1) E [(w,) m/ V
Since our NN satisfies the pt,L,, condition from Eq. due to its width, we therefore have:
((W(41) S (1 ((w))
which concludes the proof.
B HESSIAN FOR POISSON'S EQUATION
Proposition B. Let H be the Hessian of the loss function of an infinitely-wide PINN for the one- dimensional Poisson equation defined by Eq. 4 with u(x) sin(Cz). Then, we have H. O(C*).
Proof. The Hessian of loss the collocation points is calculated following the methods introduced Wang et (202T), where the gradient the loss function is:
OC, Ow
a S
p'u O Og
0
2dz
Ow
Here, w E w, u(x) the target solution (admitting some parameter C), and Uhy (x) is the NN approximation of the output. Assuming that the approximation is a good representation of the actual solution, can be written as (x) u ( where (x) is smooth function taking values in [0, 1], such that (x) - 8 for some > 0 and (=) 5, where have the L norm.
The Hessian of the loss function will therefore be:
JCC. Ow
82 (
8 )
ØI Dw
Ow
where 1 0Cr Ow
and can be calculated to be:
Uww Owe.r
puu dx?
pu
Duw
Un
1,- +
Juee Ow
@ Oxt
dr.
3x
dw
O..
Ozd
Oz
12

---

Under review as a conference paper at ICLR 2023
The above equation contains 3 terms, which call 4, Iz, and Iz respectively. Note that +
01
ov
81,
+
812
01s We calculate these terms as follows: 8m
8h
uuw (Puu 8²u
I1 2
dwdx Ox²
0x2)
0T
duotz
Glum
8uu Dx
gu,
l'
2
dw
Ow
Ox?
Ox²
8x
2 (u 2
Ow
aw
Ox?
Ox2
10
O(w' (1)) Demla)
u(x)) (c((())
8Puy
82u
1
22
dw
Or²
Ox
10
Pew(1)
pun uw
8Pu
=
(1(x))
Ow? + u(xx)
Ow?
2z
Ox?
1
Note that |u(x)] < 1 and by the chain rule |u'(x)) S C. An application of the triangle inequality then yields:
(x)
(x)
|
(((x))
+ u(x)
S
Ow?
Ow?
Ow
Ow?
Using the assumptions (2) get:
8 Uw
82u
S (C² + +2)5.
dr
dx.
Combing both of these, we see concludes the proof.
( Similarly,
012 8w
O(C3) and Sw
O(C*), which
8w
C GRADIENT FLOW FOR GD WITH MOMENTUM
Theorem C. For an infinitely wide PINN, the gradient flow of GDM
ü(x,, w()) ] (x, w())
úxxx, (()) [[(L)(x,,w())) w())
u(x6, w w(t) gxxx) p w()) h(xx,))
m that is analogous to a point mass m. undergoing a damped harmonic oscillation in viscous medium with friction coefficient of u(ai) that is function of o, Furthermore, K is defined as:
K
Kbb Kbr
Krb
K
where:
KoMca.ee V [w(( x) T VV. vor'(w,,'')
Kor V x V (w)(((w..
Krrcaa = v(ul(((((,,'))) V..,(u((,,,'))
are three NTKs associated with the boundary and residual terms. Moreover, let 7 11/2n k; be the i-th eigenvalue of K. and ni 8i Then, the solutions to the gradient flow of the form:
Aie + Aue
A = - + + v22
where A and Ag are constants.
13

---

Under review as a conference paper at ICLR 2023
Proof. Recall that as the NN becomes wider, the norm of the Hessian becomes smaller, such that in the limit as N - oo the norm of Hessian becomes O. One immediate consequence of small Hessian for NN is that its output can be estimated by linear function Lee et al. (2019). The output of NN can thus be replaced by its first-order Taylor expansion:
de (w) = u(w) Iwo + (w wo) Vu(())mo
The update rule for GDM can be written as Du 2019):
+ a(w, W 7/ w ((w).
Wi+1
The discrete updates to the output of NN become (see Appendix C.I
U u + a((" u V (w) (W))V)))))))
In the rest of this section, for simplicity, we will drop the "lin" term. The dynamics of GDM are analogous to the equation of motion of point mass undergoing damped harmonic oscillation (Appendix
mü - pmù V (w) =0
where f (2) is linear function of and ş is the friction coefficient that is related to the momentum term in GDM as such: a m+ Qian (199).. Thus, the gradient flow of u and Dzu](z, w(t)
can be written as:
mü((x,, w()) púl(xo, w(t) Kod (w) w(t) g(xb))
Krdr (D(ul (xr, w()) h(xx))
(C.1)
(x = [D[ri(xx, (()) K Korcarry (u)(,, w(t) g(x)))
K (w) (D) w()) h(x,)).
As mentioned earlier, in wide NNs if the last layer of the network is linear then the tanget kernels are static.
We write Eq. in matrix form as follows:
Üx w()) (x., (() - (xr w())) D(ul (xy, w(()) h(xx,)
((xo (())
u(x6, w w()) g(xb))
m
K
(C.2)
As K is a positive semi-definite matrix Wang et al. (2021) Eq. C-2 can be viewed as a set of inde- pendent differential equations, each one corresponding an eigenvalue A; of the kernel. These give rise to the individual general solutions of the form:
A An + Aiz
A1, - + v
where A₁ and Ag are constants, 7 == and K. Eu Of note, a and are by the user, which in turn define the value of momentum. The choice of values of m and will determine the rate of decay of the above equation.
mo
C.1 LINEAR NN UPDATES
The Taylor expansions of the outputs at time steps + 1 and are:
4 (w) = u(w)) (w)) + (We+1 - wo) Vu(w)lw. lin (w) = u(w)l.. + (we -w)- Vo(w)](]]
Thus, the difference between the outputs in the interval is: u (w) - u, (w) Vu(w) (We++ - wc)
= Vu(w)) (a(w: we-1) L(w:))
where we used the update rule of GD with momentum within the equation. Similarly, we have: (w)
lin We
Jin ui (w) Vu(w)) ww.- wc-1)
and:
u (w) lin (ww) Vu(w))mo (a [wli"(w) with Vu(w) wwo
( (wr)
Jin
-n V (VV)(we))
Thus:
"4t1 (w) (w) v/W.)) Vu(w)) a(uli" (w) Iin (w))
lio
14

---

Under review as a conference paper at ICLR 2023
C.2 RELATION BETWEEN GDM AND DAMPED OSCILLATION
The dynamics of a point mass m undergoing damped harmonic oscillation with a friction coeffi- cient of j are given by:
mi + pv = -V (w((w)
where V [w) is the force field. Qian 1999) showed that Eq. as:
3 in its discrete format can be written
We+Ar Wt- 2w:
Wx - - vww.(.).
At
At
After some algebraic simplifications, the above equation becomes:
At2
W WL+
We m. A (W. W:-D()
m. AAi Vww(
Clearly, m+tai is equivalent the momentum term in GDM, and m+pAr be treated as the learning term. The dynamics of the output of wide network (Appendix C.I under GDM similar.
At
D ANALYSIS OF ADAM FOR BAND-LIMITED FUNCTIONS
Adam is commonly optimizer which can be interpreted as GDM adapted for variance (Kingma & Ba 2014 Balles & Hennig 2018). Thus, it has the same properties as GDM related to the diminishing of the spectral bias because of the momentum term (see Appendix [ for more details). However, because of the adaptive learning rate, it can be expected to be even faster than GDM. Here, we briefly provide more insights into the general behavior of Adam.
Optimizing with Adam, we define:
g(w) = Σ Ve(w;x)/M, xEX
where e = Cb,rs M is the batch size and X is the batch. Furthermore, the pointwise variance of var g(w). We will use subscripts denote entries, so that a; is i-th entry of
g(w) is written a. With V being the true gradient, at each epoch, Adam updates w; with magnitude inversely proportional an estimator of CV (Balles & Hennig 2018
The goal of the training process is to find a weight w which minimizes the loss L across all collo- cation and boundary points from Eq. The role that variance adaptation plays in this speedup is readily seen when the solution to PDE is (or is apel-appro by) band-limited function. A band-limited function on a finite set of frequencies has form of:
u (xx)
Σ an exp(2irixx).
KEK
When using GD, the solutions eventually satisfy the following bound with high probability (Basri et al. 2019 Arora et al. 2019):
2T ΣLEK azk²
N
This indicates that a network optimized with GD will learn better approximations of u(x) if lower frequencies k are present (especially when compared another solution with otherwise identical amplitudes (4)). Conversely, if u(x) is primarily high-frequency then L has very weak bounds. Since our NN is -nfi satisfies the μ-PL, condition V 2 uL of Eq 3 This implies similarly weak bounds on [| and thus VL, (since is constant). Furthermore, convergence requires VL, 0 for all i, if any VL. > the network has yet to converge. However, this is precisely the situation in which Adam accelerates toward the solution fastest as its updates are largest when of V C close to zero. That is, in the presence of high-frequency features resulting in poor bounds on the loss if optimizing with GD, Adam would instead benefit from accelerated convergence. Combined with the inclusion of momentum as discussed earlier, this may provide the framework for it outperforming wide NNs both GDM and Adam can converge to desirable solutions, but that Adam is even faster than GDM (see Section 4
GDM. Our numerical experiments confirm that for sufficiently
15

---

Under review as a conference paper at ICLR 2023
E RELATION BETWEEN ADAM AND DAMPED OSCILLATION
has been observed that Adam yields similar convergence rates to GDM (Kingma & Ba 2014). Here, demonstrate that Adam also has dynamics equivalent to damped oscillator, similar to GDM (Appendix C.2) First, let us briefly recall the traditional update rule for Adam:
A
Vur + m
where my and Pt are defined as:
ni
m4
m. = f1m + (1- B1)9
1 31 1+1
14
1i
V = 3214-1 + (1- 32)9
1 33 21
g(t = -w..(w).
Proposition E. The update rule for Adam can be written as:
W++1 = We P(t)(w= - wL-1) Q
where:
P()) Bi(ve-1 + - (1 1))
(Vr, + +e( 31t)
71 - B1t)
Q(1 = (Vii + e1 -110)
Proof. is straightforward show that the update rule can be written as:
1
17(1 - (1)
W++1 ( c) 3xt)
(vD + c1 Bit) V Vwc((w))
Inserting We = Wi-1 Vi
2 M, into the above equation shows:
We+1 = W/ BI(VV--1 c)(1 11-(3-(t-1 (e(t-17 1))
17 - B2t) (w: w;--) (w/h + c) 32t)
vol((w)). (E.1)
(Vir + +c((1 B1t)
By the definition of P(() and Q(1) we can rewrite Eq. E.1 as:
W++1 W. P(t)(w: w:-1) Q(t)V=L(w:))
(E.2)
as claimed.
Clearly, Eq. F.2 has the same format the GDM update rule. Thus, the weight updates for Adam follows the same dynamics as SGDM, that of oscillatory motion of a point mass under friction (from Appendix C.2).
However, based on the updated value of the second momentum of the gradient of loss function, both the momentum and the learning rate can be updated by Adam at each iteration. Thus, convergence the solution by Adam is much faster compared to GDM.
F DECAY COMPARISON BETWEEN GDM AND GD
F. NUMERICAL EVALUATION OF DECAY RATE FOR GD AND GDM
Here, the dynamics the error decay of GDM and vanilla GD under a large eigenvalue (A1 = 1
and relatively small eigenvalue (Ag 10 10 plotted. For vanilla GD, the training error decays at rate A,t. As shown in Fig. E. for the X2 the training error decays slowly, however is very fast for D (almost immediately dropping to zero). Thus, components of the target function
16

---

Under review as a conference paper at ICLR 2023
that correspond to the large eigenvalues will be learned much faster. These are the eigenvalues corresponding to lower frequencies in the target function, so clearly the network suffers from spectral bias under GD.
In contrast, learning under GDM, the training error for A decays following under-damped oscilla- tion (the red curve in Fig. E.Tp). while the training error for A2 follows over-damped oscillation (the blue curve in Fig. E. p). The training error of both eigenvalues, the large λ and small X2, decay at
approximately the same time, The components of the target function that correspond to the larger
eigenvalue are thus not learned any faster, so the effect of spectral bias is minimized.
10
lambda
lambda 1e]
075
0.8
Anpiitute
Amplttude
025
0.00
-0.25
-0.50
a]
-0.75
0.0
0
20000
42000
60000
80000
100000
50
300
150
200
250
Epochs
Epochs
Figure F.1: The comparison of the error decay for small and a large eigenvalue, a) learning decay
during GD optimization process, b) learning decay during GDD optimization process.
F.2 THE EIGENVALUES OF KEs AND Kyy
As discussed in Scction 3.2 under GD optimization, the total training error is evaluated based on e-xt, where K; are the eigenvalues of K that encapsulated Kss (NTK for boundary and initial data points) and K,r (NTK for collocation points representing the PDE). Thus, the convergence rate of the training error is evaluated based on the eigenvalues of Kλ6, and Ky, together. Consequently, there might be a discrepancy between the absolute value of cigenvalues of K5a, and Kyy, meaning that eigenvalues of one the two matrices be much larger, and the convergence rate of the training error for them becomes much faster. Hence, the network becomes biased to learn the components corresponding to those eigenvalues first.
In Fig. for Poisson's 's equation when C 15T, the eigenvalues at initialization for K (lett panel) and K,y (right panel) (in descending order) are plotted. Clearly, the eigenvalues of K,, that represents the PDE, are much larger than the eigenvalues of Kss that represents the initial and boundary data points (bcs). Hence, not surprisingly, in GD where the training errors decay based on e- the network learns the PDE general form first and becomes slow in learning the bes. This bias in learning the PDE first can be minimized while we are implementing GDM. In fact, plotting the terms of the loss function (PDE and bes terms) also confirms that under GD the bes are learnt slowly (see Fig. F2 and the rate of decay loss for bes (green curve) is much slower than the rate of decay of loss GDM (blue curve). Thus, clearly, we that under GD optimization, the
17

---

Under review as a conference paper at ICLR 2023
10
Eigenvaluss
Eigen/alues
10
10-13
10
10
10
101
10
10:1
100
Counts
Counts
Figure F.1: Left panel: Figenvalues for Kss. Right panel: Eigenvalues for K
GD bcs
105
GUN
GD
GDN roe
102
Loss
10-1
10-4
0
10000 20000 30000 40000 50000 Epochs
Figure F2: The loss function during training via GDM and GD for the PDE term of the loss as well as the boundary/initial term plotted.
network is much slower in learning the bes (compared to the PDE term). As explained earlier, this not surprising as the eigenvalues KMs are much smaller than the eigenvalues of K,y-
These observations are very insightful as they reveal significant difference between PINNs and regular neural networks. Perhaps to explain the difference, of the solution via GD for completely data-driven case. We trained a fully-connected forward neural network with the same architecture as the above PINN. The training data-sct contained 3000 synthetic data points generated based the solution of Poisson's equation, and we used the mean squared error loss function. The solutions after 5000 and 50000 epochs are shown in Fig. F3 As the fully-connected network has no peyssic-informan the correct solution. Despite this, it still exhibits spectral bias, as the estimated solutions at both 5000 and 50000 epochs represent low-frequency sinusoidal forms. However, unlike PINN there is vertical shift in the solutions. Clearly, the physics-informed regularization term helps significantly resolve the classical spectral bias (dealing with high-order frequency target functions), learning the correct sinusoidal shape faster. In return, the solutions are worse at satisfying the boundary condition values. This because the eigenvalues of the boundary kernel represent tigh-freeuency modes.
useful investigate the evolution
regularization term, it has difficulties estimating
G LOss FUNCTION OF EQUATIONS
G. Loss FUNCTION FOR POISSON'S EQUATION
For Poisson's 's equation:
f(x) CC sin(Cx), æ E [0,1]
g(x) 0, 0,1
((x)= sin(Cx:) is used the exact solution. The corresponding loss function is written
No
N-
(82
2
L(w): = No
(ū(zs', - g(xx)))) + N, Σ
Or (x (xr", w-)- f(xx²)
18

---

Under review as a conference paper at ICLR 2023
Exanc
Prodicted
Predicted
0.75
0.75
0.00
0.00
-0.75
-0.75
-1.00
0.0 02
0.1
06
3.0
1.0
0.0
0.2
0.4
0.
10
Figure F.3: The completely data-driven solution of the Poisson equation based on GD optimization process. a) after 5000 epochs, b) after 50000 epochs.
where à is the output of the network.
G.2 Loss FUNCTION FOR THE TRANSPORT FUNCTION
Using the methods of characteristics (Evans 2010), the transport function has a well-defined ana- lytical solution: u(x,f f) g(x Bt). This used as the exact solution. The corresponding loss function is written as:
No Σ
Nr N₂ Σ
(Oū(x,' 2 ", w) +
Hi(xrr Lb", w)
2
((w): = No.
(ū(xs", - 9(zo0))²
Ōt
Ox
where it is the output of the network.
H EXPERIMENT PLOTS
H.1 POISSON EQUATION
1.0
10-2
Pointisse Eroor
05
0.0
10
-10
0.0
0.8
0.0
0.2
0.4
Figure G.1: The estimation of the solution of the Poisson's 's equation for C' = 15xr, using the vanilla GD algorithm after 180000 epochs.
19

---

Under review as a conference paper at ICLR 2023
H.2
REACTION-DIFFUSION EQUATION
A reaction-diffusion equation contains reaction and diffusion term. Its general form is written as:
du d1
vAu+ f(u)
(G.1)
where u(x,() f) the solution describing the density/conncentatiion substance, A is the Laplace operator, is diffusion coefficient, and f(u) is smooth function describing processes that change the present state of (for example, birth, death or chemical reaction). Here, we assume a one- dimensional equation, where f(u) pu(1 - u). For p independent of x and f, and with the defined fu),, Eq. G.I, can be solved analytically (Evans) 2010). To estimate the solution using PINNs, the loss function for the ID
PDE written as:
No
1 ((w) = No
((aa, t/ w) g(xt))²
1=1
(op 26 b,
w)
a1 bo w) (a(x);,))( 0(x,, tQ))
Ol
Ox2
i=1
For consistency with Krishnapriyan al. 2021 used N, 1000, and No 100. We also chose [m-a]² and u(0,t) t) = u(2T, () respectively. #²/2 Similar to the previous experiments, for larger choices of and p the model trained via vanilla GD (after 85000 epochs) had difficulty converging the solution (Fig- G.3, left panel). However, models trained via GDM and Adam (after 45000 epochs) could provide solutions with low emror
the initial and boundary conditions u(x, 0) exp
The plots estimated and exact solutions for the three algorithms when v 3 and p = 5 are shown in Fig. G.2
(a)
(b)
(c)
6-
6
0.5
0.5
4-
4-
0.
0.0
2-
2
-08
0
.10
-10 0
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0 0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
1.0
1.0
1.0
4
4-
4.
0.0
0.
0.0
2.
oos
.0s
2
0.4
0.6
110
0.0
0.2
0.6 0.8
1.0
0.0
0.2 0.4
0.6
1.0
1.0
6-
0.8 6-
0.06 6
0.01
4
0.006
2
02 2-
0.02 2
0.002
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure G.2: ID reaction-diffusion equation for = 3 and 5. The solutions are obtained by training a 4-layer network with width=500 at each layer. Top panels: The exact (analytical) solution. Middle panels: The estimated solution. Bottom panels: The absolute difference between the exact and estimated solutions. (a) Vanilla GD, (b) GDM, (c) Adam.
20

---

Under review as a conference paper at ICLR 2023
LEPOS SOOM
177
Retative Emer
1074
(0.10.3300.1
0.7)
(7.90
Epodas
Figure G.3: Left Panel: The training losses of the network, when v = 9 and p 7), via AW, C- learning, wide GDM, and wide Adam are plotted. Right Panel: The relative error of the estimated solutions for different values of y and are plotted.
We further compared the solutions the wide Adam, C-learning, and AW. Similar to the previous experiments, for higher-frequenyy modes Adam showed superior results. The relative errors of the estimated solutions for different values of and were computed (Fig. (left panel)). Furthermore, when / 9 and - = 7), observed that the loss function under GDM and Adam had much faster decay than under AW. But, the loss under L-BFGS decayed as fast as under Adam. We reasoned as why in Section
H.3 TRANSPORT FUNCTION: FURTHER COMPARISON WITH DIFFERENT INITIAL CONDITIONS
We implemented different initial conditions for the transport function and ran several experiments. For the initial condition of tanh(x), when 3 both C-learning and AW estimated solutions of showed large relative errors the order of 10 However, the estimated solution of wide network
trained via Adam had an acceptable relative error on the order 10 2 (Fig. G.4). The estimated
solutions for all three optimizers when 3 = 6 are shown in Fig. G.5 All networks were trained using a 4-yeyer network, and AW and Adam both had width 500 neurons. The estimated solutions are based training the model for 85000 epochs.
loamning
1.10"
Figure G.4: The relative errors of the estimated solutions of 1D transport function, for different values of 3 using the curriculum learning method (red dashed line), and our approach (blue dashed line) are plotted.
21

---

Under review as a conference paper at ICLR 2023
4
2
-0.4
-0.4
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
6
0.4
0.4
4
x
x
0.0
2
0.0
0.2 0.4 0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
The estimated solutions of the transport function based on the initial condition of ta)h(z) when 6: a) The exact solution b) Estimated solution of wide network trained via Adam c) Estimated solution of C-learning d) Estimated solution of AW
Figure G.5:
For the initial condition of sin(x) cos(2), C-learning had small estimated errors (on the order of 10-2) for 3 < 20, however for higher values of 3 the error grew larger (Fig. G.6 AW had errors on the order of 10-2 for 3 < 10, however the error began to grow for larger values of Adam had errors the order of 10-2 even up 8 25 (Fig. G.6). The estimated solutions based on C-learning, AW, and wide Adam for B = 25 are shown in Fig. G.7 All networks were trained using 4-layer network. The estimated solutions based training the model for 85000 epochs.
102
Relative Error
101
100
0.5
10-1
10-2
2
5
10
15
20
25
B
Figure G.6: The relative errors of the estimated solutions of transport function, based on the initial condition of sin(2) cos(x), for different values of 3 using the curriculum learning method (red dashed line), and our approach (blue dashed line) are plotted.
22

---

Under review as a conference paper at ICLR 2023
(a)
(b)
6
0.4
0.4
4
0.0
0.0
2
-0.4
-0.4
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(c)
(d)
6
0.4
0.4
4
4
X
X
2
2
-0.4
0
0.0
0.2
0.4
0.6
0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure G.7: The estimation of the solution of the transport function based on the initial condition of sin (x) cos(z) when 25: a) The exact solution b) Estimated solution of wide network trained via Adam c) Estimated solution of C-learning d) Estimated solution of AW
23


Reviews:
[{"review_id": "nfC6haACNe", "reviewer": "Reviewer_3BS7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "PINNs often fail to converge. The authors propose to use neural tangent kernel to solve this issue, and propose to use the Adam optimizer. They empirically demonstrate this to work on problems with high-frequency features.", "summary_of_the_review": "PINNs and their convergence behaviour is an understudied area where any insights can be very valuable to future research. The present work attempts to do some fundamental work to investigate PINNs. It is good to do some theoretical analysis, but novelty is limited.", "strength_and_weaknesses": "# Strengths\n\n- for the first time, PINNs under SGDM, and Adam are analyzed, and their relation to solving spectral bias is discussed\n- theoretical analysis\n\n# Weaknesses\n\n- it has been shown that the Adam optimizer works better than vanilla SGD in many cases (unrelated to PINNs), why would you assume that it does not work for PINNs?\n- the paper aims to solve spectral bias. However, the phenomenon of spectral bias is not clearly defined or introduced in the paper\n- figures are of bad quality\n\n\n\nminor issues:\n- spelling mistakes (week -> weak)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mX6zXKitPZ", "reviewer": "Reviewer_droF", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the spectral bias phenomenon when using momentum-based methods to optimize physical-informed neural networks. The authors simplify the neural networks as a linear model (i.e., NTK), and prove that the continuous approximation of GDM converges to the high-frequency solution faster than SGD. The authors also investigate the convergence of Adam over PINN through a specific model. Experiment results are provided to show that Adam and SGDM converge faster than SGD when solving the PINNs.\n\n\n", "summary_of_the_review": "Based on the evaluation above, I believe that this paper is below the bar of ICLR. The paper can be improved by refining the writing and the theoretical results.", "strength_and_weaknesses": "**Weakness**\n\n1. The biggest weakness of this paper is that the analysis does not utilize the property of physical-informed loss. In other words, the same results also hold for image classification tasks as the analysis does not rely on which loss is used. This makes the result hard to unveil any information specific to the NN-solving-PDE task.\n\n2. There are some overclaims in the paper. It is said that \"using the SGD with momentum (SGDM) optimizer can reduce\nthe effect of spectral bias in the networks\", but the paper only proves the result for a continuous approximation of GDM with no approximation error provided.\n\n2. The paper is hard to follow, and many places lack explanations. For example, at the end of page 3, it is said \"the norm of the Hessian $\\mathcal{H}(w_t)$ is order $\\mathcal{O}(C^4)$\". However, the parameterization of $u(w,x)$ is not even provided (although it is explained in the appendix). On page 4, it is said that \"the decay rate analysis becomes more involved as Eq. 7\". However, there is no Eq. 7 in the main text. Also, why does using SGDM lead to faster convergence for high-frequency components than SGD? How do we know it from Theorem 1?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eMqnVeE9szI", "reviewer": "Reviewer_Ntw5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors study the convergence of infinitely wide PINNs on PDEs that have high frequency modes to examine the effects of spectral bias. Their analysis proceeds using NTK theory of convergence. They show that while SGD can learn high frequency modes that the learning rate has to be small. Further they show that SGD with momentum as well as ADAM reduces the effects of spectral bias and speed up the learning. The paper also features empirical results on the poisson equation, reaction-diffusion equation and transport function.  ", "summary_of_the_review": "Overall a good paper, easy to understand what the authors aimed to do. Seems like an important and useful result and ties in considering the spectrum of the physics problem in consideration with convergence guarantees of the network.", "strength_and_weaknesses": "-The authors do a good job of introducing the reader to spectral bias in a general context in addition to spectral bias in the case of PINNs.---Very clear and concise analysis for section 3.1 and 3.2.  \n-Nice experimental results, they seem to confirm theoretical convergence results. \n\n\n-The authors could flush out section 3.3; maybe move some of the material in the appendix for convergence results into main paper and list equations in format similar to 3.1 and 3.2\n-Would be nice if authors concocted a numerically tractable ay to extract exact eigenvalues of Hessians to see how it matches with theory.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "60xGVaHX3oJ", "reviewer": "Reviewer_3LnV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper uses the NTK to discuss the effect of momentum on the training success of PINNs. It is shown that momentum (or adaptive momentum) can improve the convergence speed compared to standard SGD-based training, using both theoretical and empirical arguments.", "summary_of_the_review": "An interesting paper that studies the effect of momentum on the training success of PINNs. While interesting and novel, the experimental evidence is not yet fully convincing.", "strength_and_weaknesses": "The paper is well-written and covers an interesting and timely topic. While the authors do not propose a novel method, their experimental evidence is in favor of their hypothesis. The supplementary material contains additional information that helps shedding light on the interplay between the spectral properties of the solution, the selection of the optimizer, and training success.\n\nWhat prevents me from giving a better recommendation is that some of the experimental evidence is unconvincing, and that -- I believe -- alternative hypotheses first need to be ruled out. Indeed, judging from the results, it appears as if spectral bias is not the problem underlying convergence problems. Looking at, e.g., Fig. E.2 or Fig. 2, we see that SGD manages to learn the high-frequency components, but with a shift, trend, or offset added to it. Therefore, the spectral bias, which suggests that the high-frequency components are learned later, seems not to be a valid explanation. As an alternative hypothesis, it may be that learned solutions $u$ satisfy the PDE. While it is known that trivial (zero) solutions are attractive for PINNs (arXiv:2109.09338,arXiv:2203.13648), I assume that the same holds for non-trivial solutions that satisfy the PDE. Essentially, I argue that the solution $u$ provided by the PINN satisfies the PDE, but not the initial/boundary conditions. This needs to be confirmed or rejected by investigating the loss term $\\mathcal{L}_r(w)$ throughout training, and not only the compound loss $\\mathcal{L}$. If this former loss is small, then the PDE is at least approximately satisfied.\n\nFurther, the results in Fig. E.4 seem questionable, as a standard neural network should be capable of learning this simple function. Please correct me if I am missing something important.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject