Fourier PINNs: From Strong Boundary Conditions to Adaptive Fourier Bases
Authors: Madison Cooley
Da Long
Robert Kirby
Shandian Zhe
Venue: Submitted to ICLR 2023 (2023)
Keywords: Physics Informed Machine Learning
Fourier Analysis
Scientific Machine Learning
Partial Differential Equations

Abstract: Interest in Physics-Informed Neural Networks (PINNs) is rising as a mesh-free alternative to traditional numerical solvers for partial differential equations (PDEs). While successful, PINNs often struggle to learn high-frequency and multi-scale target solutions—which, according to prior analysis, might arise from competition during optimization between the weakly enforced boundary loss and residual loss terms. By creatively modifying the neural network architecture, some simple boundary conditions (BCs) can be satisfied exactly without jointly optimizing an additional loss term, thus avoiding the aforementioned competition altogether. Motivated by this analysis, we first study a strong BC version of PINNs for Dirichlet BCs and observe a consistent improvement compared to the standard PINNs. We conducted a Fourier analysis and found that strong BC PINNs can better learn the amplitudes of high-frequency components of the target solutions. While BC PINNs provide a promising improvement, constructing these unique architectures is an intricate process made difficult (if not impossible) by certain BCs and domain geometries. Enlightened by our analysis, we propose Fourier PINNs—a simple, general, yet powerful method that augments PINNs with pre-specified, dense Fourier bases. Our proposed architecture likewise better learns high-frequency components but places no restrictions on the particular BCs. We developed an adaptive learning and basis selection algorithm based on alternating NN basis optimization, Fourier and NN basis coefficient estimations, and coefficient truncation. This schema can flexibly identify the significant frequencies while weakening the nominal to better capture the target solution's power spectrum. We show the advantage of our approach in learning high-frequency and multi-scale solutions in a set of systematic experiments. 

Under review as a conference paper at ICLR 2023
FOURIER PINNS: FROM STRONG BOUNDARY CONDI- TIONS TO ADAPTIVE FOURIER BASES
Anonymous authors Paper under double-blind review
ABSTRACT
Interest in Informed Neural Networks (PINNs) is rising as a mesh-free alternative traditional numerical solvers for partial differential equations (PDEs). While successful, PINNs often struggle to learn high-frequency and multi-scale target solutions-which, according prior analysis, might arise from competition during optimization between the weakly enforced boundary loss and residual loss terms. By creatively modifying the neural network architecture, some simple boundary conditions (BCs) can be satisfied exactly without jointly optimizing an additional loss term, thus avoiding the aforementioned competition. Motivated by this analysis, we first study strong BC version of PINNs for Dirichlet BCs and observe a consistent improvement compared to the standard PINNs. We con- ducted Fourier analysis and found that strong BC PINNs amplitudes of seeh-freuenccy components of the target solutions. While BC PINNs provide improvement, constructing such architectures is an intricate process made difficult (if not impossible) by certain BCs and domain geometries. Enlightened by our analysis, we propose Fourier PINNs method that augments PINNs with pre-specified, dense Fourier bases. Our pro- posed architecture likewise better learns high-frequency components but places no restrictions on the particular BCs. We developed an adaptive learning and basis selection algorithm based on alternating NN basis optimization, Fourier and NN basis coefficient estimation, and coefficient truncation. This scheme can flexibly identify the significant frequencies while weakening the nominal to better capture the target solution's 's power spectrum. We show the advantage of our approach in a set of systematic experiments.
better learn the
simple, general, yet powerful
Introduction
mhysics-informet neural networks (PINNs) (Raissi et al. 2019a) are emergent mesh-free approaches
to solving partial differential equations (PDH)s. They have shown successful in many scientific and
engineering problems, such as bio-engineering (Sahli Costabal et al.) 2020 Kissas et al. (2020). fluids
mechanics (Raissi et al. 2019b] Sun et al., 2020: Raissi et al. 2020). fractional PDES (Pang et al.,
2019b; 2020), and material design (Fang & Zhan 2019 Liu & Wang 2019). The PINN framework
uses neural networks (NNs) to estimate PDE solutions, in light of the universal approximation ability of the NNs. Specifically, consider PDE of the following general form,
F(u] = f(x) (x E Q), u(x) g(x) (x 02)
(1)
where F is the differential operator for the PDE, 12 the domain, 02 is the boundary of the domain. To solve the PDE, the PINN uses a deep neural network (x)x) to represent the solution u, samples
N collocation points (x {x from S and M points { from 0n, and minimizes the loss,
LM
0 argmin (0) + (0)
where (0) M7 1== T (x0(x)) g( s(x2)) is the boundary loss to fit the boundary condition, and L, (0) N Em (F())(x - fxx)) 2 is the residual loss to fit the equation.
Despite their success, the training of PINNs is often unstable, and the performance can be poor from time time, especially when solutions includes high-frequency and multi-scale components.

---

Under review as a conference paper at ICLR 2023
From the optimization perspective. Wang al. (2020a) pointed out that due to the imbalance of the gradient magnitudes of the boundary loss and residual loss (the latter is often much larger), the training can be dominated by the residual loss and hence underfits the boundary condition. Wang et al. (2020c) confirmed this conclusion from a neural tangent kernel (NTK) analysis on the training behaviors of PINNs with wide networks. They found the eigenvalues of the residual kernel matrix are often dominant, which can cause the training to mainly fit the residual loss. On the other hand, Rahaman et al. (2019) found the "spectrum bias" in learning standard NNs, namely, the low frequency information can be easily learned from data but grasping the high frequencies is much slower and harder. To alleviate this issue, (Tancik et al. (2020) proposed to randomly sample a set of high frequencies from a large-variance Gaussian distribution, and use these frequencies to construct random Fourier features as the input to the subsequent NN. [Wang et al. (2021) gave a justification of this approach via NTK analysis, and used multiple Gaussian variances to sample the frequencies and construct Fourier features so as to capture multi-scale information in the PINN framework. While effective, the performance of this method is sensitive to the number and scales of the Gaussian variances, which are often difficult to choose because the solution is unknown apriori.
The contributions of our work are as follows:
Motivated by the prior analysis from the optimization perspective. we investigated a strong boundary condition (BC) version of PINNs for simple Dirichlet BC's. This PINN variant satisfies the boundary conditions exactly (through specific NN architecture construction) and does not require boundary loss term during optimization-tiee avoiding the competition with the residual loss term. We observed significant improvement upon the standard PINN, especially for higher frequency problems.
Different from the previous investigation, we conducted a Fourier analysis on our strong BC PINNs using specific boundary function the function that constrains the NN output to satisfying the boundary conditions exactly. Through Fourier series and convolution theory, we found that, interestingly, multiplying the NN with the boundary function enables faster and more accurate learning of the coefficients of higher frequencies in the target solution. By contrast, standard PINNs exhibit hardship in capturing correct coefficients in the high frequency domain.
Enlightened by our analysis, we developed Fourier PINNS simple, general. yet powerful extension of PINNs, independent of any particular boundary condition (unlike Strong BC PINNs). The solution is modeled as an NN plus the linear combination of a set of dense Fourier bases, where the frequencies are evenly sampled from large range. We developed an adaptive learning and basis selection algorithm, which alternately optimize the NN basis parameters and the coefficients of the NN and Fourier bases, meanwhile pruning useless or insignificant bases. In this way, our method can quickly identify important frequencies, supplement frequencies missed by the NN, and improve the amplitude estimation. We only need to specify a large enough range and small enough spacing for the Fourier bases, without the need for worrying about the actual number and scales of the frequencies in the true solution as in previous methods. All these can be automatically inferred during training. and multi-
We evaluated FourierPINNs in several benchmark PDEs with frequency solutions and solutions that couple figgufryyuency components with plain func- tions. In all the cases, Fourier PINNs consistently achieve reasonable and good solution errors, e-g., 10 the strong BC PINNS were much worse than our method (though better than the standard PINNs). The PINNs with random Fourier features often failed under a variety of choices of the Gaussian variance number and scales. The performance is highly sensitive to this choice. We also tested PINNS with large boundary loss weights ( (Wight & Zhao 2020), and with an adaptive activation function (Jagtap et al. 2020). FourierPINNs consistently outperformed both methods.
10-2 As comparison, the standard PINNs always failed, while
2 Strong Boundary Condition PINNs
According to prior analysis (Wang et al. 2020a c) the instability of PINNs is likely from the competition between the weakly-enforced boundary loss and the residual loss during optimization. To sidestep this issue, a natural idea is to design a surrogate model that satisfies the boundary condition, and hence the training no longer needs weakly-forced boundary loss. To this end, consider
2

---

Under review as a conference paper at ICLR 2023
simple Dirichlet boundary condition for 1d problems,
E b], u(a) u(b) 0.
(3)
To solve the equation (z) f(x) construct surrogate model by Ug (2) (xx))o(xr)
(4)
where (1: is neural network parameterized by 0, and
(xx) a) xr)
(5)
a boundary function. We can see that by construction [a{aa (b) O, namely, the boundary condition (3) is automatically satisfied. To estimate the parameters of i. we only need to minimize the residual loss (0) (see (2). The construction is straightforward. Similar strategies has been used in the recent work (Lu et al. (021b) to incorporate hard constraints for topology optimization and inverse design, and early works, e.g., (Lagaris et al. 1998; 2000). However, here we want to investigate if such construction can benefit solving forward problems, especially when the solution includes high frequency information.
Shooort PAC
Shrong HW
Standeed H-M
String RC MAR
To this end, we examined the strong boundary con- dition (BC) PINN (4) on a ld Poisson equation,
Au = f(x), æ E [0, 2m].
(6)
and a 1d Allen-Cahn equation,
(a) ID Poisson
(b) 1D Allen-Cahn
Uaz + uuu² - 1) = f(x), x E [0, 2x).
(7) Figure 1: Solution error with frequency k.
both subject to the Dirichlet condition (3) where a = 0 and = 2m. We consider the fabricated solution in the form of uax)= = sin(kx). Therefore, by varying k, can examine the performance the strong BC PINN when the solution includes different frequency information. We compared against the standard PINN, which explicitly incorporates the weakly-enforeed boundary loss and residual loss. We used the same NN architecture for the strong BC PINN and the standard PINN, which includes two hidden layers, 100 neurons per layer, and tanh activation. Both methods used the same set of 10K collocation points randomly sampled from the domain. We varied A: from 2 to 34, and ran each method for five times. The average relative L2 error and the standard deviation is reported in Fig. [ As we can see, the strong BC PINN consistently outperforms the standard PINN, including bigger choices of k. That means, the strong BC PINN can more accurately capture the high frequeney information in the solution.
3 Fourier Analysis of Strong BC PINNS
Now, we attempt to explain why the strong BC PINNS can improve upon the standard PINNs. We look into this problem via Fourier analysis. A plain NN model is known to easily capture the low frequency information yet difficult/very slow grasp the high frequency information (Rahaman ct al. 2019). We first investigate if physics-ifformed NNs exhibit similar issues. Fig. 2a shows typical example of the frequency spectrum (obtained from discrete Fourier transform), which comes from the standard PINN trained to solve the ID Poisson in Sec. 2 with true solution u(x) (i1)). We can see that although the PINN is able identify the target frequency, its coefficients for higher frequencies are too big, i.e., heavy tails. This leads to the inferior solution accuracy. By contrast, from the frequency spectrum of the strong BC PINN (Fig. 2a). we can see that not only does the strong BC PINN captures the target frequency (as accurate as the standard PINN), its coefficients for higher frequencies decay much faster, and hence the influence of unnecessary high frequencics are greatly weakened (or even excluded), leading to much better accuracy.
To understand why the boundary function (5) (after multiplied to the NN in 44 can help obtain better coefficients for high frequencies, we first seek to represent the boundary function (5) with an infinite Fourier series (synthesis analysis).
B(x) (x)
Σ
B|n]] inz
3

---

Under review as a conference paper at ICLR 2023
Standard Strong BC FINN
10a
Biw -v)
Ampltuat
Amplitute
Nivl
10a
100
10
Frequency
Froquency
(a)
(h)
Figure 2: The frequeney spectrum of the learned solutoin surrogate and the convolution operation in the Strong BC PINN with w = 15. both cases, the ground-truth -ouund-truth solution (15x..
where indicates the imaginary part, and we have
1 / x(2m x) in -n
2 n
B(n) 2x 2x
Here expand the boundary function Bxx) to be periodic (i.e., duplicating its definition in [0, 2x] to other intervals) and the period is 2x. We can now obtain the Fourier transform of (x))),,
B(w) Σ
[[n]
2T
where (() is the Dirac delta function. Since [B[n]] x 1/n², we can see that the amplitude of the frequencies of (ar) decays quadratically fast with the increase of the absolute frequency value.
We then look at the frequency spectrum of the surrogate model ((x) in (4) Since it is the product of
(xx) and ((x), we can use the convolution theorem (McGillem & Cooper 1991 to obtain
(w) B B+ J
B(w (iw-v)((yje
where means convolution, and N is the Fourier transform of the neural network. Suppose w > we know that ((w v) obtained by first reflecting B(v) about the y-axis, and then move w to the left. Since B is symmetric (see (8),, we actually just move the original function to the left. Then, we integrate the whole (1)) weighted by B(w - v). The larger the frequency the more left B is moved to so to obtain ((w v). That means, the more tail (right) part of the B is used to weight (v) during the integration. Since the frequency coefficients of B decays quadratically fast ([D[]] x 1/n²), the larger the portion of the tails used, the smaller the integration result. That means, with the help of the boundary function B, the corresponding amplitude ū(w)) decrease fast when w increase. See Fig. 2b] an illustration. The same conclusion applies when we consider < 0 and |w|] increase. In this way, during the course of training, the boundary function pushes the surrogate model (4) to inhibit/weaken irrelevant high frequency components so as to reduce flat tails and to better capture the frequency spectrum.
Fourier PINNs
Based on our analysis (Sec. 3). the bottleneck of the standard PINN model is the inefficiency of learning the gigh-frequency components of the solution the estimated amplitudes are overly big. While the strong BC PINNs alleviate this issue by multiplying the NN with boundary function to enable faster decay of the amplitudes, the benefit can still be restricted since the decay rate is up to the boundary function itself rather than is driven by the ground-truth solution. As can be seen from Fig. [ k grows, the performance of the strong BC PINN degraded, implying that is not powerful enough. In addition, the strong BC PINNs restricted particular boundary conditions, e.g.. the Dirichlet conditions (3). which severely limit its applicability. To overcome these problems, enlightened by analysis and propose Fourier PINNs, simple, general yet powerful method, which can handle various boundary conditions like the standard PINNs. Specifically, in order flexibly and comprehensively capture the frequency spectrum, we first introduce set of dense frequency candidates, {wn 1Q ===1 which are evenly sampled from large range [-K, K] with spacing A. We then augment the NN with set of Fourier bases,
ū(x) No Σ
aj cos (2Ttwn)rr) + by simT2w....).
(10)
4

---

Under review as a conference paper at ICLR 2023
Since the NN can be viewed as the linear combination of set of nonlinear bases, the surrogate can be further written as
Σ H+2Q
x(x)=
j=1 wy wyy (x)
(11)
where for 1 < < H. Φj is an NN basis, namely, the j-th output of the second last layer, for H #+133 < H Q Oj (x) = cos(2 (n = H), for H + +Q S S H + +Q, ()( sinsniiiiaaaa 2TT (n
H Q), and wy is the coefficient of each basis. We then use loss similar to PINNs (see to estimate the parameters,
I( w) (sbb,, w) (zb, w) + ||wl
M (m(xt) x(xx))) 2 + N
1 - (x s(x)))
a w/
(12)
where zb denotes the parameters of the NN bases, ||w| is the square norm to regularize w, and is the regularization strength.
(W),... ..... 2x+22) all the basis coefficients,
The ground-truth frequencies can be much fewer than the candidates. From our analysis, to improve PINNs, it is critical to prune or inhibit useless frequencies, e.g. those that are overly big (see Fig. 2ap. In order to flexibly identify the significant and prune or weaken the insignificant frequencies, develop adaptive learning and basis selection algorithm. Specifically, we jointly optimize &b and w to a stage. Then we fix all the bases (i.e., with &b fixed), and iteratively truncate the basis coefficients with threshold (e.g.. lwyl < 10-3), and re-update the remaining coefficients. We repeat this procedure until a maximum number of iterations are finished.
When the PDE operator F is linear, the updates of can be simply done via least square ridge regression, because according to (II). F-)
Ej wyFlo,l() is linear in w. and the loss has the form w) = W + M EM [E wy by (x s(x}))
1
Note that the method of alternating least square and truncation has achieved great success in sparse linear regression for data-drive operator selection and governing equation discovery (Brunton et al.] 2016) Chen et al. (202T). We use similar strategy rather than the conventional L1 regularization for simplicity and efficiency. When the operator F is nonlinear yet can be decomposed as
N EE W, F(o) -
FHI +
(13)
where G is a linear operator and S is the nonlinear part, e.g., in the Allen-Cahn equation 77) Glu] = Uzz and S(u] = u(u² 1), we can still conduct efficient least square ridge regression to update w. Specifically, we evaluate Slū] with current on the collocation points, fix the results, and combine with the source function values on the collocation points to construct the regression output in the residual loss,
N L, L, N - -10))(x))
(14)
where A = (x!) + ((x). Here, since f; is determined from the current w but fixed constant in the least square estimation, the update of is in essence a type of fixed point iteration. In experiments, we found this method is efficient and robust. When F does not include any linear operator (which might be rare), we can use any continues optimization algorithm, e.g., L-BFGS, to update Finally, our adaptive learning and basis selection is summarized in Algorithm []
While we present our method in the single input case, it is straightforward to extend to multiple inputs. Consider [ : ]bح] for example. We the same set of frequencies to construct Fourier bases sin(2 11,,... cos (200)g))) ), sin (2TTw))))) )}, and for each single input, ((x)) = (2ss( cos(2 (2Twgaa))) sin (inTwrgg)))) Then apply cross- (xx2) 222) (2
product to obtain the Fourier bases for x. The surrogate model is given by
(15) where vec(.) is the vectorization, and ,3 are the coefficients of the Fourier bases. Note that the number of bases grows fast with the increase of input dimension, and can be too large for high input dimensions. In that case, we can pre-prune the bases according to the total frequency or re-organize the coefficients & into tensor matrix, and introduce a low-rank decomposition structure to parameterize B3, so as to save the cost (Novikov et al. 2015 Ye et al. 2018).
((x) = (x) + ,3 vec (6(2)))((2))
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 FourierPINN(C, T, A, E)
Initialize &b and w randomly.
Run ADAM for T epochs to jointly optimize up and w. repeat
for A iterations do
Fix vb. estimate w with least square ridge regression.
For each wi,, if |uy] < 6, prune the corresponding basis (-) and delete wy from w.
end for
Run ADAM for T epochs jointly optimize ap and w.
until Σ iterations are done Run L-BFGS until convergence
5 Related Work
There a large body of related work. Due to the space limit, we briefly discuss the most related lines of work. PINNs have shown promise as a computationally efficient alternative to expensive mesh- based numerical methods such as finite elements and finite volumes (Reddy 2019). PINNs (typically) work by seftly constraining neural networks to abide by physical constraints given well-studied
governing equations - a strategy that is beneficial particularly in the small data regime (Rackauckas) al. 2020 Lu et al. 202 Weinan et al.
2017 Raissi et al. 2019a] Hennigh et al. 2021). Although
PINNs have many success stories, e.g., (Raissi et al. 2020; Chen et al.) 2020 Jin et al. 202
Sirignano & Spiliopoulos 2018 Zhu et al.) 2019 Geneva & Zabaras 2020 Sahli Costabal et al.
2020). the training of the PINNS is known to be challenging, which is partly due to that applying
complicate the loss landscape (Krishnapriyan et al. 202T). Recent works have analyzed common failure modes of PINNs which include modeling problems exhibiting figh-frequenyy. multi-scale, chaotic, or turbulent (likely among many other) behaviours (Wang et al. [2020c a 2022), or when the governing PDEs are stiff (Krishnapriyan et al.
differential operators over the NN in the residual
2021
Mojgani al.) (022).
One class of approaches to mitigate the training challenge of PINNs is to set different weights for the boundary and residual loss terms. For example, Wiggtt & Zhao (2020) suggested to set a large weight for the boundary loss to prevent the dominance of the residual loss. Wang et al. (2020a) proposed a dynamic weighting scheme based on the gradient statisties of the loss terms. Wang al. (2020c) developed an adaptive weighting approach based on the eigen-values of NTK. Liu & WWang (202T) employed a mini-max optimization and update the loss weights via stochastic ascent. McClenny & Metga-Neto|ae (2020) used multiplicative soft attention mask to dynamically re-weight the loss
term on each data point and collocation point. Another strategy is to modify the NN architecture so to exactly satisfy the boundary conditions (Lu et al., [02 Lyu al., 2020 Lagaris et al.
1998. 2000, McFall & Mahan 2009; Berg & Nyström| 2018 Lagari et al.) 2020 Mojgani et al.
2022 Lagaris al. 1997 Pang et al. 2019a Yu et al. 2022). In this way, the boundary loss term IS no longer needed in training and so its competition with the residual loss is eliminated. In our work, we investigated such a version for PINNs in learning high-frequency and multi-scale solutions. However, these methods are restricted to particular types of boundary conditions, e.g., the Dirichlet conditions, hence are less flexible than the original PINN framework. Taaneik et al. (2020); Wang et al. (2021) used Gaussian distributions to construct random Fourier features to improve the learning of the geggefrryeecyy and multi-scale information. The number of Gaussian variances and their scales are critical to the success of these methods. But the hyperparameters can be quite difficult to choose.
6 Experiment
To evaluate our method, we benchmarked against state-
Number
Scales
1,20, 50, 100, rand(1 11 K)
of-the-art PINN models
rand (3, (1, 20, 50, 100, rand( 1, [1, K))) , (2, [1, K])
for solving high-frequency and/or multi-scale solutions:
rand (3, (1, 20, 50, 100, rund 11 K]))) rand [1, K])
5
r2nd((2 [1, KJ) x (1. 20, 50, 100), (5. [1, KJ)
(I) standard PINNs, (2)
Random Fourier Feature PINNS (RFF-PINN) (Wang) et al. 2021). To en- sure PIN- achieves
Table 1:
The number and scales of the Gaussian variances used in RFF- PINN, where rand(k, A) means randomly selecting i elements from the set A without replacement, x denotes the Cartesian product, and K is the maximum candidate frequency used by FourierPINN.
the best performance, we followed (Wang al. 2020c) to dynamically re-weight the loss terms

---

Under review as a conference paper at ICLR 2023
I OFF-PINM scales)
PINK scales)
PINN (5 scalos)
Fourier
String BC PINN
A.PINN
Relative to Error
10
10-1
10-7
10
20
10-1
sming Bc Paoe
rads
W W-PM
RFF mase (1)
SPF pate (20)
HiF srtes (9s)
mite air arf sine pel
ass Di inn do
ter van un tu aaee ler
eir aate to 41, fsur
seeamie (1, zo d9 so 1009
RFF RFF- BrF-POm cs 3. 47 79. 136)
rstt
(1. 20 194, 127, 199)
R PoI- (S0. 65, 83, 104, 138y
so, as, 1001
Fouster PHON (K=
Fourier PARN (K=200)
Fourter PaW (C-250)
Fnurter PANN (Ko 300)
te lene
ias C2 wapee
oo withe
por aareee
to
piiie ii seiie
to
my-- E1,
(a) ((r) sim(100x))
Relative la Error
10-1
10
10-2
W-Pa0N
Aran
RFF oonn (1)
RFF nuth (20)
RST ReF netd (84)
prr-e (90)
Rer- pri- (100)
RFF peeees (1. (es
RSF snter (s. 20)
arr frr- Hied (2%, (9)
noT- PIN 11, 2a, 194)
711
mats (50, 10m
ASF PINN mine (20. so. 100)
RPP RPP- (38, 112, 119)
so, 189)
RFF oiter (44, 47, 47.1
APF FINN 20 So, 1001"
RFF omm (1 20, so, es, 1001
BFF-omg erF- (1. 20. 104, 197, 1991
RFF arr-rmelt (So. 65 83, 104, 139)
36, 07, "9. 136)
Founte paw (K+ 150)
Fouater PIN (K=200)
Fourter Pitel (K-250)
Fourier PHNN (f+ sony
RPF
(b) u(x) = sin(x) + 0.1 sin + 0.05 cas( 100x))
Relacive (2 Error
10-1-
20-2
10
MAN
W-PBWN
RFF pia (1)
RFF MaN (20)
RFF PFF- (S))
RFF mun (84)
Res pasl (100)
REF- nst-PwN (1. 5D)
eiter (s, 201
RFF iwe (19, 72)
RFF- paiN (50. eg)
REF Poo 150, 200)
RFF-PINN ct, 20, 194)
RFF, Pask sn 189)
aPF pan (20, so, 100)
RfF Fres (38, 112. 119)
RFF sits 47, 185)
RFF rine 20, sn, 200)
arp ran 20, so, es, 200)
PPN- (1. 20. 194, 197, 1391
RPF ene 36, 87, 79, 1361
RPF PINN (so. 65, 53, 104, 139)
foutee paa (K-320)
Fourter PIN (K-3500
Fourter Pilni ot (K-400))
Fourier pana (60455))
REF
(c) u(x) = 5z c0000)0
Figure 3: Solving ID Poisson equation.
based on NTK eigen-values (Wang al. 2020c). as suggested by the authors. (3) Weighted PINNs (W-PINN) that down- the residual loss to reduce its dominance and to better fit the boundary loss. (4) Adaptive PINNs (A-PINN) with parameterized activation functions to increase the NN capacity and to be less prone to gradient vanishing and exploding. All the methods were implemented by PyTorch. For W-PINN, we varied the weight of the residual loss from {10- 10 10033_ 1,104), For A-PINN, we introduced a learned parameter for the activation function in each layer and update these parameters jointly during training. To run RFF-PINNS, we need to specify the number and scales of the Gaussian variances to construct the random features. To test broad coverage, we varied the number of variances from [1, 2, 3, 5). For each number, we set the variances to be the commonly used values suggested by authors, combined with randomly sampled ones. The detailed specification is given by Table There are in total 20 settings.
7

---

Under review as a conference paper at ICLR 2023
6.1 1D Equations
We first tested with set of ID Poisson equations (see (6). We varied the form of the true solution in the following, and examined the accuracy of each method.
(x) sin (100x)) (single scale)
(16)
" (r) = sin (a) + 0.1 sin(20x) + 0.05 c.05coos(000)
(multi-scale)
(17)
u (z) 5x cos (10xx)r + x² (hybrid)
(18)
The boundary conditions are derived from the true solution. Note for the single scale case (6). the boundary condition is still Dirichlet, and hence compared with the strong BC PINN. For all the methods, we used a neural network with two hidden layers, with 100 neurons per layer, and tanh activation function. For PINN, W-PINN, A-PINN and RFF-PINN, for 40K epochs (learning rate 10-3), and then L-BFGS until convergence (the tolerance level 10-9). For FourierPINN, we a 0.1, the number of iterations 7 to 1K, the inner iteration number A to 5, the truncation threshold 8 to 10 10-4, and the outer iteration number E' to 40. For RFF-PINN, we set K- 150 for (T6) and (17.. and K 300 for ( see Table We used the same K for FourierPINN to specify the range of the frequency candidates [- K, K]. To further examine the stability, we also tested K 200, 250, 300 for (T6) and (I7). and K 350, 400, 450 for [T]. The spacing 1. We ran every method for five times, and each time with different random initialization. We report the results in box plots Fig. 3 We also examined ID Allen-Cahi equation with u(x) = sin(100r)).
first ran ADAM optimization
The result is reported in Fig. 5 of the Appendix. Note that, for W-PINN, we only report the setting that gives the best performance to save space.
We can see that, first, standard PINN, W-PINN, A-PINN and strong BC PINN all failed to obtain good solutions, and the relative error is large for all the cases (e.g., ~ 1.0). It implies that, when the true solution includes very high frequency signals, these methods are not capable enough. Second, while with some number and scale choices, RFF-PINN can achieve very small relative errors, in most choices, which take 60% 70%, RFF-PINN failed, and the solution error is large. The results show that the success of RFF-PINN is very sensitive to the choice of number and scales, and there seems no patterns. For example, for the single-scale true solution (see Fig. Bay. the successful cases of RFF-PINN include two, three or even five scale settings, e.g., (1, 20, 194). yet it failed with all the single-scale settings. This can bring difficulties in practice. As a comparison, FourierPINN always achieves reasonable, good solution accuracy in all the cases: ~ 10-4 for the single-scale case, 10-3 for the multi-scale case, and 10-2 2 for the hybrid case. Our method is also robust to the choice of the frequency range K. As long K is large enough, our method nearly always selected the target frequency, and returned similar solution accuracy. Note that, to focus on the evaluation of the key idea, FourierPINN did NOT employ any re-weighting scheme further improve the accuracy, such as the NTK re-weighting used in RFF-PINI or mini-max updates. But it is straightforward to integrate them into the proposed algorithmic framework.
6.2 2D Equations
We next tested on 2D Poisson and Allen-Cahn equations, which are give by
Au = [(x,,), w), a, y E [0, 2m], Ugz + Byy + u(u² - 1) f(xx(()) æ,y E [0, 2T), respectively. We tested with the following ground -ouund-truthha solutions,
(19)
(20)
u(æ, y) - sin sin(100z) sin(100y), (2D Poisson)
(21)
u(x, y) = (sin(x) + 0.1 sin() + cos(100x))
(sin(y) 0.1 sin (20g) + cos(100y)).
(2D Allen-Cahn)
(22)
u(z, 3) 5x cos (100x) (y) + + 3². (2D Poisson)
(23)
For the NN of all the methods, we used three hidden layers with 100 neurons per layer and tanh activation. We used the same settings as in Sec 6. to ran all the methods. For RFF-PINN and FourierPINN,
set K 200. The solution error (over five runs) is reported in Fig. 4 Note that with all the single-scale settings, RFF-PINN failed and gave large relative errors, and we did not report their results in the figure. Obviously, the 2D cases are much more challenging for every method. In
8

---

Under review as a conference paper at ICLR 2023
10c
Relative L.
10-4
20-1
Strong BC Hatt
W-rees
RFF eost 50)
RRR-r (3, 20)
RFF PPP (10, 733
(39, 69)
AFF PBN (I, 2a 1941 1907
Bm PRRP 11, so, 1891
ASF PF-- nu So, 1001
1ott
RFF- (1, so, e5, 100)
1Sot
195-00aa (I, 104, 107, 1991
ae so 1007
RFF parl (6, 47, 79, 1367
JEFF-9I do, 63, 104. 1391
Founier PAON (Kis (K0x200)
A
RFF-PHON RPF-PIN0 (So,
RPV-PHNN A3F MMn-ea
(a) 2D Poisson with u(x,)) sin(100x) sins (100g)
Relative L, Error
10
10-0
PHW
WARN
RFF- PaIN 5o)
APRNN
RiF PIRH 2000
HFP PRP- 139 69)
721
HFF PRR 20, 194)
(S0, 160)
ROF FOEN CO. 56, 100)
(R. 50, 189)
REF PE--PIIN (4A, 4T. 165)
PFF-PINN CL 30 30, 100)
RYF-PINRH BAR-PHN RVE-PINN 30, 67. 79, 136)
(1. 5o, 05, 100)
20, 104, 197, 199)
BFF-FINN 65 83, 104, 130)
Fourner mAi (Kw201)
BFF-FING
RAF RAF-PANN
HFF- WON-PONN CIS,
(b) 2D Allen-Carl Cahn with u(x, v) = (sin(x) + 0.1 sin(()n)) + (100z)) (sini)) 0. (20g) + (000))
10r
Relative La
10-1_
A.MW
WPR-PIIN so)
RFF PMW 20)
Ht' F--- (29, 73)
FIN (50, 69)
RPF Parti LSG, zan)
RPF PPP-PINN 20, 194)
mwn so, 189)
RFF PAN (20, so, 100)
RFF Pwoo (38. 112. 119)
RFF PINN (44, 47, 165)
RFF MINI (1. 20, 50, 100)
RSF- 11 20, sa, as, 100)
RFF-PIN (1. 20. 194, 197, 199)
RFF RY--RII (6, 36, 67, 79, 136)
RFF PFN- 65, 83, 104, 139)
Fourter PAIN (Cwco0))
(c) 2D Poisson with u(x, w) 5z cos( 100z) sin(y) + 3
3.3
Figure 4: Solving 2D equations.
particular, for the hybrid case 23) RFF-PINN nearly failed with every choice of the number and scale set. FourierPINN consistently outperforms all the competing methods with all the settings, in most cases by large margin. Together these results have confirmed the advantage of FourierPINN.
Conclusion
We have presented FourierPINN, a novel PINN extension to capture high-frequency and multi-scale solution information. Our adaptive basis learning and selection algorithm can automatically identify target frequencies and truncate useless ones, without the need for tuning the number and sales. The performance a set of benchmark equations is encouraging. Currently, (granularity) of the candidate frequencies is fine- grainga enough, which can be overly optimistic. In the future, plan develop adaptive spacing approaches further capture the granularity.
assume the spacing

---

Under review as a conference paper at ICLR 2023
References
Jens Berg and Kaj Nyström. A unified deep artificial neural network approach to partial differential equations in complex geometries. Neurocomputing, 317:28-41 2018.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of 2016.
sciences, 1 113(15):3932-33
Yuyao Chen, Lu Lu, George Em Karniadakis, and Luca Dal Negro. Physics-informed informed neural networks for inverse problems in nano-optics and metamaterials. Optics express, 28(8): 1618-11688,,,, 2020.
Zhao Chen, Yang Liu, and Hao Sun. Physics-informed learning of governing equations from scarce data. Nature communications, 12(1))1-13, 2021.
Zhiwei Fang and Justin Zhan. Deep physical informed neural networks for metamaterial design. IEEE Access, 8:24506-24511 2019.
Nicholas Geneva and Nicholas Zabaras. Modeling the dynamics of pde systems with physics- constrained deep auto-regressive networks. Journal of Computational Physics, 403:109056, 2020.
Oliver Hennigh, Susheela Narasimhan, Mohammad Amin Nabian, Akshay Subramaniam, Kaus- tubh Tangsali, Zhiwei Fang, Max Rietmann, Wonmin Byeon, and Sanjay Choudhry. Nvidia simnetTM; An ai-accelerated multi-physics simulation framework. In International Conference on Computational Science, pp. 447-461. Springer, 2021.
Ameya D Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics, 404:1091330 136, 2020.
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426: 109951, 2021.
Georgios Kissas, Yibo Yang, Eileen Hwuang, Walter R Witschey, John A Detre, and Paris Perdikaris. Machine learning cardiovascular flows modeling: Predicting arterial blood pressure from non- invasive 4d flow mri data using physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 358: 12623, 2020.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael Mahoney. Char- acterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 14:26548-26545 2021.
Pola Lydia Lagari, Lefteri H Tsoukalas, Salar Safarkhani, and Isaac E Lagaris. Systematic construc- tion of neural forms for solving partial differential equations inside rectangular domains, subject to initial, boundary and interface conditions. International Journal on Artificial Intelligence Tools, 29(05):2050000 2020.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural network methods in quantum mechanics. Computer Physics Communications, 104(1-3):1-14, 1997.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9050988710000 1998.
Isaac E Lagaris, Aristidis C. Likas, and Dimitris G Papageorgiou. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, 11 (5):1041-104, 2000.
Dehao Liu and Yan Wang. Multi-fidelit phystie-constrainee neural network and its application in materials modeling. Journal of Mechanical Design, 141(12), 2019.
Dehao Liu and Yan Wang. A dual-dimer method for training constrained neural networks with minimax architecture. Neural Networks, 136: 12-1122, 2021.
10

---

Under review as a conference paper at ICLR 2023
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: A deep learning library for solving differential equations. SIAM Review, 63(1):208-22) 228, 2021a.
Lu Lu, Raphael Pestourie, Wenjie Yao, Zhicheng Wang, Francesc Verdugo, and Steven G Johnson. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6):B 1111111111110 B1132, 2021b.
Liyao Lyu, Keke Wu, Rui Du, and Jingrun Chen. Enforcing exact boundary and initial conditions in the deep mixed residual method. arXiv preprint atXiv:2008.01491, 2020.
Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-informed neural networks using a soft attention mechanism. arXiv preprint Xii::2009.04544 2020.
Kevin Stanley McFall and James Robert Mahan. Artificial neural network method for solution of boundary value problems with exact satisfaction of arbitrary boundary conditions. JEEE Transactions on Neural Networks, 20(8):1211111) 2009.
Clare D McGillem and George R Cooper. Continuous and discrete signal and system analysis. Harcourt School, 1991.
Rambod Mojgani, Maciej Balajewicz, and Pedram Hassanzadeh. Lagrangian pinns: A causality- conforming solution to failure modes of physics-informed neural networks. arXiv preprint arXiv:2205.02902, 2022.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.
Guofei Pang, Lu Lu, and George Em Karniadakis. fPINNs: Fractional thysics-informen neural networks. SIAM Journal on Scientific Computing, 41(4):42603-A2226 jan 2019a. doi: 10.1137/ 18m1229845. URL http::ttt////c http:pssss//d...rg//111oo0// 1ww..2F111111111 0rt.
Guofei Pang. Lu Lu, and George Em Karniadakis. fpinns: Fractional thysics-informet neural networks. SIAM Journal on Scientific Computing, 4144),2203-A22666 2019b.
Guofci Pang, Marta D' Elia, Michael Parks, and George E Karniadakis. nPINNs: nonlocal physics- informed neural networks for parametrized nonlocal universal laplacian operator. algorithms and applications. Journal of Computational Physics, 422: 109760, 2020.
Christopher Rackauckas, Yingho Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001 04385, 2020.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pp. 5301-5310. PMLR, 2019.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707,2011
Maziar Raissi, Zhicheng Wang, Michael S Triantafyllou, and George Em Karniadakis. Deep learning of vortex-induced vibrations. Journal of Fluid Mechanics, 861:119 19-137, 2019b.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, (1266)4111126-03000 2020.
Junuthula Narasimha Reddy. Introduction to the finite element method. McGraw-Hill Education, 2019.
Francisco Sahli Costabal, Yibo Yang, Paris Perdikaris, Daniel E Hurtado, and Ellen Kuhl. Physics- informed neural networks for cardiac activation mapping. Frontiers Physics, 8:42, 2020.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339-1134 1364, 2018.

---

Under review as a conference paper at ICLR 2023
Luning Sun, Han Gao, Shaowu Pan, and Jia Wang. Surrogate modeling for fluid flows based on physics-constraiinn deep learning without simulation data. Computer Methods in Applied Mechanics and Engineering, 361:112732, 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologies in physics-informed neural networks. arXiv preprint arXiv:200 1.4556,, 2020a.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks. arXiv preprint arXiv:2012.10047, 2020b.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. arXiv preprint 24Xi::0007.14527, 2020c.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 384:1 13938, 2021.
Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-infomedd neural networks. arXiv preprint arXiv:2203.074004 2022.
Weinan, Jiequn Han, and Arnulf Jentzen. Deep learning-based numerical methods for high- dimensional parabolic partial differential equations and backward stochastic differential equations. Communications Mathematics and Statistics, 5(4):349-380, 2017.
Colby L Wight and Jia Zhao. Solving allen-cahn and physics informed neural networks. arXiv preprint arXiv:2007.04542, 2020.
equations using the adaptive
Jinmian Ye, Linnan Wang, Guangxi Li, Di Chen, Shandian Zhe, Xinqi Chu, and Zenglin Xu. Learning compact recurrent neural networks with block-term tensor decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9378-9387, 2018.
Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Karniadakis. Gradient-enhanced physics-informed informed neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering, 393: 14823, 2022. ISSN 0045-7825. doi: http:///doi.org/1000111100
2022. 14823. URL https: /www. www.scciieeeiceeeeeeeeeeoeooeeee http://////ceeee//rrriiill/a/a/a SDC44782522001
Yinhao Zhu, Nicholas Zabaras, Chaedon-Stelion Koutsourelakis, and Paris Perdikaris. Physics- constrained deep learning for high- aigh-dimmnssonal surrogate modeling and uncertainty quantification without labeled data. Journal of Computational Physics, 394:56-81, 2019.
A Appendix
1D Allen-Cahn
We report the relative L2 emor for solving ID Allen-Cahn equation with the Dirichlet boundary condition (see (7). The exact solution is u(x) = si1(10 As shown Fig. FourierPINN consistantly outperforms all the competing methods.
12

---

Under review as a conference paper at ICLR 2023
104
Relative la
18
10-22
Strong Piay
Worm
HFF PaNN so)
HFF HFF-PION Ca. 201
RFF-PINN (19, 711
REF FIEN (50 199)
REF (L 2O 1941
RFF RIF (1 50, 189)
C2D, 1001
RFF- PFE-PPNN 1651
RFF PION 11. 20, s0 85, 180)
RUFPINN 20. 194, 197, 189)
RFF PIN- 36, 67, 136)
RFF- (50, 65, 83, 104, 189)
Fourter PINN (Kx
RFP PPP-PPNNN (99,
PINN (38. 112,
MIN 20 49, s0,
Figure 5: ID Allen-Cahn equation with true solution w(x) si000)
13


Reviews:
[{"review_id": "aNqXOlpx5jc", "reviewer": "Reviewer_nwfE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposes Fourier PINNs, a variant of PINN that exactly satisfies a simple boundary condition. In addition to the reinforcement of boundary conditions, this manuscript also did a Fourier analysis of the proposed neural architecture and proposed an efficient implementation of Fourier PINNs that has a better speed and accuracy than previous methods.", "summary_of_the_review": "The manuscript proposed an effective variant of PINNs which can reinforce simple boundary conditions exactly. I would like to see some explanations and experiments added. I am open to raising the score based on the response.", "strength_and_weaknesses": "Strength:\n- The manuscript compared their results against a lot of baseline methods, which effectively illustrated their performance.\n\nWeaknesses:\n- I wonder how this proposed method compares to [1]. In my opinion, the proposed method and [1] both pruned a span of Fourier features but were trained differently by supervised loss and physics-inspired loss.\n- The results are very quantitative. It will be interesting to show some visualization of learned PDEs to capture a qualitative sense.\n- I would expect more ablation studies of the choices in the proposed methods: for example, how does the number of iterations affect the results?\n\n[1] Li, Zongyi, et al. \"Fourier neural operator for parametric partial differential equations.\" arXiv preprint arXiv:2010.08895 (2020).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UVWdusuu1x", "reviewer": "Reviewer_9ZeR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Since PINNs often struggle to learn high-frequency and multi-scale target solutions,\nauthors investigated a strong boundary condition (BC) version of PINNs. By modeling\nthe solution as an neural network plus the linear combination of a set of dense\nFourier bases, authors developed Fourier PINNs. Finally, authors evaluated Fourier\nPINNs in several benchmark PDEs with high-frequency and multi-frequency solutions\nand solutions that couple high-frequency components with plain functions. In\nall the cases, Fourier PINNs consistently achieve good solution errors.", "summary_of_the_review": "In summary, although the work is interesting, strong boundary condition does not have enough novelty and the design of Fourier PINNs still have lots of open problems to be solved. There exists plenty of room to improve the quality of the paper at this point, and as such the manuscript should not be published in the present form.\n", "strength_and_weaknesses": "Both strong boundary condition PINNs and Fourier PINNs achieve good errors in numerical experiments and provide new ideas for solving PDEs with high-frequency and multifrequency\nsolutions. While the author's approaches are interesting, there are some weaknesses should be pointed out.\n1. The main idea of strong boundary condition PINNs is to construct the solution with\n$$\n\\widetilde{u}_{\\theta}(x)=B(x)\\mathcal{N}_{\\theta}(x),\n$$\nwhere $\\mathcal{N}_{\\theta}(x)$ is a neural network parameterized by $\\theta$, and $B|_{\\partial\\Omega}=0$. This approach use be used in penalty-free neural network (PFNN), see Shen and Yang 2021. \n2. Furthermore, PFNN extended this approach to complex geometries and nonhomogeneous Dirichlet boundary conditions.\n As we all known, $\\{\\cos(\\pi nx)\\}_{n=0}^{\\infty}$ (or $\\{1\\}\\cup\\{\\sin(\\pi nx)\\}_{n=1}^{\\infty}$) is an orthogonal basis of $L^{2}(0,1)$, which means that any function in $L^{2}(0,1)$ can be written as a liner combination of $\\{\\cos(\\pi nx)\\}_{n=0}^{\\infty}$ (or $\\{1\\}\\cup\\{\\sin(\\pi nx)\\}_{n=1}^{\\infty}$). The authors should interpret the motivation of using the NN basis in (10) or (11).\n3.  Removing the linear combination of NN basis in (10) or (11) yields the spectral methods. In fact, we only need to solve a least square problem in spectral methods, but need to solve non-convex optimization problems many times in Fourier PINNs. I recommend the authors to explain the advantages of Fourier PINNs comparing with the spectral methods.\n\nRef:\nHailong Sheng and Chao Yang. Pfnn: A penalty-free neural network method\nfor solving a class of second-order boundary-value problems on complex ge-\nometries. Journal of Computational Physics, 428:110085, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2MQin7w2bH", "reviewer": "Reviewer_gWnw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents FourierPINN, an extension of PINNs to capture high-frequency information. This is achieved by conducting a Fourier analysis of strong boundary condition PINNs where a specific boundary condition is used. The result is that multiplying the neural network with the boundary function should enable a faster and more accurate learning of the coefficients. The findings are tested on small synthetic datasets.", "summary_of_the_review": "The paper discusses an interesting extension to PINNs. However, it is hard to judge on the experiments how significant the new method is. ", "strength_and_weaknesses": "Strengths:\nThe paper identifies one specific problem of PINNs (capturing high-frequency information) via Fourier analysis on strong BC PINNs, which is quite neat.  \n\nWeaknesses:\nAll the experiments are rather toy-like, and thus it is hard to judge how well the proposed method works. I would like to see at least one temporal PDE, and also some standard PDE problems to test if the approach is still valid there are harms the PINN performance on any other problem besides the toy problems. Which f(x) is needed to get the true solutions outlined in the paper?\nPINNs are known for having problems with high-frequency information, there is tons of work on that. Would it be possible to put that into better perspective, both method-wise and comparison-wise? I doubt that RFF PINNs are the only competitors here?\nI am missing details on the training behavior of the FourierPINNs, again hard to judge if notorious problems in PINN training can be alleviated by this approach or if they get worse.\nOne claim of the paper is that FourierPINNs achieve a “faster and more accurate learning of coefficients of higher frequencies”, yet there is nothing in the experiment section which supports the “faster” claim.\nAgain experiments, I would have liked to see experiments for multiple inputs.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject