Fourier PINNs: From Strong Boundary Conditions to Adaptive Fourier Bases
Authors: Madison Cooley
Varun Shankar
Mike Kirby
Shandian Zhe
Venue: Accepted by TMLR ()

Abstract: Interest is rising in Physics-Informed Neural Networks (PINNs) as a mesh-free alternative to traditional numerical solvers for partial differential equations (PDEs). However, PINNs often struggle to learn high-frequency and multi-scale target solutions. To tackle this problem, we first study a strong Boundary Condition (BC) version of PINNs for Dirichlet BCs and observe a consistent decline in relative error compared to the standard PINNs. We then perform a theoretical analysis based on the Fourier transform and convolution theorem. We find that strong BC PINNs can better learn the amplitudes of high-frequency components of the target solutions. However, constructing the architecture for strong BC PINNs is difficult for many BCs and domain geometries. Enlightened by our theoretical analysis, we propose Fourier PINNs --- a simple, general, yet powerful method that augments PINNs with pre-specified, dense Fourier bases. Our proposed architecture likewise learns high-frequency components better but places no restrictions on the particular BCs or problem domains. We develop an adaptive learning and basis selection algorithm via alternating neural net basis optimization, Fourier and neural net basis coefficient estimation, and coefficient truncation. This scheme can flexibly identify the significant frequencies while weakening the nominal frequencies to better capture the target solution's power spectrum. We show the advantage of our approach through a set of systematic experiments.

Published in Transactions on Machine Learning Research (01/2025)
Fourier PINNs: From Strong Boundary Conditions to Adap- tive Fourier Bases
Madison Cooley Scientific Computing and Imaging Institute University of Utah
mcooley utah. edu
Varun Shankar
shankar@cs. utah. edu
Scientific Computing and Imaging Institute University of Utah
Robert M. Kirby
kirby@cs utah. edu
Scientific Computing and Imaging Institute University of Utah
Shandian Zhe
zhe@cs. utah. edu
Kahlert School of Computing Universily of Ulah
Reviewed on OpenReview: https: // openreview. net/ forum? id= KyRna
Abstract
Interest rising in to traditional numerical solvers for partial differential equations (PDEs). However, PINNs often struggle to learn high-frequency and multi-scale target solutions. To tackle this prob- lem, first study strong Boundary Condition (BC) version of PINNs for Dirichlet BCs and observe consistent decline in relative error compared to the standard PINNs. We then perform a theoretical analysis based on the Fourier transform and convolution theorem. We find that strong BC PINNs can better learn the amplitudes of high-frequency components of the target solutions. However, constructing the architecture for strong BC PINNs is difficult for many BCs and domaín geometrics. Enlightened by our thcoretical analysis, we propose Fourier PINNs simple, general, yet powerful method that augments PINNs with pre-specified, dense Fourier bases. Our proposed architecture likewise learns high-frequency components better but places no restrictions the particular BCs problem domains. We develop an adaptive learning and basis selection algorithm via alternating neural net basis optimization, Fourier and neural net basis coefficient estimation, and coefficient truncation This scheme can flexibly identify the significant frequencies while weakening the nominal frequencies to better capture the target solution's power spectrum. We show the advantage of our approach through of systematic experiments.
Neural Networks (PINNs) as a mesh-free alternative
1 Introduction
are innovative, mesh-free approaches for solving partial differential equations (PDEs). They offer alternatives to traditional mesh-based numerical methods such as finite elements and finite volumes (Reddy) 2019). The optimization of PINNs involves softly constraining neural networks (NNs) through customized loss functions designed to adhere to the governing equations of physical processos. Researchers have successfully appliod PINNs in various domains for example, they have been used to simulate the radiative transport equation (Mishra & Molinaro 2021)
Physics-informed neural networks (PINNs) Raissi et al. 2019
which is crucial for radio frequency chip and material design (Chen al. 2020
Liu & Wangl 2019). Further.

---

Published in Transactions on Machine Learning Research (01/2025)
cardiovascular flow modeling (Kissas et al. 2020) is another application, along with various fluid mechanics
problems (Cai et al.
2021
and high-speed aerodynamic flow modeling (Mao et al. 2020). among many
others (Raissi et al., 2020 Chen et al. 2020 Jin et al. 2021 Sirignano & Spiliopoulos 2018 Zhu et al.
2019
Geneva & Zabaras
2020 Sahli Costabal et al.) 2020 Sun et al. 2020 Fang & Zhan 2019
Despite these successos, the training of PINNs remains challenging in some instances. Recent studies have analyzed common failure modes of PINNs, particularly when modeling problems that exhibit high-frequency,
multi-scale, chaotic, turbulent behaviors (Wang et al. 2022 2021c|bb 2024b), or when the governing PDEs are stiff (Krishnapriyan et al. 2021 Mojgani et al, 2023) Rahaman et al. (2019) attributes the slow convergence to the high-froquency components of the target solution, identifying it as "spectrum bias" in standard NNs, while learning low-frequency information from data is straighuforward. Wann et al. (202cc) confirmed that this bias is also present in the PINN setting. These challenges often arise because applying differential operators over the NN in the residual complicates the loss landscape (Krishnapriyan et al. 2021 From an optimization perspective, Wang et al. (2021b) highlighted that the imbalance in gradient magnitudes between the boundary loss and residual loss (with the latter often being much larger) causes the residual loss to dominate training, leading to poor fit to the boundary conditions. [Wang et al. (2022) confirmed this conclusion through neural tangent kernel (NTK) analysis of PINNs with wide networks, finding that the dominant eigenvalues of the residual kernel matrix often result in the training primarily fitting the residual loss.
One class of approaches designed to mitigate the training challenges in PINNs involves setting different weights for the boundary and residual loss terms. For example, Wight & Zhao (2020) suggested incorporating large multiplier for the boundary loss term to prevent the residual loss from dominating the training process. In contrast, WWang et al. (2021b) proposed dynamic weighting scheme based the gradient statistics of the loss terms, and Wang et al. (2022) developed an adaptive weighting approach based on the eigenvalues of the NTK. Liu & Wang (2021) employed a mini-max optimization to update the loss weights via stochastic ascent, and McClenny & Braga-Neto (0044) used a multiplicative soft attention mask to dynamically ro- ro-weight the loss term for each data point and collocation point. To alleviate the spectrum bias in NNs, Taacik el al. (2020) randomly sampled a set of high frequencies from a large- Gaussian distribution to construct random Fourier features input to the NN. Additionally, Wang et al. (202cc) used multiple Gaussian variances to sample frequencies for Fourier features, aiming to capture multi-scale solution information within the PINN framework. While effective, the performance of this method is sensitive to the number and scales of the Gaussian variances, which are user-specified hyperparameters that are often difficult to optimize.
Another strategy to improve these challenges is to modify the NN architecture to exactly satisfy the boundary conditions (BCs) [Lu et al.] 2021 Lyu et al. 2021 Lagaris et al. 1998 2000 McFall & Mahan 2009 Berg & Nyström 2018 Lagari et al. 2020; Mojgani et al. (023). We refer to these approaches as "strong BC PINNs" Despite their effectiveness, these approaches face several limitations. They are usually limited to tasks with relatively simple and well-defined physies, requiring significant craftsmanship and complex implementations even in straightforward problem settings. Consequently, these strategies are less flexible than the original PINN framework, which employs a soft constraint approach for boundary condition satisfaction. Additional complications arise for challenging physical systems governed by invariances or conservation laws, such as energy or momentum conservation, due to often poorly understood and imprecisely defined physical laws. Incorporating these laws effectively into the NN architecture makes extending strong BC PINNs to handle more complex tasks difficult. Further, designing custom ansatz requires tailoring the NN architecture for each specific boundary condition and domain, which is often impractical for complicated domains. However, when properly designed, these techniques can achieve highly accurate solutions.
In our work, we delve deeper into the training challenges of PINNs for learning high-frequency and multi- scale solutions. We analyze the mechanisms behind the success of strong BC PINNs and explore ways to incorporate these successful strategies into a more general PINN architecture. Our specific contributions are as follows:
We first examine a strong BC PINN architecture for simple Dirichlet boundary conditions as pro- posed by Lu al. (2021). This variant integrates a fixed polynomial boundary function into the NN architecture to exactly satisfy the boundary conditions. While it shows significant. improvements
2

---

Published in Transactions on Machine Learning Research (01/2025)
over the standard PINN, especially for higher frequency problems, it struggles to predict solutions with frequencies above certain threshold due to its static nature. To address this limitation, we propose a new strong BC PINN architecture featuring an adaptive parameter optimized during train- ing. This parameter adjusts the boundary function's sharpness to match the true solution, thereby improving accuracy for higher-frequency solutions compared to the static polynomial variant.
We conduct a Fourier analysis on both strong BC PINNs compared to the standard PINN. Through the Fourier series convolution theory, we discovered that multiplying the NN by the strong boundary function significantly enhances the learning speed and accuracy of the higher frequency coefficients in the target solution. In contrast, standard PINNs struggle to accurately capture coefficients in the figu-freyyenccy domain. This analysis complements and confirms the aforementioned NTK work.
Inspired by our Fourier analysis, we develop Fourier PINNs. This novel PINN architecture enhances frequency learning within the true solution, comparable to strong BC PINNs, regardless of specific boundary conditions, domain, or underlying physical properties. The Fourier PLNN architecture integrates standard NN with a linear combination of Fourier bases, with frequencies uniformly sampled from an extensive pre-sel range. We implement an adaplive learning and basis selection algorithm that alternately optimizes the NN basis parameters and the coefficients the NN and Fourier bases while pruning insignificant bases. This approach efficiently identifies significant fre- quencies, supplements those missed by the NN, and improves frequency amplitude estimation (i.e., the basis coefficient) while maintaining computational efficiency. Unlike previous methods, this ap- proach only requires specifying a sufficiently large range and small spacing for the Fourier bases without concern for the actual number and scales of frequencies in the true solution.
We evaluate Fourier PINNs several benchmark PDEs characterized by high-frequency and multi- frequency solutions. In all cases, Fourier PINNs consistently achieve low solution errors (e-g., 10 or ~ 40_ In contrast, standard PINNs invariably fail to achieve comparable accuracy. PINNs with random Fourier features (RFF-PINNs) often fail across various Gaussian variances and scales, indicating high sensitivity to these parameters. Additionally, we test spectral methods, PINNs with large boundary loss weights (Wight & Zhao 2020), and PINNs with adaptive activation functions (Jagtap al., 2020 Fourier PINNs consistently outperform all these methods.
The remainder of this paper is structured as follows: Section 2 provides the necessary background and notation, while in Section 4 we present our Fourier analysis of strong BC PINNs and explain the success of the strong boundary ansatz methodology. Our new Fourier PINN architecture and its training routines are described in Section 5 Section 0 details our numerical experiments and findings, including assessments of computational cost and accuracy compared to baseline methods. Finally, Section discusses future work regarding scalability and Scction discusses the results and outlines specific future rescarch directions.
2 Background
In this section, we first describe the general optimization problems we are addressing with physics-informed neural networks (PINNs) following the formulation presented in Raissi et al. 22019). We then discuss the specifics of strong boundary condition enforcement.
2.1 General Overview of Ihysics-Informed Neural Networks
The PINN framework uses NNs to estimate solutions to partial differential equations (PDEs). Consider PDE of the following general form,
Flu](x) f(x), x E Z
(1)
where FF is a linear or non-linear differential operator and u represents the unknown solution and Ω is the problem domain in Rdd The general boundary conditions are then,
Blu[(x) g(x), x E 8n
(2)
3

---

Published in Transactions on Machine Learning Research (01/2025)
where U is the boundary of the domain and B is general boundary-condition operator.
To solve the PDE, the PINN uses deep NN, un (x: 0), to approximate the true solution u(x). For clarity in later sections, we follow the convention of Cyr et al. (2020) and define the output of the NN with width W as a linear combination of a set of nonlinear bases such that
U (x: c, 0t) = Σ (x
(3)
where each 2 are nonlinear activation functions (such as Tanh) acting on the hidden layer outputs. Each ej for j ,.... and 0 are the weights and biases in the last layer of the NN and hidden layers, respectively. Therefore, 0 = 0") form the set of all network parameters. Then, finding the optimal network parameters 0t involves minimizing the following composite loss function between boundary and residual loss terms,
0. arg ming ALo(O) + (0).
(4)
Here,
1 M M
(0)
Σ
(B() 9(xx)))
(5)
is the boundary loss to fit. the boundary condition with Lagrange multiplier X, and
(0)
- - f(x))))
(6)
N
is the residual loss to fit the equation. We minimize Equation by sampling N collocation points {x]] M
from the domain Z and M points { from the boundary the domain On, and iteratively modifying the network parameters through gradient descent.
2.2 Strong Boundary Condition PINNs
Equation can approximately enforce various types of boundary conditions, including Dirichlet, Neumann, Robin, and periodic. However, prior analysis by Wang et al. (2021b) 2022) suggests that the instability in training PINNs likely arises from the competition between the weakly enforced boundary loss ( and the residual loss 6 terms during optimization. This competition likely occurs because both loss terms are minimized simultancously during training. However, the optimization algorithm tends to prioritize the minimization of the residual loss, as it typically dominates the gradient of the combined loss function. Consequently, this can lead to a scenario where the residual loss converges effectively but at the expense of the boundary loss, which remains inadequately optimized. As a result, the boundary conditions may not be satisfied, leading poor model performance on new, unscen data or data at the domain's 's boundarics.
One solution to mitigate the competition between the loss terms is to design surrogate model that inherently satisfies the boundary conditions. This approach typically works by modifying the network architecture to satisfy the boundary conditions exactly, thus climinating the need for boundary enforcement loss term and further reducing the computational cost. This method of exact imposition of boundary conditions has become a standard approach the PINN literature and is especially prevalent for Dirichlet and periodic boundary conditions (Lu et al. 2021 Yu et al. 2022 Wang et al. 2024a).
This work primarily focuses on Dirichlet boundary conditions to simplify the analysis and implementations while addressing significant and common scenario in physical systems. Dirichlet boundary conditions provide clear and straightforward framework to demonstrate the effectiveness of the surrogate model approach, as they require the solution to take specific values at the boundaries, which is relatively easy to enforce exactly on simplified domains. To illustrate, we consider the 1D Dirichlet boundary condition defined as,
E b], u(a) u(bb = g(x).
(7)
4

---

Published in Transactions on Machine Learning Research (01/2025)
For this case, the surrogate model is typically defined as,
uo(x) g((x) o(z)ux(z),
(8)
such that (x) is the solution function, g(x) is the provided boundary function, ux(x) is the output of the neural network, and 6(x) is composite distance function that zeroes out on the boundaries, ensuring the boundary conditions are met without explicit penalty terms. By construction, uo(a) up(b) = g(z),
and thus the boundary condition in Equation 7 is strictly and automatically satisfied. To estimate the parameters of uo, wc only need to minimize the rcsidual loss (0). Most PINN surrogate model designs rely on distance functions based on the theory of R-functions (Sukumar Srivastava, 2022); which are smooth mathematical functions that encode Boolean logic and facilitate the combination of simple shapes to form complex geometries (Rvachev & Sheiko 1995 Some meshfree Galerkin methods have utilized the
capability of R-functions to enforce boundary conditions smoothly and exactly to enhance the precision and
robustness of numerical simulations for solving boundary-value problems (Shapiro & Tsukanov 1994 Tsukanov & Posireddy 2011
1999: Akin
101
au a= 0.75
sup a 1.25
10
Gepp. == 2.25
@sspi a= 3.75
5.75
8.75
10
1
2
3
5
6
X
Figure 1: This figure shows the polynomial distance function Q (x) and the exponential distance fanction (zsp(2) with different a values. The curves for @zsp(z) correspond to different a parometers, demonstrating how the shape of (x) "sharpens" around the domain boundaries increases.
In this work, compare the standard PINN to two strong BC PINNs using different distance functions to enforce Dirichlet boundary conditions exactly. The first distance function we consider is the polynomial distance function proposed by (Lu et al. 2021), defined as,
(poly(I) = (x a)(b x).
(9)
This function satisfies the Dirichlet boundary conditions by zeroing out at the x = a and = b endpoints. While Lu et al. (2021) demonstrated promising results using this formulation, the polynomial distance function (poty(z) (x) remains static throughout the training process. The lack of adaptability limits the model's ability to handle different problem domains and boundary conditions, which can hinder performance when faced with varying complexitics within the solution space.
To address this limitation, we propose adaptive distance function that introduces flexibility in enforcing boundary conditions, which define
(1-
c))(e-))
(10)
where a₂ and ag are parameters that be pre-set optimized during training. These parameters allow the function to adjust to the characteristics of the problem domain dynamically, ensuring robust enforcement of boundary conditions across different scenarios. As we demonstrate in subsequent scctions, this flexibility improves accuracy and enhances the efficiency of the training process by enabling the network to adapt to the complexity of the solution space. Allowing independent values for as and as enables additional flexibility and allows the function to adapt asymmetrically different boundary conditions. Although the proposed adaptive distance function Equation I0 may appear incremental, its primary contribution lies in demonstrating scalable and flexible framework for enforcing strong boundary conditions. This method
5

---

Published in Transactions on Machine Learning Research (01/2025)
significantly improves over traditional PINNs and non-adaptive distance functions, particularly for problems that require the exact imposition of Dirichlet boundary conditions. Additionally, in later sect sections, we present a thorough theoretical analysis that elucidates the mechanisms driving the improvements in enforcing strong boundary conditions, further highlighting the advantages of the proposed method.
The general form of the adaptive distance function for a d-dimensional rectilinear domain Q C Rdd with spatial coordinates (21, Ta); is:
Pexp (x)
II (1-
-0, (1-
(11)
where di, b; represent the boundaries in the i-th dimension, and Caa, Obd, are the corresponding steepness parameters. This form ensures that the boundary conditions are enforced independently in each cimension,, making the approach casily scalable to higher dimensions in rectilincar domains.
One potential stralegy for adaptation for irregular domains is using Fourier extension methodss which allow non-periodic functions defined on arbitrary domains to be extended into periodic ones. Doing so can enforce the boundary conditions in more structured domain, allowing the network to handle irregularities better. Another promising direction is leveraging custom network layers that map irregular domains to rectilin- ear spaces, as demonstrated by methods such as the GOFNO network Liu et al. (2023). Extending this framework to more complex boundary conditions, such as Neumann or Robin conditions, is an important future direction. The adaptive distance function can be modified for Neumann boundary conditions to have non-zero slopes at the boundaries, ensuring correctly enforced derivatives. This would require careful and non-trivial adjustments to the distance function, but the general framework remains applicable.
Figure I illustrates the behavior of the polynomial distance function (p)ly to the exponential distance function (x: with various values of a. As varies, (x))(() adjusts its shape, showcasing its ability to conform to different problem domains and complexities. We expect this dynamic adaptability to enhance the model's performance by more precisely enforcing boundary conditions across diverse scenarios. We alfirm this through a numerical example in the next section.
3 Boundary Condition Pathologies in PINNs
Recent works have employed strong boundary enforcement strategies (Lu al., 2021 Yu al. 2022) but offer few insights into the specifie surrogate model design process. The design and implementation of the surrogate models are often ad-hoc, with no systematic approach to their architectural design. This complicates their application and limits their effectiveness across boundary conditions and physical problems, especially when dealing with complex physics and domains. Moreover, the standard PINNs frequently encounter difficulties with high-frequency and multi-scale solutions, leading to inaccurate predictions (Wang iet al. 2021c). We show that while the strong BC PINN model improves solution quality for problems with higher frequency compared to the standard PINN, the solution quality nevertheless degrades with larger frequencies. In the following, we empirically demonstrate how the standard PINN solution quality degrades the actual frequency of the solution increases in both linear 1D Poisson problem and non-linear 1D steady-state Allen- Cahn problem. Specifically, of frequencies k E [2, 6, 10, 14, 18, 22, 26, 30, 34] and æ E [0, 2T]. Therefore, by varying k, we can examine the performance of the standard PINNs compared to the strong BC PINNs when the solution includes different frequency information.
consider the fabricated solution u(x) sin (in(kz) for range
Specifically, we examine the 1D Poisson equation given by,
Au f() u(0) u(1) 0,
(12)
(13)
6

---

Published in Transactions on Machine Learning Research (01/2025)
and derive the forcing function f(x) from the fabricated solution u(x) sin(kx) which gives f(x) - h2 sin(..)) We also consider the 1D Olen-Cahn equation defined as,
Uaw + u(u² f())
(14)
(()) = u(1) 0,
(15)
which gives the forcing function f(z) = -k² sin(kr) + (ar) -)) Both are subject to the Dirichlet condition in Equation where 0 and b = 2x.
We test these problems using standard PINN, which explicitly incorporates the weakly enforced boundary loss and residual loss terms. We compare these results to two strong BC PINNs employing different distance functions ( and show that this method exhibits slower solution degradation with increased Trequency. Specifically, we tested two neural network architectures, each with 100 neurons per layer and depths of and four, and assossod model performance using the relative & error, defined as
1y Sll ||ul2
V
9 9.)2
1 VE
where 9 is the predicted solution and is the true solution. All networks used the Tanh activation and are trained using 10K equally spaced collocation points sampled from the domain. Relative &2 errors are reported on separate testing set of 20K points. All experiments used 32-bit floating-poin precision (float32) to ensure computational efficiency. Each model was trained for 100K ilerations using the Adamn optimizer (Kingma & Ba, 2015) with an initial learning rate of 10-3, decaying exponentially by 0.9 every 1000 iterations, followed by L-BFGS optimization until convergence (tolerance 10-9). For the strong BC PINN with Pexp: initialized a to 0.5 and jointly optimized it with all other parameters. Results are averaged over five random trials and were conducted on GeForce RTX 3090 GPU with CUDA version 12.3, running Ubuntu 20.04.6 LTS. Additionally, we implemented second-order accurate Finite Difference Method (FDM) bascline for solving the 1D Poisson and steady-state Allen-Cahi equations using 1000 discretization points and ren differences. Table in the Appendix provides detailed hyperparameters and experimental design information.
Standard PINN (d-2)
Strong BC PINN 4poly (d-4)
Strong PINN 4 (d-4)
Standard PINN (d-4)
Strong BC PINN 6exp (d-2)
FDM
Strong PINN pplyy (d-2)
1D Poisson
1D Allen-Cahn
10a
10
10a
100
Relative 12 error
Relative 1z error
10-1_
10-2
10
10-3
10-4
10
10-5
10
10
2
10
14
18 k
22
26
30
34
2
10
14
18
22
26
30
34
Figure 2: Relative & error in predicting the solution u(x)) sin(kx) function of frequency k. The plots compare the performance of the standard PINN, the strong BC PINN with polynomial boundary function Opoty, the strong BC PINN with exponential boundary function dezps and FDM for (left) 1D Poisson and (right) 1D steady-state Allen- Cahn equations for NN with hidden layers (d-2) and 4 hidden layers (d-4) each with 100 newrons. The shaded regions represent the error variability.

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 2 demonstrates that both strong BC PINNs, using Opoly and dexp boundary functions, effectively suppress high-frequency components, leading to higher accuracy solutions compared to the standard PINN. The deeper architecture (d=4) consistently reduces errors across most frequencies for both boundary fune- tions, but the static polynomial distance function Opoly shows diminishing benefits as the frequency increases. In contrast, the strong BC PINN with deep maintains superior accuracy, out outperforming both the slandard PINN and the strong BC PINN with Opoly across broader frequency range. Although the advantages of strong boundary enforcement diminish at very high frequencies, Φexp consistently retains its edge over Φpoly-
Additionally, the Finite Difference Method (FDM) achieves the lowest relative Eg errors across all frequencies in both the 1D Poisson and 1D Allen-Cahn problems, serving as benchmark for evaluating ncural network approaches. The FDM demonstrates exceptional performance, particularly at high frequencies, highlighting the limitations of PINN-based approaches in capturing rapid oscillations. However, the strong BC PINNs with Peep narrow the gap with FDM at lower and mid-range frequencies, showcasing their robustness com- pared to standard PINNs.
While the FDM is reliable and efficient approach for solving partial differential equations, it has notable limitations that drive the need for advancements in Physice-Iuformed Neural Networks (PINNs). FDM relies on structured grids, making it challenging to apply to problems with complex geometries, and it struggles with scalability in high-dimensional problems due to exponential grid growth. Unlike FDM, PINNs can also integrate noisy or partial data into their models, making them more flexible for real-world applications. Furthermore, advancements improved the practicality and efficiency of PINNs. By addressing current challenges, such as resolving high- frequency solutions and improving convergence, PINNs have the potential to provide a scalable, versatile alternative to traditional numerical methods like FDM.
machine learning architectures and hardware accelerators have significantly
Training times (in seconds) for various network configurations across 1D Poisson and 1D Allen-Cahn problems. The values in parentheses represent the standard deviations of training times.
Table 1:
Network
1D Poisson (sec)
1D Allen- Cahn (sec)
Standard PINN (d=2)
202.07 (2.52)
103.11 11 (0.10)
Standard PINN (d=4)
319.09 (1.61)
200.14 14 (91. 20)
Strong RC PINN polyy (d=2)
204.28 (1.95)
213.41 (78.60)
Strong RC PINN palay (d=4)
311.83 (10.83)
122.91 (15.90)
Strong BC PINN @exp (d=2)
299.56 (2.18)
403. 60 (260.00)
Strong BC PINN <lep (d=4)
401.63 (1.98)
472.59 (56.00)
FDM
4 Fourier Analysis of Strong BC PINNs
Both NNs and PINNs are known to exhibit spectral bias, meaning they readily capture low-frequency components of target function but struggle with high-frequency details (Basri et al. 2020: Rahaman et al., 2019 Xu et al., 2020 Wang el al. 2021c). As demonstrated earlier, this bias causes the solution qualily of PINNs to degrade the target solution's frequency increases. However, strong BC PINNs appear to mitigate or delay this issue, achieving better performance for higher-frequency solutions.
This section uses Fourier analysis to investigate why strong BC PINNs outperform standard PINNs in addressing spectral bias. The infinite Fourier series expresses a periodic function u(x) with period L as:
(I))
(16)
where the Fourier coefficients are given by:
1 /" u(x)e -2nn// dx.
ü|n]
(17)
8

---

Published in Transactions on Machine Learning Research (01/2025)
4.1 Numerical Fourier Analysis of Strong BC PINNs
To analyze spectral bias, we compute the Fourier coeflicients for several manufactured solutions: 1. A single sine wave to demonstrate the baseline frequency handling of PINNs. 2. A combination of sine waves with differing frequencies to explore how multi-scale components are captured. 3. A Gaussian-modulater sine wave to evaluate performance on solutions with contimuous spectral content. 4. A sine wave and polynomial combination to examine behavior under non-homogeneous boundary conditions.
For standard PINNs, the frequency spectrum typically shows a pronounced low-frequency peak, reflecting effective learning of low- cow-frequency components and gradual decay for higher frequencies, often resulting in heavy tails. These heavy tails indicate residual noise and the network's 's difficulty in accurately filtering out high-frequency inaccuracies. By contrast, as we will show, strong BC PINNs exhibit faster decay in the higher-frequency components, enabling better suppression of noise and improved representation of fine details in the solution.
We trained standard PINN and a strong BC PINN for each example with four hidden layers, 100 nodes per layer, and Tanh activations. Table 4 lists the specific training and hyperparameter delails. We then computed the Discrete Fourier Transform (DFT) of the learned solutions uz(z), which represents their frequency spectrum. For N equally spaced points zj sampled over the domain [0, 2T), the DFT is defined as:
N
N
To kk] =
Σ
us (z 12nkn/N k
1.
2
2
Figure 3:
Frequency spectrum of the learned solution (left) and the convolution operalion (right) in the strong BC PINN and slandard PINN. The ground-truth solution is sin(kx) with k == 15. The left graph shows frequency handling by standord and strong BC PINNs, while the right demonstrates how connolution reduces high-frequency noise.
Figure (left) compares the frequency spectrum of a standard PINN and a strong BC PINN trained on the ID Poisson problem with the ground-truth solution u(x) = sin(kr), for I [0, 2T] and = 15. While the standard PINN successfully captures the target frequency, also retains significant high-frequency noise, evident in the heavy tails of the spectrum. In contrast, the strong BC PINN exhibits much faster decay in higher frequencies, effectively reducing high-frequency noise and improving accuracy. This indicates that the strong BC PINN filters out unnecessary components more efficiently, enhancing the clarity and precision of the learned solution by isolating the dominant frequency.
The frequency spectrum of the leurned solutions (lop) and absolute error of the Fourier coefficients (bottom) for the strong BC PINN and standard. PINN compared to the ground truth. The ground-true truth solution. is u(x) = sin(2x) + sin(16x).
Figure 4:
Building on the analysis of single-frequeecy solutions in the previous section, we now examine the performance of standard and strong BC PINNs on a multi-scale target solution where multiple frequencies are present simultancously. This sctup provides a more challenging scenario, testing each model's ability to balance details.
between capturing dominant frequueue components and accurately representing For multi-scale target solution, u(x)= sin(2.) + sin(16x), Figure q iillustrates the frequency spectrum of the
9

---

Published in Transactions on Machine Learning Research (01/2025)
learned solutions. To simplify the analysis and focus on the unique information, we transition to visualizing only the positive frequency components, as the Fourier coefficients are symmetric for real-valued functions.
While the standard PINN captures the primary low-frequency components, it struggles to filter the higher- frequency components effectively, allowing residual figh-freyyency noise to persist. The strong BC PINN, however, captures both frequencies more accurately and suppresses extraneous high-frequency noise, leading to a lower overall error. This result highlights the strong BC PINN's advantage in multi-scale problems, where it can effectively balance between capturing necessary frequencies and eliminating noise.
The frequency spectrum of the learned solutions (top) and absolute error of the Fourier coefficients (boltom) for the strong BC PINN and standard PINN compared to the ground truth. The ground truth solution is u(x)= = - u.322 sin(16x).
Figure 5:
To evaluate the strong BC PINN's performance on solutions with continuous spectral content, we used a Caussian-modulatee simusoid, u(z) = e 0.5x" sin(16z). Figure 5 shows the Fourier spectrum for this case. The Gaussian modulation introduces broad spectrum of frequencies, challenging the model's ability to suppress irrelevant high frequencies while accurately capturing the primary frequency content. In this example, the strong BC PINN demonstrates robust noise reduction by filtering unnecessary frequencies, including low frequencies. This capability is essential for handling continuous spectra. The strong BC PINN can gencralize well to functions with non-discrete frequency components, making it applicable to a broader range of real-world problems.
Figure 6:
The frequency spectrum of the learned solutions (lop) and absolute error of the Fowrier coefficients (bottom) for the strong BC PINN and standard PINN compared to the ground truth. The ground truth solution is u(z)= = x² + sin(16z).
Finally, we explore the strong BC PINN's performance on non-homogeneous boundary conditions using the manufactured solution u(x) x² + sin(162). Figure 6 illustrates the Fourier spectrum for this non- homogeneous boundary case. The polynomial term x² introduces a low-frequency component that is not harmonically related to the oscillatory term sin(16r). The standard PINN struggles to capture this low- froquency component fully. At the same time, the strong BC PINN demonstrates improved performance in accurately representing both the polynomial boundary effect and the oscillatory component. This ex- ample underscores the strong BC PINN's effectiveness handling non-homogeneous boundaries, as it can accommodate mixed low- and high-frequency components in the solution.
These results collectively illustrate the strong BC PINN's enhanced ability to manage various frequency cases, from simple single-frequency cases to continuous spectra and complex non-homogencous boundary conditions. The strong BC PINN demonstrates robustness and flexibility by effectively suppressing high- frequency noise and maintaining accuracy across various spectral components, making it suitable for a wide array of boundary-conditionee problems in applied settings.
4.2 Analytical Fourier Analysis of Strong BC PINNs
To analyze why multiplying the distance function (o(x)) with the NN in Equation g helps to obtain more accurate coefficients for high frequencies, first represent the general distance function ((x) as infinite
10

---

Published in Transactions on Machine Learning Research (01/2025)
Fourier series in its discrete form as in Equation 16
((x) (x)
Σ [[n] e
(18)
with period L = 2x. We first analyze the polynomial distance function by substituting Equation for Ø in Equation
18 From Equation 17 get the Fourier coefficients,
/ z(2n - x) e inn dx = n² ppoty [n] = 2xt 2x
(19)
Similarly, to compare against our proposed adaptive distance function, we substitute in Equation 10 for / in Equation 18 and obtain the Fourier coefficients,
/ 11-e a1) -e cale-2r)) e -in a(1 (11-22ma) 2 2na 2T 27
Seapl [n]
einz
(20)
Taa2 + n22)
Here, we expand the boundary functions o(x) to be periodic (i.e., duplicating its definition in [0, 2T] to other intervals); and the period is 2x. We can now obtain the Fourier transforms of each (zo(2) by evaluating,
((w) = Σ H|u] 5 (w
n
(21)
2T
where 5() is the Dirac delta function and w is the continuous frequency. Specifically, each ?(w) represents the Fourier transform of do)())), meaning F-1 { ((w)) (x) where F-1 denotes the inverse Fourier transform.
Since [n]] x 1/n², the amplitude of the frequencies of (x) decay quadratically fast with the increase of the absolute frequency value. This rapid decay is beneficial for denoising as it effectively suppresses high- frequencies, leading to smoother approximations. However, it may result in the loss of fine details, especially if essential signal components are present at higher frequencies. Alternatively, [d=mp[=]] x a/(a² + n²), indicating that the frequency amplitudes follow Lorentzian decay controlled by the decay rate a. Lorentzian decay is less aggressive reducing high-frequency components, which helps preserve fine details and sharp transitions in the signal. This flexibility helps tune the decay based on signal characteristics. However, the slower decay rate may lead to less effective noise suppression.
We can now look at the frequency spectrum of the full surrogate model ua(x) in Equation We use the convolution theorem [McGillem & Cooper, 1991] to obtain
(o(w)= = - UN 1 سlw - v) (v) du
(22)
where denotes convolution, and uN is the Fourier transform of the NN. Suppose > O, know that H(w - v) is obtained by first reflecting ò(v) about the y-axis, and then shifting the frequency left along the spectrum. Then, we integrate (x) weighted by س(w -v). The larger the frequency w, the more & is moved left to obtain ((w-v) v) resulting a stronger weighting of the high-frequency components during integration. Since the frequency coefficients of å decay quadratically fast ([ō[n]] x 1/n²), the larger the portion of the tail used, and the smaller the integration result. Specifically, with Φ the corresponding amplitude of top (w) decrease fast when w increase:?] See Figure 3 for an illustration in the discrete case. During training, the boundary function pushes the surrogate model in Equation g to weaken irrelevant high-frequenc, components to reduce large tails and betler capture the frequency spectrum.
While the analysis here is presented in the discrete Fourier series form for clarity and consistency with the numerical implementation, the results be extended to the continuous Fourier trausform by replacing summations discrete indices with integrals over the continuous frequency domain.
2Th same conclusion applies when we consider < 0.
11

---

Published in Transactions on Machine Learning Research (01/2025)
5
Fourier PINNs
Our Fourier analysis in the previous section revealed that the standard PINNs inefficiently capture the true amplitude of frequencies in the power spectrum of their predicted solutions. Specifically, the analysis showed that although standard PINNs could identify the true solution frequency, they failed to prune large and noisy frequencies, as indicated by the large amplitudes assigned at the tail end of the power spectrum in Figure 3 While the strong BC PINNs address this issue by multiplying the NN with a boundary function which helps decay high-frequency amplitudes faster- their overall benefit remains limited. As shown in Figure 2 performance degrades k increases, indicating the boundary function's 's inability to learn ultra- high frequency solutions. The strong BC PINNs are also restricted to specific boundary conditions, such Dirichlet conditions in Equation 7 limiting their applicability.
5.1 Fourier PINNs
We introduce Fourier PINNs, novel PINN architecture to overcome these challenges. Fourier PINNs seam- lessly handle diverse boundary conditions, like the standard PINNs, while emphasizing the true solution frequencies during training akin to the strong BC PINNs. Specifically, in order to flexibly yet compre- hensively capture the frequency spectrum, we first introduce a set of dense frequency candidates, {wa) evenly sampled from the range K]. From this set of frequency candidates, we define a set of trainable Fourier bases ug as,
K
un(x;a,b) = Σ
Un cos (2TT,)))) + b, sin (2Twin x).
(23)
The Fourier PINN architecture additively combines the outputs of both the NN from Equation 3 and the Fourier bases from Equation 23 to generate the augmented prediction up. Specifically, the full output of Fourier PINNs is,
W uF(x; 0) = (x; = 0 + Σ an cos(2Tun + bn sin (2Tri,))
K
(24)
j=1
We define w fei,..., CW; 01, .... ak, , .... bx} as the set of all basis coefficients, and 0 =O"H, w] form the set of all trainable network parameters.
While present method in the one-dimensional input case, extending it to multiple dimensions is straightforward. Consider the two-dinensional input x = [1) for example. We use the same set of frequencies to construct Fourier bases for each input. Specifically, we construct
((11) [00w(27))11), sin(2mw171), .... cos(22wx:1)) sin(2mwx11),,
and
((x2) (2Tu)) sin sin(22)172) cos(2mwxa2)) sin(2rwxx2))
Then, we apply the cross-product of (xx1) and (xx2) to obtain the tensor-product Fourier bases for x. The surrogate model given by,
= = + B vee (#(11)((22))
(25)
where vec(-) denotes the vectorization, and 9 are the coefficients of the Fourier bases. Note that the lensor-product of bases grows exponentially with the problem dimension, quickly becoming computationally intractable. One potential solution is to generate more sparse Fourier bases, such as total-degree or hyperbolic cross bascs. Another option is to re-organize the cocfficients into a tensor or matrix and introduce low- rank decomposition structure to parameterize /3 to save computational cost (Novikov et al. 2015). These explorations are left. for future work.
12

---

Published in Transactions on Machine Learning Research (01/2025)
5.2 Adaptive Basis Selection Training Algorithm
Given that the ground-truth solution likely contains significantly fewer frequencies than the augmented candidate bases (ug), and our previous analysis showed that models often struggle to prioritize learning necessary Irequencies, we developed an adaplive learning and basis selection algorithm. This algorithm flexibly identifies meaningful frequencies while pruning the inconsequential ones allowing the model to focus on learning and improving the bases that significantly contribute to the solution by inhibiting the learning of unnecessary (and likely noisy) frequencies.
To aid in identifying the significant frequencies, we add an Le regularization term to the basis parameters w. While L2 regularization does not promote sparsity, in our specific case, we use L2 regularization with a pruning strategy, which indirectly promotes sparsity by pruning coefficients after driving them toward a threshold. Methods such as the SINDy (Sparse Identification of Nonlinear Dynamics) algorithm Zhhan & Schaeffer (2019) have explored this strategy, where Le regularization helps to stabilize the coefficients during optimization before pruning-yielding sparse solutions. This combination effectively balances stability and sparsity, as shown in previous work Wang et al. (001a)). In this case, we chose not to use L, regularization due to practical challenges, including the instability it can introduce during training, particularly when optimizing neural networks for complex systems. L2 regularization, combined with a careful pruning process, offers more stable convergence while still leading to sparse solutions after pruming the small coefficients.
This additional regularization penalizes large cocfficients for the basis functions, thereby promoting sparsity and encouraging the model to focus on learning the most siguificant frequencies. The full loss function we use is similar to PINNs (see Equation 4 but with the addition of the L2 regularization term. Specifically, the loss function is,
L( = (0) + (0) - 2 ||w|?
(26)
where ||w|] represents the L2 regularization of w and is the regularization strength defined by the user before initiating training. This regularization term helps prune or inhibit unnecessary frequencies, allowing the model to focus more on learning and improving the bases that significantly contribute to the solution.
Linear Operators Our adaptive basis selection algorithm optimizes the regularized loss function in Equa- tion 26 by building upon the hybrid least squares gradient descent method proposed by CCr et al. (0220) Cyr et al. 2020) formulates the output of a neural network as a linear combination of nonlinear basis func- tions, as shown in Equation 3. Their algorithm alternates between optimizing the hiddon weights (0# via gradient descent and the final layer coefficients (c) of the NN through the least squares problem of the form,
- arg mine ||Acl yylle,
(27)
such that y, = (x,) for the data points 1x) Ay = (x;; 0²)] for j = 1,... W, and C is some linear operator. Equation 27 extends to problems with multiple operators, such as those defined by Equation T and Equation 2 when JF and B are linear operators.
Representing the output of Fourier PINNs in Equation 23 as a linear combination of both adaptive bases (i.e., the hidden layers of the NN) and an augmented Fourier series ensures our model's compatibility with the general approach of the alternating least squares and gradient descent optimization. While Cyr et al. (2020) hold the coefficients of the last. layer in the NN (c) constant while optimizing the bases 0 through gradient descent, we alternatively choose to jointly optimize both the Fourier PINN's basis coefficients w (which are comprised of both the NN basis cocfficients along with the Fourier basis coefficients dı,..., aK and bi,...,by bk) and the adaptive bases 0# during the gradient descent step. During the least squares step, we solve similar problem to Equation 27 modified to suit the Fourier PINN architecture. Specifically, we solve the least squares problem of the form,
arg min AA'w y1
(28)
13

---

Published in Transactions on Machine Learning Research (01/2025)
such that y, (x,) for the data points fx, ) and
(xxs 0²]]
for j 1y.., W,,
A/3 =
for j = W + 1,.,,, W + K,
(29)
(2sin(2mu)xx)
for j W + + 2, W 2K.
We define Equation 28 for one operator L, but the method extends to multiple operators. Additionally, we extend our least-squares method to nonlinear operators, unlike many works that focus only on linear cases.
Non-Linear Operators When the operator & is nonlinear, updating the coefficients w requires addi- tional care to ensure the method accurately handles the nonlinearities and achieves convergence. Nonlinear operators can often be decomposed into a combination of linear and nonlinear components, simplifying the process of updating w during optimization. Specifically, we docompose the operator as follows:
c|- = gl.l + SH
(30)
where G represents the lincar part of the operator, and S denotes the nonlinear component.
To illustrate this, consider the Allen-Cahn equation, where the operator L contains both linear and nonlinear terms. In this case, the linear part G is Uge, and the nonlinear term Slu] is u(u² - 1), leading to the dccomposition:
glu]=
Ugzs
Slu)
-(u²-1). 1).
(31)
We solve this system by incorporating the Fourier PINN formulation. For the least squares solution of the system, the nonlinear operator S is applied at each iteration using the current approximation of u, which is based the current coefficients and is given as,
S; = S(uf](x:) = (x (xF(( w)² w
(32)
Here, ur(x;; w) is the Fourier PINN's evaluation at point xi, using the current parameters w. The linear part G is handled directly in the least squares system, and the matrix A' is defined as:
(x 0²)]
for j = l.... W,
A/g
Glcos (2TW,x))
for j W + 1,..., W + K,
(33)
((sin(2Twyxi)
for j = W + K + 2, W + 2K.
The nonlinear lerms 8; are included the target vector such that y; [[]]((x)) - 8,, which is updated iteratively and reduces the problem to linear least-squares system. As this method incorporates the current values of w into calculating the nonlinear components, the optimization process can be viewed form of fixed-point iteration. We note that for cases where consists primarily of nonlinear terms does not include linear operator, the optimization problem might need to be handled using continuous optimization algorithmss such as L-BFGS, which effectively updates w based on the entire system to ensure convergence.
We integrate custom basis pruning routine into the optimization algorithm eliminate insignificant bases and refine model training. Our modified loss function defined in Equation 26 enhances this hybrid optimiza- tion algorithm with an La regularization term to identify and prioritize significant frequencies. Algorithm [ outlines adaptive learning routine. The algorithm begins with joint optimization of 0 and to provide warm start to the model parameters. Next, we fix l while alternating between solving for the coefficients w using the least squares method and truncating bases with coefficients below predefined threshold. Specifically, bases corresponding to |wy| I 10-3 are removed along with wy. After completing the alternating solve and truncating steps for set number iterations, the algorithm jointly optimizes all remaining parameters and bases using the Adam optimization algorithm. We repeat this routine for fixed number of iterations, followed by L-BFGS optimization until the final convergence.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm 1: Adaptive Basis Sclection Hybrid Least Squaaces/Gradiets Descent
Input: 0f (initialized hidden parameters)
Output: Optimized parameters 0 and weights w w < TS(O))
// Solve LS problem for w // Initialize 0f and
0" w ADAM(O w)
for i == 1,... do
for k = L,... do
w t -SS(
// Solve LS problem for w
if Wyy < for each wy then
Prune the corresponding basis;
Delete wy from w;
end end
8" w + ADAM(0", .w);
// Jointly optimize all parameters using ADAM
end
Run L-BFGS until convergence;
6 Experiments
This section comprehensively evaluates our methods across a diverse set of partial differential equation (PDE) problems. The test cases include the 1D and 2D Poisson equations, which serve as canonical examples of linear elliptic PDEs, and the 1D and 2D steady-stale Allen-Cahn equations, representing nonlinear elliptic PDEs with nonlinear reaction term. To further challenge the models, we analyze the ID one-way Wave equation, which incorporates first-order time derivative, and the 1D non-steady-stata Allen-Cahn equation, stiff PDE with non-harmonic characteristics. We first describe the baseline methods for comparing Fourier PINNs and then outline the experimental setup. Finally, we present and discuss the results, highlighting the relative performance of each method on the benchmark problems.
6.1 Baselines
We benchmarked against the following statc-of-tho-art PINN models for solving high-frequency and multi- scale solutions:
(1) (PINN) standard PINNs as formulated in (Raissi et al. 2017);
(2) (RFF-PINN) Random Fourier Feature PINNs (Wang et al.) 2021ch with dynamically re-weighted loss terms derived from the NTK eigenvalues in Wang et al. 2002
(3) (W-PINN) Weighted PINNs that down-woight the residual loss term to reduce its dominance and to fit the boundary loss Wang et al. 2022 better;
(4) (A-PINN) Adaptive PINNs with parameterized activation functions to increase the NN capacity and to be less prone to gradient vanishing and exploding Jagtap et al. (2020); and
(5) (Spectral) A single Fourier layer comprised of Fourier bases and trainable coefficients. Basis coeffi- cients are optimized through the same gradient descent routines as all other baselines
We do not test against the conceptually similar Fourier Noural Operators (FNOs) [Li ct al. 2021), ) as they are designed for inverse problems and rely on a data loss term. In costr our method focuses on solving forward problems and does not require training data within the domain. Additionally, while the methods listed above represent widely-used PINN architectures, we acknowledge the existence of alternative neural
Throogghuu the experiments, we used the same of Fourier hases for the spectral method and Fourier PINNs.
15

---

Published in Transactions on Machine Learning Research (01/2025)
network approaches based random features and random projections, such Random Vector Functional
Link networks (RVFLNs) [hang & Suganthan| 2016), Extreme Learning Machines (ELMs) (Ding et al.
2013
Fabiani et al. 2021 Calabrò et al. 2021). Random Projection Neural Networks (RPNNs) (Rahimi &
Recht
2008 Reservoir Computing (Pathak et al. 2018), and recent advancements like RandONet (Fabiani
et al.
2025 While methods such as RandONets and RVFLNs demonstrate significant computational
efficiency and excel inverse problem-solving tasks, their ability to address spectral bias in forward high-
frequency problems remains underexplored.
By contrast, the proposed Fourier PINN architecture directly targets spectral bias a key limitation of exist- ing PINN approaches in learning high-frequency and multi-scale solutions. Unlike conventional architectures, the Fourier PINN framework is both modular and adaptable, with the integration of Fourier bases capable of enhancing or complementing any of these alternative methods. Given its flexibility and minimal downsides, exploring how Fourier PINNs could be combined with these approaches to improve scalability and accuracy remains an exciting direction for future work.
The number and corresponding scales of Gaussian var variances utilized in the RFF-PINN. Seales were selected randomly from set range of [00]. Preference was given to the subset [1, 20, 50, 100] to ensure balanced distribution across the available range.
Table 2:
Number
Scales
(20), (50), (84), (100)
2
(1,50), (3, 20), (19, 71), (39, 69), (50, 100)
3
(44, 47, 165); (1,20, 194), (20, 50, 100), (1,50, 189), (38, 112, 119)
5
(1,20, 49, 50, 100), (1, (1, 50, 85, 100)
(1,20, 104, 197, 199), (6,36, 36, 67, 79, 136), (50, 65, 83, 104, 139)
6.2 Hyperparameters and experimental details
For each experiment, we trained the models in two stages: first using the Adam optimizer (Kiggma & Ba 2015), followed by T-BFGS optimization (Liu & Nocedal 1989) until convergence, with the tolerance set to Specific hyperparameters and training details are provided in the following tables: Table for the 1D Poisson and 1D Allen-Cahn experiments, Table 6 for the 1D Wave equation experiments, Table 7 for the 2D Poisson and 2D (steady-state) Allen- experiments, and Table the 1D (nou-steady-statte) Allen-Cahn experiment. All experiments use 32-bit doating-point precision (float32) to ensure computational efficiency and consistency across models and were conducted on GeForce RTX 3090 GPU with CUDA version 12.3, running on Ubuntu 20.04.6 T.TS. Tn Fourier PINNs, we determined the range of frequency candidates for the Fourier bases by setting maximum frequency K and including all equally spaced frequencies in the set {1,2,..,,K) K}. For the alternating optimization algorithm (Algorithm D. we set the basis truncation threshold = 10-1 and the regularization strength parameter a = 10-1 Optimization used the same number of Adam iterations across experiments, with pauses at every 1000th iteration, to solve the LS problem for five iterations. RFF-PINNs required specifying the number and scales of Gaussian variances used to construct the random features. To evaluate the effects of these hyperparameters, we tested 20 different settings, varying the number of scales (one, two, three, and five) and using variances suggested in (Wang et al., 2021ch along with randomly sampled values. Table 2 provides the exact sets of scales. For W-PINN, we varied the weight of the residual loss from {10-1, 10-3, 10 10-4} and reported the configuration achieving the best results. For A-PINN, we introduced a learned parameter for the activation function in each layer and updated these parameters jointly. We ran every method for five random trials and reported the five number summarics obtained on seperate testing dalasets for each experiment boxplot form. All code is implemented using the PyTorch C++ library Paszke et al. 2019
10
&Code is available at https:/ http:://ggthhub.oom/eeo htt.a/bbco.oooaaaaaaaaaaa/eee Kaw.nakaaaaeeneeeeeaaaaaaeee/e tteee///eee///m//
16

---

Published in Transactions on Machine Learning Research (01/2025)
6.3 1D Numerical Experiments
This section evaluates Fourier PINNs on three progressively challenging 1D Poisson and Allen-Cahn bench- marks, demonstrating their superior performance and robustness over the baseline methods. These bench- marks systematically assess the models' ability to handle high-froquency, multi-scale, and hybrid-frequeecc solutions for linear and non-linear elliptic PDEs. The analysis includes relative le error comparisons, conver- gence dynamics, and frequency spectrum decompasition to highlight the strengths of Fourier PINNs. Table 5 in the Appendix lists each example's hyperparameters and experimental details.
6.3.1 1D Poisson Problem
The 1D Poisson problem evaluates the ability of Fourier PINNs to capture frequeryy signals within a linear context. We test the following manufactured solutions with increasing frequency complexity:
u(x) sin(100.r),
(34)
u(x) sin(z) + 0.1 sin()0x) + 0.05 cos(l00
(35)
m(x:) = sin cos
(36)
The boundary conditions are derived directly from the true solutions. These tests progressively introduce multi-scale and hybrid frequency components used to study the robustness and scalability of each model.
(a)
(b)
RFF-PINN (3 scales)
AFF-PINN (2
Fourier PINN
REF-PINN scales)
Relative & Error
107-
107-
Relative I, Error
10-
0
5000
10000
15000
20000
25000
30000
35000
40000
Iterations
10-4
Re 12 Error
10a.
107-
10-2_
RFF-PINN (2, 50)
RFF-PINH 20)
AFF-PINN IS, 20, 194)
RFF-PIRN (1, 20, 49, su, 100)
RFF- IFF-PINN 12, su, 189)
RFF-PINN LI, 20, so, 85, 200)
RFF-PINN (1, 20, 104, 197, 199)
RFF-PIAN (6, 36, 67, 78, 136)
Fourier PINN (K=150)
Fourier PINN (K=200)
Fourier PINN (K=250)
Fourler PINN (K=300)
10
10a Training Time (s)
PINN
RFF-PINN [3., 26, 104, 197, 199)
Fourter PINN 200)
RFF-PINN [3., 54)
(c)
(d)
Tourte
10011
Figure T: Results for the single-scale true solution u(x) = si0(100x). (a) Borplot showing the relative &a errors for Pourier PINNs and the top-perforning baseline methods. (b) Convergence plots comparing the relative error across iderations (top) and training time (boltom) for selected models. Vertical lines indicate when each model reached an error of 10-2- (c) Frequency spectrum of the solution u(x) compared to PINN and Fourier PINN approzimations. (d) Decomposition of the frequency components in Fourier PINNs between the NN output and the Fourier layer output, highlighting differences in spectral representation.
Single-Scall Scale Solution Results. Figure 7a shows the relative & error distributions for Fourier PINNs and top baseline models, while Figure 15a the Appendix includes results for all models. Fourier PINNs
17

---

Published in Transactions on Machine Learning Research (01/2025)
consistently achieve significantly lower errors than baselines, including RFF-PINNs and spectral methods. Notably, RFF-PINNs demonstrate sensitivity to the chosen scale configurations, which impacts their overall performance. Figure 7b illustrates the convergence behavior across iterations and training time to analyze training dynamics. Fourier PINNs cxhibit rapid and stable convergence due to the alternating optimization and basis truncation routine. In contrast, standard PINNs plateau at higher errors, struggling to approximate the high-frequency features of the solution accurately.
As shown in Figure [c. the frequency domain analysis compares the Discrete Fourier Transform (DFT) of the solutions from standard PINNs and Fourier PINNs against the ground truth. Fourier PINNs generate frequency spectra almost identical to the ground truth, effectively capturing the high-frequency features. Conversely, standard PINNs fail to resolve the high-frequency components, underscoring their limitations for high-frequency problems. To further dissect Fourier PINN solutions, we decomposed the predicted solution, up = uN + "B, into its neural network (NN) and Fourier layer components. Figure 7d presents the froquency spectra of each component. The absence of green bars (representing the NN component) indicates no contribution from the NN basis. This result aligns with the basis truncation outcome, where all NN bases were truncated during training, leaving only single Fourier basis. This observation confirms that the optimization algorithm accurately captures the frequency components of the true solution.
(b)
RFF-PINN scales)
RFF-PINN scales)
Fourier
Relative i; Error
201
10-k
10
20-
5e00
10000
15000
20000
25000
3000n
35000
4n0n0
Relative
10-4
Itarations
200-
10.5
Relative Ig
20
REFF RLFF-PINW (20)
RF IFT-PINN (L. (0s
dmM-ieaa 13, 20)
RFE-nIM (L, 20, 196)
BFF-DIIN (1, 20, 40, so, 100)
RFiJSIsN (I. so, 189)
aFF-PINN 20, so, as, 100)
RPF-PINN (1, 20, 104, 197, 199)
RPF RPF-PINN 36, 07, 75, 136)
Fouriel PINN (K-)
Fourler PINN (K=)
Fourier PINN (50)
Fourier PINN (K=300)
20-1
10
102
10P
Training Time (s)
PNN
RFF- PINN (b. 20, 105, 197, 1991 RFF-PINM (3. 50)
Fourfer PINN DC 200g
(c)
(d)
10
Sex
Stavibs PROM
10
Figure 8: (1D Puisson) Results for the maalii-scale true solution u(x) sin(x) 0.1 sin 0.05 cus (100x). (a) Bozplot showing the relative & errors for Fourier PINNs and the lop-performing baseline methods. (b) Convergence plots comparing the relative la error across iterations (top) and training time (bottom) for selected models. Vertical lines indicate when each model reached on error of 10-3 (c) Frequency spectrum of the solution u(x) compared to PINN and Fourier PINN approzimations. the NN output and the Fourier layer output, highlighting differences in spectral representation.
(d) Decomposition of the frequency components in Fourier PINNs between
Multi-Scale Solution Results. The multi-scale problem introduces combination of low- and high- frequency components, presenting a greater challenge for baseline methods. Figure Salshows the relative & error distributions for Fourier PINNs and top bascline models, while Figure [5b in the Appendix includes results for all models. Fourier PINNs outperform most baseline methods; however, subset of RFF-PINN configurations achieves slightly better results. These cases highlight. the sensitivity of RFF-PINNs to scale
18

---

Published in Transactions on Machine Learning Research (01/2025)
selection, whereas Fourier PINNs demonstrate robust performance across all tested basis configurations. RFF-PINNs exhibit more significant variability across configurations, with inconsistent performance due to their reliance on precise scale tuning.
The frequency spectrum comparison in Figure Sc illustrates that Fourier PINNs accurately reconstruct both low- and high-frequency components, highlighting their adaptability to multi-scale structures. In contrast, the spectrum of the standard PINN output shows that it accurately captures low-frequency components but struggles to resolve the solution's high-frequency features. Figure [dd] further analyzes the frequency decomposition of the Fourier PINN solution. The Fourier layer captures all high-frequency components of the target solution (c.g., sin(20x) and cos(100z) in Equation 35), while the neural network (NN) output contributes to modeling the low-frequency component (sin(x)). This result aligns with the basis truncation outcomes: three Fourier bases were retained, and approximately 10% of the NN basis functions were trun- cated during training. These findings demonstrate that the hybrid architecture of Fourier PINNs, combined with the alternating optimization and truncation algorithm, effectively balances the modcling of both smooth and oscillatory components in multi-scale solutions.
(b)
Spectral PINN scales) PINN scales)
Fourter PINN (Alteratiin spt-)
Rolative J, Error
101
10a-
10
20-24
Ralative Fa Error
20-1
10000
20000
40000
50000
Rerations
10a
Relative f; Error
10a_
10
Spectral aPk eak 11)
ten- (co)
art pwen (re)
apt apr-mmm (14)
ary Pamu s0, 100)
PHRN (160)
Fourter FIsh (K- (20)
Fouder pann (5-1
Fourter (ko (ox220))
Foorter Prow tin Adum only)
Faurter Fiust (Kx2100)
Yourter vmmi 150, Adamn only)
foutter rosn of uzoo,, Adamn ooly!
Fourter pons (Kx250 dutuss owly)
(K- (8-1000)
Fourter rilm co-300, Adum Ooly)
20-1
foorter
101
10 Training Time (s)
10
Fourter Fuin (5-3500
et Aaam deay
251, Adam
Pownter mas
Poutter POUS Le- 130, Adua Oaly)
Figure 9: (1D Poisson) Results for the single-scale true solution u(x) sin cos(1002). (a) Bozplot showing the relative E errors for Fourier PINNs and the lop-performing baseline methods. (b) Convergence plots comparing the relative la error across ilerations (lop) and training time (bollom) for selected models.
Hybrid Solution Results. This fubrrddfrrquuency example introduces closely spaced high freeufryyyyncy com- ponents, which increases the complexity of disentangling and modeling each frequency. Figure Dal illustrates the relative l error distributions for Fourier PINNs and top baseline models, while Figure 15c in the Ap- pendix presents results for all models. To specifically evaluate the impact of the alternating optimization algorithm on Fourier PINNs, we tested multiple basis configurations using the alternating optimization mothod described in Algorithm [ as well as traditional optimization routines (i.c., without the least-squares (LS) basis coefficient optimization).
The results demonstrate that Fourier PINNs, when combined with the alternating optimization strategy, consistently achieve the lowest & errors across all tested configurations, highlighting their ability to effec- tively model solutions with coupled and closely spaced frequency components. In contrast, Fourier PINNs without alternating optimization fail to accurately capture the true solution, emphasizing the critical role of the LS basis coofficient updates in resolving hybrid-froquency structures. The convergence behavior shown in Figure 9h further underscores the efficiency of Fourier PINNs in learning hybrid solutions. Their rapid convergence and stable performance across iterations showcase their robust and adaptability to challeng-
19

---

Published in Transactions on Machine Learning Research (01/2025)
ing problem setups with complex frequency interactions. Additionally, there is little variation in training time with increasing Fourier basis sizes.
6.3.2 1D Steady-State Allen-Cahn Problem
Next, we examine the 1D steady-state Allen-Cahn equation, which is a nonlinear reaction-diffusion system. This problem serves as more complex PDE benchmark for evaluating Fourier PINNs and baseline models. To ensure consistency, we test the same manufactured solutions used in the 1D Poisson case, defined by Equation 34 Equation 35 and Equation 36
Results. Figure 16 presents the relative l error distributions for Fourier PINNs and the top-performing baseline models across all manufactured solutions, while detailed results for all models can be found in Figure 16 in the Appendix. Across all confiuraatios,, Fourier PINNs consistently achieve lower relative & crrors. Additionally, their faster convergence, as highlighted Figure 11 undorscores the efficiency of the Fourier PINN framework for solving the 1D steady-state Allen-Cale Cahn problem.
811-288 soin!
Fourter Fiint
saleet
395.39
12
12x23
180022
areaaees B80
seroroee 200
preanss stoe
2.25 Dy toes
fourter se-atoop
75 Liee
rourtey poout du-gtoy
Fencter (eudas))
er
AFF WERE AL
Prester POHH
Rt,
287,
are veevio
www. Cos
aOP 25,
Fouster
Fouris
Figure 10: (1D Cahnn Boxplot of relative 82 errors for Fourier PINNs and perfofffooinn baseline methods across the manafactured solutions: left, u(x) = sin(100x); middle, u(x) = sin(x) + 0.1 sin(ii + 0.05 oos000zz;; and right, u(x)= sin(6x) c100x....
Ralative trror
Relative,, Error
Relative Error
Relative Erer
100
100-
307-
10k
10.7
set
3.
10000
20000
30000
40000
10a
10'
10000 20000 Iterations
30600
40600
10'
10'
16'
iterations
Training Time (s)
Training Time (s)
HMN
RFF-PINN (2, 29, 197, 1991
PINN
HFF-FNNN 11, 104, 197, 1993
Fourier PAN (C 200)
RFF-PINN (1. 50)
Fourter PAN (K 2001
RFF PBN- 11, 50)
Figure 11: relative l2 error as a function of iterations (left) and training time (right). Results are presented for u(x) = -is (100z) (left two plots) and u(x) sin(z) + 0.1 sin(20x) 0.05 cos( (10x)) (right two plots).
Convergence behavior of select models on the 1D steady-state Allen- Cahn problems, showing the meon
6.4 2D Numerical Experiments
We evaluate Fourier PINNs on two 2D PDEs: the 2D Poisson problem and the steady-state Allen-Cahn oquation. These examples test the model's scalability, ability capture high-frequency components, and approximation of multi-scale solutions in higher dimensions. Table in the Appendix lists each example's hyperparameters and experimental details.
20

---

Published in Transactions on Machine Learning Research (01/2025)
6.4.1 2D Poisson Problem
The 2D Poisson problem is given by:
Au= = [(x, 2, y E [0,2m]]
(37)
with the following ground-trute solutions:
(1, y) sin (1000)) sin (100g),
(38)
u(x, y) sin cos((20x) + sin sin(Gy) cos(20y).
(39)
Relative trror
107-
100
Relative I, Error
10
Relative Error
1.0
101
30-1
10-1
10-1
20-4
10000
20000
20000
40000
10
10000 20000 20000
49000
10t Training Time (6)
Iteations
Training Time
Iterations
2891
(K- 200)
WFT-PINN
Foutier MNN 2001
HIF-NNN
Figure 12: Convergence behavior of seluct models the 2D Poisson problems, showing the mean relutive la error as function of iterations (left) and training time (righd). Resulls are presented for Equation 38 (left two plots) and Equation 39 (right two plots).
Results. For the high-frequency solution (Equation 38 Fourier PINNs achieve an average relative le error of 0.0384, outperforming all baseline methods, with the spectral method being the closest competitor at 0.0423. The Appendix provides detailed results for all models in Figure 17 Notably, other methods fail to capture the high-freqeency features, with errors exceeding 1. For the multi-scale solution (Equation 39 Fourier PINNs demonstrate significant advantage, achieving surpassing the performance of all other baselines. Figure 12 highlights the superior convergence behavior of Fourier PINNs in both scenarios, underscoring their ability to handle challenging high-frequency and multi-scale problems effectively.
average relative & error of 0.00085, lar
6.4.2 2D Steady-State Allen-Cahn Equation
The Allen-Cate Cahn problem is defined as:
Uez Myy + uuuu== -1) f( 2,y E [0,2m],
(40)
with the ground-truth solution:
ul (z, 3) (sin(z) 0.1 sin(20x) + cos(100r)) (sin(y) + 0.1 sin(20y) + cos(100y))
(41)
The Allen-Cahn example highlights Fourier PINNs' challenges with nonlincar PDEs. The model achieves relative & error of 0.9, outperforming all other methods (> 1). In the Appendix, Figure 18 provides detailed results for all models. This suggests the need for further optimization for highly nonlinear problems.
Results:
6.5 1D One-Way Wave Problem
We next tested the ID One Way Wave equation of the form,
4t + 10u, v(x,t). t). E [0, 1]. [o, u(r 0, t) u(x = =
(42)
u(z, f 0) sin(Tz)) + (2
21

---

Published in Transactions on Machine Learning Research (01/2025)
Spectral PINN
Strong W-PINN
Fourier
RFF-PINN scales)
10
Relative is Error
2
10
10
Spectral
Strong BC PINN
PINN
W-PINN
Fourier PINN (reg prune)
A-PINN
RFF-PINN -IN- (20)
Fourier PINN (no reg + prune)
Fourier PINN (reg no prune)
Fourier P PINN (no reg no prune)
Figure 13: (ID One- One-Way Wave) Bosplot of relative la ervors for Fourier PINNs and top-performing baseline methods.
with a true solution of,
ulx,t) sin(xxx) cos(() + sin(2xx)) cos
(43)
This problem introduces a time-dependent component, testing the ability of models handle first-time derivatives within high-frequency context. We conducted an ablation study to assess the roles of basis pruning and regularization by evaluating Fourier PINNs without these mechanisms. This study highlights the distinct contributions of basis pruning and regularization to model accuracy and computational efficiency. Table in the Appendix lists all hyperparameters and experimental details.
Figure shows the relative & error distributions for Fourier PINNs and top- top-performing baseline models across all manufactured solutions. Detailed results for all models are provided in Figure 19 in the Appendix. As expected, Fourier PINNs with both basis regularization and pruning (denoted as 'reg + prune") achieve the best rosults, while the configuration using neither (denoted "no reg + prune") yields the worst performance.
PONNIer-PINN Without Basis Pruning. In configurations without basis pruning, all frequency candi- dates up to the maximum frequency were allowed to contribute to the solution. Two variations were tested: with basis regularization ("reg + no prune") and without regularization ("uo reg + no prune"). Both configurations resulted worse relative le errors than the pruned counterparts, although the regularized version performed slightly better than the unregularized one.
Fourier-PINN Without Regularization. To further evaluate the impact of regularization, tested configurations without the regularization term. Two setups were compared: with basis pruning ("no reg + prune") and without pruming ('no reg + no prune"). The configuration without regularization but with basis pruning ('no reg + prune") achieved better relative le errors than the setup using regularization but no basis pruning ("reg + no prune"). This result suggests that basis pruning has slightly more significant influence on solution quality than regularization.
6.6 1D Non-Steady-Stan Allen-Cahn Problem
The time-dependent Allen-Cahn equation well-establisedd benchmark for testing the limitations of PINN models, particularly due to its stiffness, sharp transitions, and nonlinear dynamics (Wight & Zhao 2020: Wang et al. 2024bb For simplicity, focus a one-dimensional case with periodic boundary conditions
22

---

Published in Transactions on Machine Learning Research (01/2025)
over t E [0, and [-1,1]:
ut 0.0001 + 5u 5u 0,
u(0, x) = x² cOS(
u(t,- -1) u(t, 1), uu(t,-11 (t, 1).
This problem is particularly challenging because the Allen-Cahn PDE introduces complex, multi-scale dy- namics driven by a combination of stiff reaction terms and spatial diffusion. Unlike problems with harmonic solutions, the nonlinear terms generate sharp transitions and localized structures in both space and time, testing the model's adaptability to varying frequency components and its robustness to sharp gradients. Table in the Appendix outlines the hyperparameter and experimental settings used in the example. In this experiment, we randomly sampled 8192 collocation points within the domain at each iteration. Figure [4 compares the convergence behavior of Fourier PINNs with baseline models (standard PINN and RFF-PINN).
10a
10a
Relative Error
10-3
10-1
Relativa
2
10-2
10-3
10
20000
40000 Iterations
60000
80000
100000
10a
10*
10a
10a
Training Time (s)
RFF-PINN
Fourier-PINN
Figure 14: Convergence behavior of different models on the 1D non-steady-statee Allen-Cahm equation. Fourier PINNs achieve superior accuracy and convergence, effectively copturing sharp transitions and localized structures, while baseline methods falter.
Results. As shown in Figure 14 the Fourier PINN significantly outperforms both the standard PINN and RFF-PINN. The relative & error decreases steadily for Fourier PINNs, reaching values below 10-3 after sufficient training iterations, while the baseline models plateau higher error levels. Specifically, the standard PINN struggles with stiffness, failing reduce the error below 10 even after extended training. The RFF-PINN shows some improvement over the standard PINN but is highly sensitive to scale settings, resulting in slower convergence and larger errors compared to Fourier PINNs. Fourier PINNs leverage the augmented Fourier layer structure to efficiently represent both smooth and sharp features, as evidenced by their rapid convergence and low final error. This experiment underscores the potential of Fouricr PINNs to handle challenging nonlinear PDEs with sharp transitions, where conventional PINN architectures struggle.
6.7 Discussion
The 1D and 2D problems results reveal that standard PINN, W-PINN, and A-PINN consistently fail to achieve reasonable solutions, with relative fg errors remaining large (~~ 1.0) across all cases. This indicates that these methods struggle to capture high-frequency signals effectively. While RFF-PINNs achieve lower relative errors under specific scale configurations, they exhibit significant sensitivity to the choice of the number and range of scales. For most configurations (60-70% of the tested settings listed in Table 2 RFF- PINNs failed with large solution errors. Notably, even in the single-scale solution case (Figure 15ah, successful configurations often involved multi-scale settings (e.g., (1, 20, 194)), whereas all single-scale settings failed. These findings highlight the lack of clear relationship between scale configurations and solution accuracy, posing a significant challenge for RFF-PINNs. In contrast, Fourier PINNs consistently delivered accurate results, with relative & errors ranging from 10-3 to 10-4 across almost all test cases. A key advanlage sufficiently
of Fourier PINNs is their robustness to the choice of the frequency range K. Provided large, the method reliably selects the target frequencies, achieving comparable solution accuracy across cases. is important to note that Fourier PINNs do not employ advanced loss re-ieighting schemes (c.g.. NTK re-ghinhting used in RFF-PINNs mini-max updates) in this stu,,, as the focus evaluating
23

---

Published in Transactions on Machine Learning Research (01/2025)
the core algorithm. However, such enhancements could be scamlessly integrated into the Fourier PINN framework to further improve accuracy. Additionally, Fourier PINNs outperform baseline methods in higher- dimensional problems, demonstrating scalability and the ability to handle high-frequency and multi-scale solutions. However, their performance on (some) nonlinear, higher-dimensional PDEs, such as the 2D Allen- Cahn example Equation 4T highlights areas for future work. Specifically, improving training elficiency and refining the network architecture better address the challenges posed by nonlinear dynamics and stiffness could further enhance the applicability of Fourier PINNS.
We finally visualize the solution of each method in solving 2D Poisson equation (see Figure 17b to visualize the point-wisc errors for each method within the two-dimensional domain. The ground-truth solution is given in Figure 20 and the solution obtained by each method is shown in Figure 21 While the spectral method also obtains good results, the point-wise error biases much on the regions close to the boundary, and the relative l2 error is two orders of magnitude larger than Fourier PINN's. RFF-PINN (with the best number and scale choice) can roughly capture the solution structure, but the accuracy is far worse than that of Fourier PINNs and the spectral method. For example, in Figure 17b RFF-PINN nearly failed with every choice of the number and scale set. We can see that Fourier PINNs best recover the original solution.
7 Future Work: Improving Scalability with Tensor Decomposition Techniques
One limitation of our current approach is its scalability in higher-dimensional settings. The computational cost grows exponentially with the problem's dimensionality, making it infeasible for very high-dimensional cases. Future work could explore tensor decomposition techniques, such as Tensor- Tensor-Train (TT) decomposition and Tucker decomposition, to reduce the computational complexity and memory requirements.
Specifically, in high-dimensional spaces, the solution (2,,, ..,...,, (.)) can be represented tensor-product over basis functions o(xz), with the weights forming high-dimensional tensor UI
11
u(11, 22. ................................
da...... Di (22) (22) (x.) Σ
(44)
Without decomposition, storing and computing with U requires II n2 parameters, which is impractical in high dimensions.
One promising approach to mitigate this through Tucker docomposition, where U is docomposed into a core tensor C and set of factor matrices {Az}:
12
rd Ch.samniiaa (1(1,,1) Az(i2, ... Ad(ia:ja),
Un dg
(45)
11=
11=1
where each factor matrix Ay E Rayxrs maps the original tensor to lower-dimensional representation. This decomposition reduces the number of parameters to Ej=1 nyr, + I[ Ty. allowing scalability as it grows linearly with under fixed ranks ry- Another promising approach is Tensor-Train (TT) decomposition, where U is expressed as a chain of lower-order tensors, known as TT-cores:
fd-1
a) a2( ... Ga(aa-1rda), Od-1=1
Σ
(46)
where G2 E R G2 E Rra-1xma, and intermediate cores G, E Rt- for j == 2...., .d 1. The TT-ranks control the complexity of the representation, enabling a linear scaling of parameters with for fixed ranks.
8 Conclusion
We have presented Fourier PINNs, a novel extension to capture high-frequency and multi-scale solution in- formation. Our adaptive basis learning and selection algorithm can automatically identify target frequencies
24

---

Published in Transactions on Machine Learning Research (01/2025)
and truncate useless ones without tuning the number and scales. We assume the spacing (gramularity) of the candidate frequencies is fine-grained enough, which can be overly optimistic. In the future, we plan to develop adaptive spacing approaches to capture this granularity, Applying tensor decomposition techniques, such as Tucker and Tensor-Train, to the weight tensor U over basis functions could provide a viable path for extending our approach to higher dimensions. By reducing the elfective dimensionality of the param- eter space, these methods offer a promising direction overcome scalability challenges and maintain the advantages of meshless approaches in high-dimensional spaces.
Acknowledgments
We acknowledge the use of large language models (LLMs) for manuscript editing. This work has been supported by MURI AFOSR grant FA9550-20-1-0355, NSF CAREER Award IIS-2046295, and NSF OAC- 2311685 and the Army Research Office (ARO).
References
J. E. Akin.
Finite Flements for Analysis and Design. Academic Press, 1994. ISBN 0120476541.
Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In Proceedings of the International Conference on Machine Learning (ICML.), pp. 685 694. PMLR, 2020.
Jens Berg and Kaj Nyström. A unified deep artificial neural network approach to partial differential equations in complex geometries. Neurocomputing, 317:28 41, 2018. doi: 10.1016/j new.om....mmm0000 w0w...m........... 0..066
Shengze Cai, Zhiping Mao, Zhichong Wang, Minglang Yin, and George Em Karniadakis. Physics- informedd neural networks (PINNs) for fluid mechanics: review. Acla Mechanica Sinvica, 37(12)11227-1733 1738, 2021.
doi:
Francesco Calabrò, Gianluca Fabiani, and Constantinos Siettos. Extreme learning machine collocation for the numerical solution of elliptic pdes with sharp gradients. Computer Methods in Applied Mechanics and Engineering, 387, 2021. doi: 10.1016/j.cmm w0tt:/.c.........mm...mm..
Yuyao Chen, Lu Lu, George Em Karniadakis, and Luca Dal Negro. Physics-informed neural networks for inverse problems in nano-optics and metamaterials. Optics Express, 1818)1116,8-1163,, 2020.
Eric C. Cyr, Mamikon A. Gulian, Ravi G. Patel, Mauro Perego, and Nathaniel A. Trask. Robust training and initialization of deep neural networks: An adaptive basis viewpoint. In Proceedings of The First Mathemalical and Scientific Machine Learning Conference, pp. 512-536. PMLR, August 2020. URL
https://proceedding mlr. www.ppees/vvee vooss//v00//yr20... html ISSN: 2640-3498.
Shifei Ding, Han Zhao, Yanan Zhang, Xinzheng Xu, and Ru Nic. Extreme learning machine: algorithm, theory and applications. Artificial Intelligence Review, 44:103-115, 2013. doi: 10.1007/s10462-11 9-99055-..
Gianluca Fabiani, Francesco Calabrò, Lucia Russo, and Constantinos Siettos. Numerical solution and bi- furcation analysis of nonlinear partial differential equations with extreme learning machines. Journal of Scientific Computing, 89, 2021. doi: 10.1007/s109155-21-0111
Gianluca Fabiani, Ioannis G. Kevrekidis, Constantinos Siettos, and Athanasios N. Yannacopoulos. Ran- donets: Shallow networks with random projections for learning linear and nonlinear operators. Journal of Computational Physics, 520:113433, 2025. doi:
Zhiwei Fang and Justin Zhan. Deep physical informed neural networks for metamaterial design. 1EEE Access, 8:24506 24513, 2019.
Nicholas Geneva and Nicholas Zabaras. Modeling the dynamics of pde systems with physics-constrained deep ruto-segressive networks. Journal of Computational Physics, 403:109055 2020.
25

---

Published in Transactions on Machine Learning Research (01/2025)
Ameya D Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Compulational Physics, 2020.
Xisowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. Nsfnets (navier-stokes flow nets): Physies- informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426:109951, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3nd International Conference for Learning Representations (ICLR), 2015. URL
org/ org/abs/ abs/
1412. 6980
Georgios Kissas, Yibo Yang, Fileen Hwuang, Walter R Witschey, John A Detre, and Paris Perdikaris. Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4d flow mri dała using physics-informed neural notworks. Compuler Melhods in Applied Mechanics and Engineering, 358:112623, 2020. doi: 200.1//..caa.219.11...
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34:26548 07726558-6660,, 2021.
Pola Lydia Lagari, Lefteri H Tsoukalas, Salar Safarkhani, and Isaac E Lagaris. Systematic construction of neural forms for solving partial differential equations inside rectangular domains, subject to initial, boundary and interface conditions. International Journal on Artificial Intelligence Tools, 29(55)0005009, 2020.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Newral Networks, 9(5):987 1000, 1998.
Isaac E Lagaris, Aristidis C Likas, and Dimitris G Papageorgiou. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, 11(5):101111049, 2000.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stu- art, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021. URL
https:///ppereet http://ww.....trrttaeeeee/ooaaaa
Dehao Liu and Yan Wang. Multi-fidelity physics-cconstrained neural network and its application in materials modeling. Journal of Mechanical Design, 141(1)):121003, 2019. doi: 10.1115/1.4043155.
Dehao Liu and Yan Wang. A dual-dimer method for training physics-constrained neural networks with minimax architecture. Neural Networks, 136:112-125, 2021.
doi: n..0016/..euumet20221...1
Dong C. Liu and Jorge Nocedal. On the limited memory bigs method for large scale optimization. Mathe-
matical Programming, 45(1 3):503 528, 1989. doi:
Ning Liu, Siavash Jafarzadeh,, and Yue Yu. Domain agnostic fourier neural operators. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), volume 36, pp. 47438-4777 4444-00 Curran Associates, Inc., 2023.
Lu Lu, Raphael Pestourie, Wenjie Yao, Zhicheng Wang, Francesc Verdugo, and Steven G Johnson. Physics- informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing,
B316):B1105-B11333 2021. doi: 10.1137 10.1137/20M13600 2060066660
Liyao Lyu, Keke Wu, Rui Du, and Jingrun Chen. Enforcing exact boundary and initial conditions in the deep mixed residual method. CSIAM Transactions on Applied Mathematics, 2(4):748-775, 2021. doi:
10.420%/ciammmm555--222..
26

---

Published in Transactions on Machine Learning Research (01/2025)
Zhiping Mao, Ameya D. Jagtap, and George Em Karniadakis. Physics-informed neural networks for high- speed flows. Compuler Methods in Applied Mechanics and Engineering, 360:1 12789, 2020. doi: 10.1016/j. cma.2019.112789.
Levi D. McClenny and Ulisses Braga-Neto. Self-adaptive physics- infirmeed neural networks using a soft allention mechanism. Journal of Compulational Physics, 474:111722, 2024. doi: 10.1016/1..ep..022.
111722.
Kevin Stanley McFall and James Robert Mahan. Artificial neural network method for solution of boundary value problems with exact satisfaction of arbitrary boundary conditions. IEEE Transactions on Neural Networks, 20(8):1221-1233, 2009. doi: 2LL099TNN.20090.00
Clare D. McGillem and George R. Cooper. Continuous and Discrete Signal and System Analysis. Harcourt School, 3rd edition, 1991. ISBN 9780030747095.
Siddhartha Mishra and Roberto Molinaro. Physics informed neural networks for simulating radiative transfer. Journal of Quantitative Spectroscopy and Rodiative Transfer, 270:107705, 2021. doi: 10.1016/)..gsrr.2222 107705.
Rambod Mojgani, Maciej Balajewicz, and Pedram Hassanzadeh. Kolmogorov n n-width and lagrangian physics-informed neural networks: A causality-conforming manifold for convection-dominated pdes. Com- puter Methods in Applied Mechanics and Engineering, 404:115810, 2023. doi: (0000///.cma.222..111..
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P. Vetrov. Tensorizing neural networks. In Advances in Newral Information Processing Systems, volume 28, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Jaideep Pathak, Zhixin Lu, Brian Hunt, Michelle Girvan, and Edward Ott. Reservoir computing approaches for solving nonlinear pdes. Physical Review Letters, 120(2):024102, 2018. doi: 10.1103 hats//yyseeeett.1222 024102.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseurch, pp. 1301-5310 PMLR, 2019.
Ali Rahimi and Benjamin Recht. On randomized algorithms and random projection networks for function approximation. Advances in Neural Information Processing Systems, 21:1232 1240, 2008.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data- driven solutions of nonlinear partial differential equations. arXiv preprint orXiv:1711.1066 2017. URL
wttp://wwwrr/aaa//1.....o. 1711 10561
Maziar Raissi, Paris Perdikaris, and George E. Karniadakis, Physics-iffrrmed informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Compulational Physics, 378:686 707, 2019. doi: 10.1016/j.jep.20 10.045..
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and 2020. doi:
pressure fields from flow visualizations. Science,
Junuthula Narasimha Reddy. Introduction to the Finite Element Method. McGraw-Hill Education, 4th edition, 2019. ISBN 9781259861901
Vladimir L. Rvachev and Tatyana Sheiko. R-functions in boundary value problems in mechanics. Applied Mechanics Reviews, 48 144: 1995. doi: 00.1115/1.305110
27

---

Published in Transactions on Machine Learning Research (01/2025)
Francisco Sahli Costabal, Yibo Yang, Paris Perdikaris, Daniel E Hurtado, and Ellen Kuhl. Physics-informed neural networks for cardiac activation mapping. Fromliers in Physics, 8:42, 2020. doi: 10.3369/Iphyy2020. 00042.
Vadim Shapiro and Igor Tsukanov. Meshfree simulation of deforming domains. Computer- Design, 31 (7):459-41) 471, 1999. doi: (0.1006/00000444559900000000
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: deep learning algorithm for solving partial differ- ential equations. Journal of Computational Physics, 375:1339 1364, 2018. doi: po.1016/j.jcp.21...0. 08.029.
Sukumar and Ankit Srivastava. Exact imposition of boundary conditions with distance functions in physics-informen deep neural networks. Computer Methods in Applied Mechanics and Engineering. 389: 114333, 2022. doi: (ttp://.cma..22.111333...
Luning Sun, Han Gao, Shaowu Pan, and Jian-Xun Wang. Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data. Computer Methods in Applied Mechanics and Engineering, 361:112732. 2020. doi:
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7555 2020.
Igor Tsukanov and Sudhir R. Posireddy. Hybrid method of engineering analysis: Combining meshfree method with distance fields and collocation technique. Journal of Computing and Information Science in Engi- neering, 11()), 2011.
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In 9th International Conference Learning Representations (ICLR), 2021a. URL
forum?id=0966 Is nPA.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologies in physics- informed neural networks. SIAM Journal on Scientific Computing, A3355: A3081, 2021b. doi:
10.1137/20M13180433
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informet neural networks. Computer Methods in Applied Mechanics and Engineering, 384:113938, 2021c. doi: 10.1016/).cma.2021. 1139988..
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail train: neural tangent kernel perspective. Journal of Computational Physics, 449:110768, 2022. doi: 20.10016//.cp.2222.1111
Sifan Wang, Bowen Li, Yuhan Chen, and Paris Perdikaris. Piratenets: Physics-informed deep learning with residual adaptive networks. arXio preprint arXiv:2402.00920 00326, 2024a. URL https://arxiv org/ /bs//
2402.00326
Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality for training physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 421:116813, 2024b. doi:
00.1016/.ccma.203.111
Colby L Wight and Jia Zhao. Solving allen-cahn and cahn-hilliard equations using the adaptive physics- informed neural networks. arXiv preprint 2rXiv:2007.00 04542, 2020. URL https://arxiv.. http:://axxvvooo//// abbb/207. 04542
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. Communications in Computational Physics, 28(5):1746 1767, 2020. doi: OA-
28

---

Published in Transactions on Machine Learning Research (01/2025)
Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Karniadakis. Gradient-enhaneed physics- inysice-informed neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering, 393:114823, 2022. ISSN 0045-7825. doi:
Le Zhang and Ponnuthurai Suganthan. A comprehensive evaluation of random vector functional link net- works. Information Sciences, 367:316-329, 2016. doi:
Linan Zhang and Hayden Schaeffer. On the convergence of the sindy algorithm. Multiscale Modeling & Simulation, 17(3):948-972 972, 2019.
Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, and Paris Perdikaris. Plysics-constrainee deep learning for high-dimensional surrogate modeling and uncertainly quantification without labeled data. Journal of Computational Physics, 394:56-63 81, 2019.
9 Additional Experimental Details and Results
Table 3: paremeterr configurations for 1D Poisson and 1D Sleady-Stade State Allen-Cahn frequency sweep experi- ments.
Parameter
Value
Architecture
Number of layers
2,4
Number of channels
100
Activation
Tanh
Learning rate schedule
Initial Adam learning rate
10 3
Initial LBFGS learning rate
10
Decay rate
0.9
Decay steps
every 1000
Decay type
exponential
Training
Training steps
100K
Collocation points
10K
Testing points
20K
29

---

Published in Transactions on Machine Learning Research (01/2025)
Table 4: Hyper-porameter configurations for IT) Poisson analysis experiments.
Parameter
Value
Architecture
Number of layers
4
Number of channels
100
Activation
Tanh
Learning rate schedule
Initial Adamn learning rate
10-3
Initial LBFGS learning rate
10-2
Decay rate
0.9
Decay steps
every 1000
Decay lype
exponential
Training
Training steps
1 x 105
Collocation points
10K
Table 5: Hyper-porameter configurations for Poisson and ID Steady-State Allen- Cahn experiments.
Parameter
Value
Architecture
Number of layers
2
Number of channels
100
Activation
Tanh
Learning rate schedule
Initial Adam learning rate
10-3
Initial LBFGS learning rate
10-2
Decay rate
0.9
Decay steps
every 1000 exponential
Decay type
Training
Training steps
4 x 10
Collocation points
10K
Testing points
20K
30

---

Published in Transactions on Machine Learning Research (01/2025)
Table 6: Hyper-porameter configurations for 1D Wave experiments.
Parameter
Value
Architecture
Number of layers
3
Number of channels
128
Activation
Tanh
Learning rate schedule
Initial Adam learning rate
10
Initial LBFGS learning rate
10-2
Decay rate
0.9
Decay steps
every 1000
Decay type
exponential
Training
Training steps
4 104
Collocation points
5K
Initial condition points
256
Boundary points
200
Testing points
100K
Table 7: Hyper-perameter configurations for 2D Poisson and 2D Steady-State Allen- Cahn experiments.
Parameter
Value
Architecture
Number of layers
3
Number of channels
100
Activation
Tanh
Learning rate schedule
Initial Adam learning rate
10-3
Initial LBFGS learning rate
10-2
Decay rate
0.9
Decay steps
every 1000 exponential
Decay type
Training
Training steps
4 x 10
Collocation points
12K
Boundary points
400
Testing points
100K
31

---

Published in Transactions on Machine Learning Research (01/2025)
Table 8: Hyper-parumeter configurations for 1D Shon-Steady-Slate) Allen- Culn experiments.
Parameter
Value
Architecture
Number of layers
4
Number of channels
128
Activation
Tanh
Learning rate schedule
Initial Adam learning rate
10-3
Initial LBFGS learning rate
10-2
Decay rate
0.9
Decay steps
every 1000
Decay lype
exponential
Training
Training steps
105
Batchsize
8192
Boundary points
200
Initial condition points
256
Testing points
100K
Weighting
Weighting scheme
Gradient Norm
Wang et al.
2022
Causal training
True, Wang et al. 2024b
Causal tolerance
1.0
Number of chunks
32
32

---

Published in Transactions on Machine Learning Research (01/2025)
A.POUN PIP- (1 scales)
BPF-PINN RPP-PIN ocales)
REP-PINN scales) Fourler PINN
PINN
W-PINN
10-1
Relative F, Error
10
10
Spectraa
Strong MC Pres
A-PIUN
ure art--rtee L1)
oer eay por-reou (to)
(20)
synernt (14))
arr-rrute L3. so)
(1n70)
are- ver--rion ser-- LsS 4a)
20)
(s5. 71
ery Pre- coo, 1000
e arr- Prrn ft. so 109
are pwre 120, su 1007
25, 1o4)
ner Press LSN 112. 12v)
are mase 47, 388)
aav. wore an, 5o, su80
aer- Ut 20 1oa, 187, 189)
are- 36, e7, 79, 1200
RoV ROV-PURE ceo, 68, es, 104, 129)
So us. 100y
Paas Pao (K- 2200
Fourter Puuh (6x250))
(K-1140) D807
mourter PaRN (in i01)
nore poees
Fourier Four
(a) 1D Poisson with m(x) = sin sin((100))) (100r)
Spedtal
Strong BC PINN
scales)
ASP-PINN scalea) PINN
RPP-PINN (1 scales)
RPP -INN
20
Relative T. EErrur
10-1
30-1
10
10
Spectral
Strong Po1
Part-
Arr-PING (oo)
erF- Prr--PiN (80)
Repoane (60)
ers ens-nwee D 20
aree pore-ttuit (LB, Tu
arreeun us, on
trr-Pio on,, 100)
aepeaa so. 1891 1801
aee pees con, 100
arr- Post (44, 47, 1059
112, 1181
are- Pre- (1 56. 100)
ner- rwn (1. 10a, 18T, 1891
eYF- muw (6, 26, 11-77.1 130
s0, 100)
45, 1ok, 180
(K0)50)
Park (K-2000
Fouriie POSS (00)
PHANN (K-2260)
art-0100
ere vee-muu 20,
are wwwe 20, 40,
Four Four
RP preeam
(b) ID Poisson with u(x) = sin(x) + 0.1 sin()0z) + 0.05 cos (100x)
Spectral
W-RINN
FINM Scides)
RIF-SIRN scales)
Fourler PINN [Altemating opt.) opt. only)
Strong PINN
10-1
10
16-4
Speccroe
pront
are Freziun pose (10)
(1)
arn (ea)
pren (rem)
art. pot-ront CL tot
arr-etni cny
ar Prr-arms (10), too)
Eyy- cosa L. yas in Pesss sia, stay trr
yu
arr: yona (66, 57 seay
aam pow 58, ets too
ot, 1a99
tor 23, 118, 197 net 67. TP 190
su, yeu
199
43, iin. (iny
yous pous (Ero 1000
pautle paus di- LSS. Adeer orey!
Pountes sen crep!
(2--200))
risn 2000 Adenn oney)
Foutte wram (ec- and (noy)
only)
art- CL.
monn
Fourter posnt
(c) ID Poisson with u(x)= = (6x( cos(
Figure 15: Combined figures for different ID Poisson configurations.
33

---

Published in Transactions on Machine Learning Research (01/2025)
PINN
W-PINN
PIP- (1 scales)
RPP-PINN scales)
Fourler PINN
10.1
Ertor
10
Spertrad
Strong MC Prtse
aFF-WEEN CJ, so)
APII
aPP-DVEN 200
R R77-aiI LLS, 713 RT
49)
aree poat 56 109
ars L2b, 80, 1800
Pree 122, 119)
an aae-nar 47, 1s5)
RPV ane-enn CL 20 104, 397, 100)
arF PIN 07, 78, 180)
es; 18s
REP USG, 48, RX. 188, 129)
Fourreer PARS (Buck00)
RPF- 20,
ner- pats ab, so.
(a) 1D Allen-Cahn equation with true solution u(2) (10(0007)
Strong BC PINN
ecales)
AEF-PINK scalea) PINN
PiNN
RPP-PINN scales)
RPP PPP-PIN
r.
10
Strong Part
Paat
RIV. ROV. RFF-nIN (A s0)
APaIN
are-num 110, 73)
20)
are-um DS, en)
nr (R,, 100)
Arr- arpoan 20, IMl)
arTe Por (20, 50, 100)
so. 1807
RTV- 28 100)
PINER (44, 47, 10n
L1D, 118)
RY 30, 5o, es. 100)
Iur PIN 20 184, 193, 199)
RFF- AFF-PRI (S0, 65, #3, 106, 189
Pouriaa Fourier Park (K-2
1300
PINts (4--200)
Fourier PINM (K-2200)
Fourier PINK (K--200)
RFF PVN- 36.
(b) ID Allen-Cahn with true multi-scale solution u(x) sin(x) +
0.1 sin (20x)) + 0.05 cos((00z))
PINN W-PihN
acales)
PINM
RFF-PPM (1 scales)
RFF-PINN 18 scales)
Fourier PINN
Reletive
10-4
Specttral
ver pre-prin On to) aspe
201
are w pre-pes 112, 119) erp-
are- pan so, 1891
(io, 200)
1041
RF (64, usdi
no 20 so, 1801
20 10a, 197, 19.7%.b0 120)
mn (e e5, 194, (r9) Pourler wuN (602300)
199)
pir-
eer: W1n- 20,
wr--II
(c) 1D Allen-Cahn equation with true solution u(z) = sin (62)) cos (100x)
Figure 16: Combined figures for different 1D Allen-Cahn configurations.
34

---

Published in Transactions on Machine Learning Research (01/2025)
PINN
W-PIN
RFP-PINN (1 scales)
RPF SPP-PINN scalisi
Fourter PINN
Fy Error
Spectral
Stromy BC Parn
ArraOmn so)
Ron-mun 20)
APP- 20, 164)
ArP pr--pmsuu 56 109)
www. 113)
RTF- PwNS 31, ab, So. 160)
100)
AP PIN- (A. 20 204, 397, 190)
RTP 36, 78, 184)
45, 100)
REP 158, 45, a2, 108, 1251
Fourrie PaNt No-200
ErK.PIR (50.
ere mart 58,
R
(a) 2D Poisson with u(z, v) sin(100z) sin (100g)
Spectral
Strong BC PINN
A-PIMN
12 scales) RFF
RPP-PINM scales) PINN
scales)
Rolative 12
20-1
Spectial
atnung SC Pote
APRiN
are-Penn (2)
ere urr-maan are-pun wre-ertn (50)
184)
arR-OI 50)
(100)
arF 12. 30)
NF RePernse (19, 73)
RPF arr-ans (99 69)
Rre Rrr-owe 0s, 100)
ALP- Pan 134)
Por-- so, 189)
RFF APP-PIUN DH6, Zoo)
NIV por- 138, 112, 2.0)
Fourier PRiN
(b) 2D Poisson with u(z,y) sin(6x)) cos((0z)) + sin (6y) (20y)
Figure 17: Combined figures for different 2D Poisson configurations.
35

---

Published in Transactions on Machine Learning Research (01/2025)
PINN
W-PINN
AFP-PINH scaleal
RFF-PINK scallss
Fourter PINN
Relative Error
107
Spertrrr
Seropp Paw
ArF-nwe TJ. soy
A-Pan
aur preeeae srs-Pam 7X1
20
NT erF- pre- Ionj
D30, d9)
nor PaNM so 1n0
nar w-re- no 1t0y
arr wrees caos 1007
nre weeeerene 122 (ett
RPP- prase 144, 47. 185)
arr 28 a8 1009
aEP-P s0 8s, zony
nut RFF 36 e7 1809
New NEV-TIN (R0, 1se 1399
197, 1997
Fourler PENV
Figure 18: Allen-Cahn with u(x, (sin(æ) 0.1 s2n()0z) + cos((00x)) (sin(y) + 0.1 sin()0g) cos(100y))
Spectral
W-PINM
REF-DINM scale))
RFF-PINN (3 scaloe)
Fourter PINN
10-1
30-4
Reletive
Specttad
Pan
Womne
A.PWN
APV seeeetee (30)
(oo)
(200)
71 10
we are www.ewee us 721
arn-ouu prne so awn
net ear prraroo cSa 112 1107
20 2oo
(10, so, iooy
ane 144 so, iov
nare pere it. 104, 1399
100)
an va larr (aax
Arr-m1n 20 so 199
are presss (so, Fouster
nee arr-ntt
Figure 19: One-Way Way Wave equation with true solution u(x, t) = sin( cos(10rt) + sin(2mz) (20mt)
Exact u(x, y)
0.75
0.25
0.00
-0.25
-0.50
Figure 20: 2D Poisson equation with true solution u(x, v) sin(6ar) cos(20z)) sin(6y) cos(20g)
36

---

Published in Transactions on Machine Learning Research (01/202)
Absolute Error Be-04)
10-1
0.00
107
RFF-RINN
(G, Erior Se-01)
10
20
-025
10
20
A-NNN
Absolute Error (Lg Error 3c+00)
0.75
0.59
-150
10-
W-RNNN
Absolute Error (Lg Error 3c+00L)
2017
0.50
10-
10-1
-350
20-4
-0.5
PINN
Absolute Errar II Error 2e-4
0.25
0.00
20-1
-575
20-9
Spectral
307
10-1
0.00
20
20-
Figure 21: Predicted solutions (left) and absolute errors (right) of each method on the 2D Poisson equation with solution u(z, = sin (6z) cos(20z) sin(6y) cos(20y)
37


Reviews:
[{"review_id": "Iwmfp7HnFz", "reviewer": "Reviewer_WjzW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper provides a strong background and an accessible introduction to PINNs, useful for readers outside of the field.\n- The Fourier analysis conducted for the BC-PINNs is well-executed and offers a solid theoretical foundation for the work. This analysis motivates the inclusion of Fourier bases to address the \"spectrum bias\" problem effectively.\n- The concept of Fourier PINNs is both simple and highly effective, demonstrating significant and stable performance across a variety of benchmark PDE problems. The adaptive selection mechanism for the Fourier bases is a practical innovation  enhances performance.\n\n\nWeaknesses:\n- The primary limitation is in the evaluation methodology. The authors test the Fourier PINNs on a limited set of problems (three one-dimensional and two two-dimensional problems). All these problems seem tailored to the strengths of Fourier PINNs, as their solutions largely consist of harmonic functions. While these functions are typical for benchmarks, and gives the authors the ability to control the prevalence of high frequencies, this choice may give a somewhat one-sided view of the method's performance.\n- The evaluation lacks sufficient explanation regarding the purpose of the chosen numerical problems. The authors briefly explain this at the beginning of the experiment section, but this is not detailed enough to give clear motivation for the different evaluations. The authors should elaborate on what aspects of network performance are being tested with each problem and how the problems are designed to introduce high frequencies. This would add depth to the evaluation section.\n- There is no comparison with the previously introduced BC-PINN with adaptable distance functions, which could have demonstrated the benefits of including Fourier components more clearly.\n- The discussion is also underdeveloped. The authors provide a large number of figures, but the accompanying analysis is insufficient, particularly with regard to the different families of equations used in the experiments.\n- The reference section is poorly formatted, with inconsistent capitalization and incorrect references for papers already published in journals but listed with Arxiv citations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript introduces some enhancements to existing Physics-Informed Neural Network (PINN) approaches for solving partial differential equations (PDEs), particularly focusing on learning high-frequency components. Standard PINNs struggle with these due to \"spectrum bias,\" where the neural network efficiently learns low-frequency information but fails to capture high-frequency components. The authors first examine strong boundary condition (BC) PINNs, where they introduce a flexible distance function to satisfy the boundary conditions, offering improvement in accuracy compared to the standard approach. By performing a Fourier analysis on the results, the authors conclude that such strong boundary conditions help with learning higher frequencies, but only to a certain amount, leading to the motivation and introduction of the Fourier PINNs. Fourier PINNs augment standard PINNs with pre-specified, dense Fourier bases, significantly improving their ability to capture high-frequency components across different boundary conditions and problem domains. They also include an adaptive basis selection algorithm during training, that automatically prunes the basis functions/frequencies that are not relevant to the problem. The authors demonstrate this with a series of numerical experiments with different 1D and 2D PDEs, and compare to several competing PINN and classical approaches. The results show that the Fourier-PINNs consistently outperform other methods, especially when faced with problems where high frequencies are relevant.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested changes:\n- Include some additional PDE problems, particularly in the 2d case. The authors should choose one or two benchmark problems that do not have an inherently harmonic structure but still contain high frequencies. (very important)\n- Include more detailed explanations on what each of the example PDE problems is aiming to evaluate, and a discussion specific to each of the problems selected. My suggestion would be to separate each of the scenarios into its own subsection, and discuss the purpose of the specific numerical problem as well as present and discuss the results in the same section. This can be finally followed by a general discussion and conclusion as the authors have done. (very important)\n- Include benchmarks for the introduced BC-PINN with adaptable distance function as introduced in section 2. (important)\n- Another interesting variant to evaluate would be the Fourier-PINN with no regularization or basis pruning. This would clearly demonstrate the effects of the basis selection method introduced by the authors (important)\n- Fix references to include proper capitalization, and refer to the actual publication venue of Arxiv papers which have already been published after a peer review. (important)\n- Ideally, the code should be included for review as well, at least for the BC and Fourier-PINN, with some examples to reproduce the results in the paper. This is crucial to foster transparency and reproducibility in our community (very important).", "broader_impact_concerns": "No broader impact concerns to report.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "chGvPhFeNu", "reviewer": "Reviewer_mVk8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The new adaptive distance function $\\phi_{\\text{exp}}$ for boundary condition PINNs is well motivated and well analyzed\n- The exposition for the Fourier-PINN is clear.\n- The comparison of the Fourier-PINN to other state-of-the-art methods is thorough and convincing for the class of PDEs that the authors care about (high-frequence and multi-frequency).\n\nWeaknesses\n- I found some of the notation hard to follow in the Fourier analysis section. Perhaps if equation (15) was in the integral form, or there was an additional equation in the integral form that you substituted the various distance functions into.\n- In the Fourier PINN, the Fourier bases grow exponentially in the problem dimension. The authors fully discuss this limitation and leave it to future work.\n- The ground truth equations for the numerical experiments consist only of sums and products of sines and cosines. However, the authors state that high-frequency and multi-frequency solutions are the focus of their method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors explore using physics informed neural networks (PINNs) for PDEs and how they handle balancing accuracy at the boundary vs accuracy inside the domain. The authors present and analyze a new method of Strong Boundary Condition PINNs that do a better job in dealing with higher frequency target problems. The authors then create a new model, the Fourier-PINN, to handle PDEs with high frequencies and benchmark it against a variety of state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major\n- How does $\\phi_{\\text{exp}}(x)$ work when not in the 1-D setting?\n- Figure 2: What value of $\\alpha$ do you use for $\\phi_{\\text{exp}}$? How do you pick your value of $\\alpha$?\n- What is the goal of the $L_2$ regularization term in your loss function? It is true that $L_2$ regularization will penalize large coefficients. However, this will not promote sparsity, in fact it is likely to do just the opposite. If you want sparsity, you probably want to be using $L_1$ regularization. If you want sparsity in the Fourier basis coefficients, why are you also applying the regularization to the coefficients of the last layer of the Neural Network? Is this just required because otherwise the joint optimization will favor the neural network over the Fourier bases?\n- Some of the choices of the optimization algorithm seem strange to me. The weirdest part in my mind is why the authors are alternating between optimizing all parameters with ADAM, and then only optimizing the last layer coefficients with least squares and pruning the bases. The losses for both steps seem very similar, and typically a single joint optimization strategy will outperform an alternating one. So I am curious how Algorithm 1 compares to just running ADAM over all parameters (possibly with the $L_1$ regularization instead of $L_2$). \n- How does the performance of the method compare with bases pruning vs without?\n\nMinor\n- Top of page 3, \"frequencies uniformly sampled from an extensive pre-set range\" (also middle of page 9). This makes me think they are sampled randomly, but they are actually all the frequencies $1,\\ldots,K$. \n- Top of 10, \"we apply the cross-product\" I have not heard cross-product used in this way, I think it would be better to refer to it as the \"outer product\" or \"tensor product\" (as you do later in the sentence) to avoid confusion with the physics cross-product.\n- In equation (24), I think it would be more clear to use the notation your paper, rather than the notation of Cyr et al. (2020), i.e. use $w$ for the final layer coefficients.\n- Minor typos on page 11, \"essense\" -> \"essence\" and \"continues\" -> \"continuous\"\n- One idea for a future work: You could also allow the frequencies to be optimized, which may allow you to start with fewer.", "broader_impact_concerns": "- None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ig7PrVACCc", "reviewer": "Reviewer_HXi7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n•The paper provides a comparative study of different methods and implementations of PINNs, evaluating their performance against numerous baseline approaches. This adds clarity to the results.\n\n•The use of an adaptive distance function could be considered a novel idea, though its originality may be debatable. It bears similarity to concepts from numerical analysis and might represent a minor contribution.\n\n•Both the strong boundary condition PINNs and Fourier PINNs achieve competitive error rates in the experiments compared to vanilla PINNs. While offering small novel contributions, this work seems more like an academic exploration focused on high-frequency and multifrequency PDE solutions.\n\nWeaknesses: Given the two methods proposed, I will outline the weaknesses of each method separately, followed by those that apply to both.\n\nWeaknesses of Method 1 (Strong Boundary Conditions):\n\n•The adaptation of the basis is the primary contribution of this method, but this may be a minor and insufficient advancement.\n\n•The change in steepness, while potentially valuable, is symmetric and cannot independently adapt to different boundary conditions, which may require different steepness levels.\n\n•This approach is demonstrated only for 1D problems, and it remains unclear how to extend the proposed distance function to higher dimensions. While generating such functions in higher dimensions is inherently challenging, this limitation restricts the method's broader applicability.\n\n•The method's extension to more complex boundary conditions, such as Neumann, Robin, or nonlinear conditions, is not addressed. It is evident that a soft constraint or a more formal approach, such as using Lagrange multipliers, might be necessary.\n\n•The Fourier spectrum analysis, in section 4, is limited to a single solution, sin(15x). Real PDE solutions often have continuous spectra rather than discrete points in Fourier space, so the applicability of this analysis to more complex situations remains unclear.\n\n•The trial function incorporates the boundary function g (i.e., u=g+phi⋅u_n), but the analysis is restricted to the homogeneous case where g=0. It is unclear how this method would extend to nonzero g, especially in higher dimensions where the Fourier transform of g could complicate the analysis.\n\n•Figure 2 results lack clarity. The relative l2 error is not well-defined in the text and should be more explicitly described. Additionally, the x-axis values do not match the text's referenced values, though this is a minor issue. The relative l2 error appears larger than expected and is not compared to higher-order methods. Crucial details, such as the training method, learning rate, and number of epochs, are not sufficiently explained. Furthermore, the large number of collocation points (10K) raises concerns about computational cost. A comparison with classical techniques like finite difference (FD) or finite element methods (FEM) is missing.\n\n•The \"threshold\" frequency, at which the method's performance degrades, is not clearly presented in the figures. The method seems to deteriorate gradually but remains better than simple PINNs, yet further clarity is needed.\n\n•The method's error does not improve beyond 10^−5, especially for higher k, which accuracy is quite limited. It would be helpful to explore whether larger networks could achieve better convergence. The relatively high error rates for high k suggest that comparing this method to FD, FEM, or spectral methods under the same conditions would provide more meaningful insights. The lack of such comparisons leaves the impression that the method might not be efficient, even for simple 1D problems, let alone in higher dimensions or more complex geometries.\n\nWeaknesses of Method 2 (Fourier PINNs):\n\n•The formulation in Eq. 21, which superimposes two bases of functions, differs from the more common approach of using Fourier features as additional hidden layers to enhance solution features while retaining the spatial variable x. What is proposed seems more like a correction approach. The simultaneous optimization, despite alternating between the two bases, might not allow sufficient time for convergence. It is unclear which term will dominate, possibly depending on initialization. Given the universal approximation property of PINNs, both bases could potentially identify the solution, and the issue with the slow spectral decay of PINNs may still persist. A spectral analysis here would have been valuable.\n\n•The numerical examples mostly involve trigonometric functions, making the use of Fourier bases appear advantageous, even artificial. Testing the method on non-periodic functions, such as sigmoids or Gaussians, would provide a more balanced evaluation.\n\n•A detailed computational cost analysis should be included. How does the training cost increase when Fourier features are introduced? Are they faster to converge, and how computationally demanding is each epoch? Reporting execution times and the hardware used (CPU, GPU, or clusters) would provide valuable context.\n\n•The approach becomes computationally infeasible in higher dimensions, as stated in eq 22, which is a significant drawback. Meshless methods are expected to be advantageous in high dimensions, yet this method appears to struggle with scalability.\n\n•The introduction of L2 regularization in Eq. 23 is problematic. The claim that L2 regularization promotes sparsity is incorrect. Sparsity is typically achieved with L1 regularization. There is no evidence that L2 regularization yields sparse solutions, making the claim unfounded.\n\n•In Eq. 24, the focus is on linear operators, but the paper also addresses nonlinear PDEs, such as the Allen-Cahn equation. The treatment of nonlinearity in Eq. 28 seems inadequate, as it overlooks the nonlinear term, effectively solving a different PDE. This raises concerns about convergence, and the method for handling nonlinearity is not clearly described.\n\nWeaknesses of Both Methods:\n\n•The comparisons are limited to other PINN-based methods. Including a comparison with traditional methods like FD, FEM, or spectral methods would provide a more comprehensive evaluation. The claim that these methods are computationally efficient alternatives to traditional numerical methods is overstated and should be removed. While promising, they are not yet efficient in practice.\n\n•The comparison seems overly focused on RFF-PINNs, but this is not the original Random Fourier Features (RFF) approach as described by Rahimi and Recht (2007-2008). The current implementation only partially incorporates randomness, making the figures difficult to interpret. Reducing the number of RFF-PINN comparisons or presenting the results in a table would help.\n\n•The authors appear unaware of alternative PINN approaches based on random features, such as Random Vector Functional Link networks (RVFLN), Extreme learning machines (ELM), Random Projection neural networks (RPNN), Randomized Netwokrs, or Reservoir Computing, which have shown greater computational efficiency. Key references, such as the works of the group of (a) Constantinos Siettos, (b) Suchuan Dong, (c) Felix Dietrich (among others), especially for the case of PDEs, and recent advances like RandOnet for learning operators in few seconds with high accuracy, should be considered for a more thorough discussion. Indeed such method, should be the real counterpart PINN approaches where the author have to compare their method. This is the state-of-the-art. Not vanilla PINNs. At least in 1-2 dimension problems. \n\n•The description of the “spectral” method, i.e. named spectral in figures, in the paper lacks clarity. The algorithm used to compute the coefficients is not well-explained. For nonlinear problems, optimization may not be the best approach; Newton's method or straightforward linearization could be more appropriate. The use of Dirichlet boundary conditions with a Fourier basis is also unclear, and a Chebyshev spectral approach might be more suitable. The poor results with the spectral method could mislead naïve readers about its effectiveness for simple PDEs.\n\n•The impact of varying the number of collocation points, neurons, and hidden layers should be investigated. The iterative nature of PINN training means convergence behavior needs to be reported in detail. Different initializations often yield varying results, and while the shaded regions in Figure 2 might address this variability, the context needs to be clarified.\n•Finally, the relative l2 error, though mentioned, does not specify the mesh of points on which it is computed. This crucial detail should be clearly communicated.\n\nClarity, Quality, Novelty, and Reproducibility:\n\nThe paper is generally well written and presents its ideas with clarity. However, there are areas where more information is needed to fully assess the significance, novelty, and reproducibility of the proposed methods.\n\nClarity:\n\nThe manuscript is clearly structured, and the authors explain their ideas effectively. However, the limited experimental section hinders a deeper understanding of the real impact of the proposed approaches. The clarity of the presentation would benefit greatly from more detailed and varied experimental results. Specifically, exploring a wider range of PDEs beyond simple trigonometric functions would provide stronger support for the claims made in the paper.\n\nQuality:\n\nThe experimental evaluation is rather weak, making it difficult to assess the practical value of the methods. The lack of comparisons with traditional numerical methods, such as finite difference (FD), finite element methods (FEM), or spectral methods, leaves a gap in understanding how these approaches perform relative to well-established techniques. The omission of a detailed computational cost analysis further limits the ability to gauge the efficiency of the proposed methods. The experiments provided are confined to basic examples, and the absence of a broader evaluation against diverse PDEs reduces the generalizability of the findings.\n\nNovelty:\n\nThe paper introduces some ideas, particularly the adaptive distance functions and the Fourier PINNs. However, the novelty of these contributions seems limited. The adaptive distance function could be seen as a small extension of existing numerical techniques, and its impact appears minor. The Fourier PINNs approach, while intriguing, largely focuses on trigonometric functions, which naturally fit into a Fourier framework. Testing the method on non-periodic solutions would help better establish its general applicability. Additionally, the handling of nonlinearity in PDEs, especially with the Allen-Cahn equation, raises concerns about whether the proposed method is truly advancing the state of the art. Overall, the contributions, though potentially useful, might be perceived as incremental rather than groundbreaking.\n\nReproducibility:\n\nWhile the paper is well written, it lacks the necessary details to ensure that the methods can be easily replicated. Key information such as training methods, learning rates, number of epochs, hardware specifications (e.g., CPU, GPU), and the precise number of collocation points should be explicitly stated. Additionally, the paper would greatly benefit from releasing the code alongside the manuscript.\n\nMinor Points:\n\n•The domain of PDEs is stated as [0, 2pi], but the left boundary conditions appear to be applied at x=1 in equations 12 and 14. This seems inconsistent with the stated domain and should be clarified.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce two distinct approaches for solving PDEs using PINNs. The first method enforces boundary conditions by incorporating simple distance functions, demonstrated in 1D domains with simple polynomial combinations and tunable exponential terms to adjust the influence on boundary conditions relative to the internal domain.\nThe second approach, termed Fourier PINNs, combines a Fourier series expansion with a neural network by superposition. The coefficients for both the Fourier basis and the neural network are trained concurrently.\nWhile the initial motivation of the paper appears to focus on handling boundary condition challenges, the work ultimately shifts towards approximating high-frequency solutions of PDEs. The central challenge remains somewhat ambiguous, as the narrative oscillates between mitigating high-frequency components as noise and focusing on high-frequency solution examples, such as sin(100x) and more complex functions.\nThe treatment of general PDE solutions is limited, as the examples predominantly involve smooth trigonometric functions. The scope of examples is also somewhat narrow, with multiple cases feeling repetitive and not contributing additional insights.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Strengthen the Experimental Section:\n\n•\tExpand the experimental section by including a broader range of test cases, particularly PDEs with non-periodic solutions. \n\n•\tProvide comparative results against traditional numerical methods such as finite difference (FD), finite element methods (FEM), and spectral methods. If a direct comparison is not feasible, at least acknowledge the efficiency and accuracy of these well-established methods, especially in low-dimensional settings. It is important that the paper does not overstate the computational efficiency of the proposed methods, as FD and FEM often outperform PINNs in terms of computational cost and accuracy, particularly for simpler problems.\n\nAcknowledge Limitations and Alternative PINNs Approaches:\n\n•The paper should explicitly acknowledge the limitations of the proposed methods, particularly in terms of scalability, computational cost, and performance relative to traditional methods like FD and FEM. It is important to clarify that while meshless methods like PINNs hold promise, they are not currently competitive with FD, FEM, or spectral methods in many scenarios, especially in low dimensions or simple geometries.\n\n•Additionally, the author should acknowledge the existence of more advanced and computationally efficient PINNs approaches, such as those based on random projections, random features, randomized networks, and extreme learning machines. \n\nClarify Methodology and Assumptions:\n\n•Clearly explain how the adaptive distance function method could be extended to higher dimensions and different types of boundary conditions.\n\n•Provide additional details on how the Fourier PINNs method handles nonlinearity, particularly in the case of the Allen-Cahn equation.\n\n•Correct the error regarding the L2 regularization claim.\n\nImprove Reproducibility.\n\nClarify and Correct Minor Issues\n\nRevise and simplify Figures for better clarity.", "broader_impact_concerns": "There are no broader ethical concerns or societal implications associated with this work, as it primarily focuses on technical methods for solving partial differential equations using machine learning techniques. The paper does not involve sensitive data, user interaction, or applications that could raise ethical concerns. Therefore, no broader impact statement is required in this context.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript received the careful evaluation of three reviewers. My summary of their feedback is the following:

**Strengths**
* The problem studied here is important and relevant to the PINs literature, and the presentation is clear and instructive.
* Their proposal and solutions are well-motivated and implemented.
* The advantage of Fourier PINs is evident.

**Weaknesses**

The contribution is somewhat limited, in the sense that:
* The authors only study Dirichlet BCs, but the method is not applicable to more complex alternatives like Neumann or non-linear boundary conditions.
* One of the advantages of learned approximations to solving PDE is the better scaling with the problem dimension. Yet, the approach presented here scales poorly with this parameter and is thus limited to small dimensional problems.
* The empirical evaluation is limited to 1D and 2D simple problems, mostly with strong harmonic components, and it is thus unclear whether the method's benefits will carry as significantly to more general settings.

### Conclusion

The responses from the authors have been satisfactory, in the sense that they acknowledge the limitations of the proposed method and mention that these will be made very clear in the revised manuscript. Furthermore, the authors will also carry out further experiments in other systems without such explicit components of harmonics, in addition to commenting on potential extensions to address the issues of scaling with dimensions and more general boundary conditions (as well as the symmetric constraint on the steepness parameter). 

There is some discrepancy between the recommendations of the three reviewers. However, the consensus is that if the authors implemented all of the changes that they mentioned in their responses, the majority of them would be happy to recommend this paper for acceptance alluding to the strengths highlighted above. Therefore, I am recommending acceptance conditioned on these changes being made. While some of these changes are non-negligible (especially the experimental ones), I regard these as "minor" in the sense that the overall methodology and motivation will remain the same, but I expect the authors to consider serious modifications to address the reviewers' questions and concerns.
Decision: Accept (poster)