Iterative Training of Physics-Informed Neural Networks with Fourier-enhanced Features
Authors: Yulun Wu
Miguel Aguiar
Karl Henrik Johansson
Matthieu Barreau
Venue: ICLR 2026 Poster (2026)
Keywords: physics-informed machine learning
random features
differential equations
spectral bias
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Abstract: Spectral bias, the tendency of neural networks to learn low-frequency features first, is a well-known issue with many training algorithms for physics-informed neural networks (PINNs). To overcome this issue, we propose IFeF-PINN, an algorithm for iterative training of PINNs with Fourier-enhanced features. The key idea is to enrich the latent space using high-frequency components through Random Fourier Features. This creates a two-stage training problem: (i) estimate a basis in the feature space, and (ii) perform regression to determine the coefficients of the enhanced basis functions. For an underlying linear model, it is shown that the latter problem is convex, and we prove that the iterative training scheme converges. Furthermore, we empirically establish that Random Fourier Features enhance the expressive capacity of the network, enabling accurate approximation of high-frequency PDEs. Through extensive numerical evaluation on classical benchmark problems, the superior performance of our method over state-of-the-art algorithms is shown, and the improved approximation across the frequency domain is illustrated.

Published as a conference paper at ICLR 2026
ITERATIVE TRAINING OF PRYSICS-INFORMED NEURAL NETWORKS WITH FOURTER-ENHANCED FEATURES
Yulun Wu, Miguel Aguiar, Karl H. Johansson & Matthicu Barreau
Division of Decision and Control Systems
Digital Futures and KTH Royal Institute of Technology Stockholm, Sweden [yulunw, aguiar, kallej, barreau)ekth.se
ABSTRACT
Spectral bias, the tendency of neural networks to learn low-frequency features first, is a well-known issue with many training algorithms for physics-informed neural networks (PINNs). To overcome this issue, we propose IFeF-PINN, an al- gorithm for iterative training PINNs with Fourier-enhanced features. The key idea is to enrich the latent space using high-frequency components through Ran- dom Fourier Features. This creates two-stage training problem: (i) estimate basis in the feature space, and (ii) perform regression determine the coef- ficients of the enhanced basis functions. For an underlying linear model, is shown that the latter problem is convex, and prove that the iterative training scheme converges. Furthermore, we empirically establish that Random Fourier Features enhance the expressive capacity of the network, enabling accurate ap- proximation of high-frequency PDEs. Through extensive numerical evaluation on classical benchmark problems, the superior performance of our method over state-of of te-thhe-n art algorithms is shown, and the improved approximation across the frequency domain is illustrated.
INTRODUCTION
Capturing freb-frreeenyy behavior is central to modeling complex phenomena such as wave prop- agation, turbulence, and quantum dynamics. Traditional numerical methods, including spectral ap- proaches (Boyd, 2001), multiscale schemes (Weinan & Engquist, 2003), and oscillatory quadrature (Iserles & Nørsett, 2005), have achieved notable success but often require problem-specific adapta- tions or become prohibitively costly in complex or high-dimensional settings.
There is need for new approximation strategies that capture high-frequency behavior without rificing stability or tractability. Deep-learning surrogates of differential equations are promising alternative, such Physics-Informed Neural Networks (PINNs), which offer grid-free alternative by combining data and physical models within a neural network framework (Raissi et al., 2017). This paradigm has shown strong performance in solving partial differential equations (PDEs) and inferring hidden dynamics, benefiting adaptability to complex geometries (Costabal et al., 2024), and high-dimensional scalability (Hu al., 2024). Related approaches such as Fourier Neural Operators (Li et al., 2021) and DeepONet (Lu et al., 2021) further expand its reach. Despite these advances, PINN methods remain limited by spectral bias-the tendency of neural networks to learn low- frequeney components first-which hinders accurate recovery of oscillatory solutions (Rahaman et al., 2019; Xu et al., 2025; Lin et al., 2021; Qin et al., 2024).
Several strategies have been proposed to mitigate spectral bias, including weight balancing (Wang et al., 2021a: Krishnapriyan et al., 2021), resampling (Lau et al., 2024; Tang et al., 2024: Song, 2025), and curriculum architecture- based approaches (Sirignano & Spiliopoulos. 2018; Waheed, 2022: Chai et al., 2024; Mustajab et al., 2024; Eshkofti Barreau, 2025; Wang & Lai, 2024). Table summarizes of the most representative approaches. While effective in certain cases, these

---

Published as a conference paper at ICLR 2026
Extenbled basks
Ex(2.
2 6 f
h
79
vip
Nominal basis
Forward passs
Backward pass
Figure I: Architecture IFFF-PINN, The first part (in yellow) generates the nominal basis vectors,
which are then extended via YD generating random Fourier features LDD (in green), and a linear combination of the extended basis (in blue) forms the approximated solution Mw.0.
Table 1: Representative methods for approximating solutions to PDE, highlighting application do- main, key idea, high-frequency handling (HF), limitations, and optimality.
Method
Domain
Key Idea
HF
Limitations Optimality
Boyd (2001); Iserles & Linear Nørsett (2005)
Global basis functions (Fourier, Chebyshev)
+++ Requires regular domains; global optimum
Weinan (2003)
& Engquist
Multiscale
Separate scales and compute effective dy- namics
Needs clear scale separation, prohlem-specifie; local optimum
Raissi et al. (2017)
Generie
minimizing physics + data loss
Struggles with high-frequency components; local optimum
Li al. (2021); eu al. (2021)
Operator
Learn mapping
Problem-specific, may require large networks; local optimum
Fourier / function space
Chai et al. (2024); Zhao ct al. (2023)
Multiscale
Network architecture or training strategy
++
Problem-specific, not robust; local optimum
Lau et al. (2024); Tang et al. (2024); Song
General
Adaptive resampling
++
Computationally expensive, no convergence guarantees; local optimum
(2025)
IFFF-PINN (this work)
Generic
Iterative training with extended basis via
++ Not adapted resampling, high memory footprint: Global opti- mum (for linear PDFs)
Fourier features
methods remain tied single-level optimization frameworks, where feature learning and coefficient fitting are intertwined in neural networks, limiting both robustness and theoretical guarantees.
To address this gap, we draw inspiration from classical numerical PDE solvers, which approximate solutions using basis functions, and propose novel neural network architecture and tailored train- ing algorithm. The key idea is to create a feed-forward neural network with three components, as illustrated in Figure 1. First, the hidden layers h generate nominal basis in the latent functional space. Next, this basis extended to d'D through Random Fourier Features (RFF, introduced by Rahimi & Recht (2007)), which may include potentially figher-frequency elements, span larger latent space. Finally, the last linear layer performs regression on these extended basis vectors. The first and last blocks can be optimized separately, resulting in two-stage iterative scheme alternat- ing between latent basis construction and regression on output coefficients. A major feature of this framework, related extreme learning machines (Dwivedi & Srinivasan, 2020), is that for linear differential equations, the regression stage is convex and achieves asymptotic global optimality. Un- like existing approaches, our method enriches the latent space representation, enabling systematic capture of high-frequency dynamics while leveraging the strengths of established PINN frameworks.
Features (IFeF-PINN), a novel iter- ative two-stage training algorithm that mitigates the spectral bias of PINNs in high-frequency prob- lems while maintaining accurate approximation on standard benchmark PDEs. Our contributions are threefold: (i) introduce a flexible building block that augments existing PINNs architectures with improved high-free geghffreuuncy estimation and demonstrate its universal approximation capabilities; (ii) propose iterative two-stage training algorithm and prove its convergence properties; and (iii) we validate the approach through extensive simulations on benchmark problems, showing sub- stantial improvements over existing methods.
In this paper, we propose Iterative PINNs with Fourier-Enhanee
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
2.1
PHYSICS-INFORMED NEURAL NETWORKS
PINNs is deep learning framework that integrates PDEs into the neural network training via the loss function, enabling data-driven learning with physical constraints (Raissi et al., 2017; Karniadakis al., 2021).
Generally, for > 0, let Ω C IR be bounded domain and W an appropriate Sobolev space of functions from R, we consider linear PDEs of the form
3l)[(2) f(z), zEb Ω
2 W (8) - g()), sETCan,
where vr E W is the solution, 3 W + (R" ,,", R) is the linear differential operator, f (S, IR) the source term, B : W (T) the linear boundary/initial operator, 9 V(T) specifies the boundary/initial conditions, where )(T) denotes the appropriate trace space. We assume that this problem is well-pooe well-posed and therefore has unique solution in W.
The objective of PINNs is to approximate the solution u with feedforward neural network uw where denotes the network parameters. Yeonjong al. (2020) and Sirignano Spiliopoulos (2018) analyzed consistency in weak formulations under suitable assumptions, motivating the fol- lowing continuum loss:
Ex(uw) J [ 10066) 28 (()w
A [ J [1(
with > where, for A a bounded set, |.1| denotes its measure. However, this version is not numerically tractable and, in practice, we use the Monte Carlo approximation
No > (l(((t) B B(unl(,))13
Ex(uw)= Nu
(3)
N
where { and {-y))i=,,...,,, are uniformly sampled on I' and Q, respectively. Finally, the optimal parameters are found as w" arg min,, (w.).
2.2 RANDOM FOURIER FEATURES
In this work, we use Random Fourier Features (RFFs) introduced by Rahimi & Recht (2007) to include high-frequency terms. Grounded on Bochner's 's theorem, RFF provides a way to explicitly construct feature map that approximates a stationary kernel, enabling the scaling of kernel methods to large datasets.
RFF has been used by Tancik al. (2020) to tackle spectral bias. The novelty extend the input to the neural network using the RFF mapping
(2 B nx) VD
<D(x)
E R22D
(4)
(2n BDz)
where the entries of the matrix Bn Dxn are sampled from a given symmetric distribution. Wang et al. (2021b) adapted this method to PINNs by using u. from the previous section with 2D inputs, so that the neural network becomes tw o TD. This new architecture can learn approximate the solution from the enriched inputs.
3 PROPOSED METHOD
We leverage the PINNs and RFFs in a novel way. Note first that the PINN training process couples two roles within a single nonconvex objective: (i) hidden layers h leam nonlinear feature basis, and (ii) a linear regression operator Pa hi h 0 finds the optimal projection coefficients 0 of the approximated solution onto the feature basis, thereby minimizing the loss L2. This coupling leads
3

---

Published as a conference paper at ICLR 2026
to PINN pathologies, where gradients from interior residuals dominate and suppress boundary terms, and spectral bias drives low lwwfrreuency learning first, leaving oscillatory components underfit and slowing convergence high- high-frreuency modes (Wang et al., 2021b; 2022).
To overcome this coupling issue, we approximate the solution u to the PDEs in (1) as a linear combination of basis functions. We thus consider the two problems in isolation: basis generation, which we will denote as the upper-level problem, and linear regression on the basis functions, which will refer to the lower-level problem.
3.1 THE UPPER-LEVEL PROBLEM: BASIS FUNCTION GENERATION
The initial step for the basis generation is to follow the classical PINN methodology and train a standard ford-forward neural network with parameters (w, W), denoted by
W.,w (x) (h)
92
minimize w, W + Lx( This typically accomplished using gradient-descent numer- ical scheme, such as ADAM (Kingma & Ba, 2014), more complex second-order solver, like L-BFGS (Liu & Nocedal, 1989). Then, the neural network h R + TRP generates a basis hw C(R, R²) of the latent space while W is the projection operator. This initial step serves warm-up for the upper-level problem. Note that u.W most likely contains only the www-frequency components of the original solution. Therefore, the surrogate M.,W might be an aliased or steady- state solution of the PDE, and the fit at the boundary points might be poor.
In our approach, the strategy is to apply upgrades the implicit linear kernel on h. to a stationary kernel, such as a radial basis function, in the adaptive feature space. Since à is probably distorted version of the real solution u, the RFF extension might bring higher frequency signals that mitigate the spectral bias.
RFF mapping to the last hidden layer features h This
1 cos(2TBrllue (x))
Concretely, we define (z(x)) TD (z)(
where BD RxP
vD sin (1))
constant matrix with entries sampled i.i.d. from NN0, o2).
3.2 THE LOWER-LEVEL PROBLEM: LINEAR REGRESSION
The linear output layer over hi induces a dot-product kernel in feature space, which can limit expres- sivity and exacerbate spectral bias toward low frequencies. Applying RFF to h. equips the adaptive features with stationary kernel without adding trainable parameters, injecting high-freqqenne com- ponents via random projections. Formally speaking, an approximate solution the PDE in (1) with 0 E 2D becomes
uw (x) = (n(()) 0, x E Ω.
(5)
As we show in Appendix B, since the operators 3 and B are linear, the loss function εx (u(,,)) is quadratic A:
Liower (0 w) == = Q( c(w) bb,
(6)
where Q and e collect boundary and interior residual terms.
Proposition 1. Assume that X >> O and that the rank condition (3) from Appendix B.1 is verified. Then Q is positive definite and there is a unique solution to arg ming Llower () -Q- (w)c(w))
The proof given in Appendix B.I. The application of the RFF mapping in the last hidden layer enables the generation of an arbitrary number of basis functions vnD independently of the network's width on which we can leverage quadratic programming to get the unique optimal solution. This would otherwise not be possible because constrained by the basis dimension.
3.3 THE GLOBAL BI-LEVEL PROBLEM
Combining the results from the two previous subsections, we get the following formulation that decouples basis learning (upper-level) from linear regression (lower-level):
w (0) arg min E* (uw,e) e= == arg min Lupper(w (w 0),
(7)
8" (w arg min 2x (xw,o) == arg min Liower (0 w).
4

---

Published as a conference paper at ICLR 2026
The classical bi-level optimization framework (Bard, 1991) proposes the following three-step merical method: (i) sample Wo, 8g randomly; (ii) solve the upper-level problem wt = w0()) (iii) solve the lower-level problem o+ 0 (wt (w+). The final parameters (wt (*) are the optimal solutions to the bi-level optimization.
However, this approach does not consider warm start and is not particularly adapted to a learning problem. For better approximation capabilities, we propose an iterative scheme. We warm start using a vanilla PINN pre-traine pre-training to get an initial value wo for the weights of the basis generator. Then we com- pute lc+1 = 0* (w:) before performing a one-step gradient-descent on wi to minimize Supper (e+1) to get 11 This leads to Algorithm 1. The conver- gence of this numerical scheme and the approxima- tion capabilities of the new neural network architec- ture studied in the next section.
Algorithm 1 IFeF-PINN for linear PDEs
Initialize network parameter wo, 6g and B
for from 0 to Nepoch do Formulate extended RFF basis 4nD Lower update:
0k+1 = ((wx) ( l((w))
Upper update:
Wk+1 = Wi V Lupper 0k+1)
end for return WNepech O" ("WN Naooc)))
Remark (Relation deep kernel learning). In deep kernel learning, learn a nonlinear feature transformation, and a Gaussian process is defined over the resulting feature space using traditional kernel function. This enables learning flexible, data-driven kernel that combines the expressiveness of deep learning with the uncertainty estimation of Gaussian processes (Wilson et al., 2016). However, the best of the authors' knowledge, learning Gaussian process with a nonlinear PDE prior is not yet possible (Jidling et al., 2017); we propose a solution in this case.
neural network to
Remark (On the warm start). Pre-training standard PINN for several hundred epochs provides initial network parameters for basis generation. This is necessary for homogeneous PDEs to prevent convergence u = O, since standard initialization yields near-zero outputs that trivially minimize the lower-level problem. For non-homogeneous PDEs, the source term prevents this issue.
3.4 EXTENSION TO NONLINEAR PDES
For nonlinear PDEs, the physics residual term A EN [3 [( becomes nonlinear in 6, making the lower-level problem L0 w) non- convex and lacking closed-form solution. We therefore replace the exact solution in Proposition with gradient descent to find an approximate local minimizer when the Second-Order Sufficient Con- dition (SOSC) holds, i.e., when the gradient van- ishes and the Hessian is positive definite. The com- plete update is given in Algorithm 2. For computa- tional efficiency, we update 0 to a local minimizer every Niower epochs. For initialization, ther warm start only the network parameters w via standard PINN pre-training initialize both w and 0 jointly via end-to-end training as discussed in Section 6.
Algorithm IFeF-PINN for nonliear PDEs
N,
Initialize network parameter wp, 6g and B
for from 0 to Nepoch do Formulate extended RFF basis v/n Lower update:
if is mod Niower then 0 z arg min Llower(wk 0x)
else
0x+1 = 02 end if Upper update:
can ei-
Wh+1
nhw Vw Cupper (x+1)
in the linear case, or
end for
return WNagwch 0* (WNoopoc))
THEORETICAL ANALYSIS
4.1 CONVERGENCE PROPERTIES OF THE BI-LEVEL ALGORITHM
We establish convergence by showing that the optimal lower-level solution (w)) is Lipschitz con- tinuous with respect to the upper-level parameters w, which ensures well-defined Lipschitz hyper- gradient for gradient descent on the upper level.
Proposition 2 (Lipschitz Continuity of the Solution Map). Let the lower-level problem be strongly convex OP problem parameterized by w. Assume that the mappings w + Q(w) and w + c(w) are locally Lipschitz continuous, and that the smallest eigenvalue of Q(w) uniformly bounded below
5

---

Published as a conference paper at ICLR 2026
by pQ on any compact wf. w. Then, the optimal solution map 0 (w) is also locally Lipschitz continuous with respect to w.
The detailed proof is provided in Appendix C.2.This also holds in the nonlinear PDE cases, when the SOSC satisfied, the local minimizer (w) retains Lipschitz continuity and differentiability in neighborhood of Consequently, the hypergradient is L-smooth, which we leverage in our convergence analysis.
Theorem (Convergence to a stationary point). Assume that 1) the functions Q and e are continu- ousty differentiable with respect to w. the upper-level loss Lupper continuously differentiable with respect to both 9 and w: 2) The lower-level problem is p-stroogly convex; 3) the objective function Supper 0) is bounded below and its hypergrudient is L-smooth.
Then, the sequence of iterates {wk Vi=o generated by the gradient descent algorithm with a constant step size 1j E (0, L) converges to a stationary point of Eupper( 0).
The assumptions made are classical in learning problems and are a direct consequence of the struc- ture of the bi-level framework. A formula for the hypergradient is derived via the Implicit Function Theorem in Appendix C.I, showing it as composition of smooth functions. Its Lipschitz continuity then guaranteed by the Lipschitz continuity of the solution map 0* established in Proposition 2.
4.2 UNIVERSAL APPROXIMATION CAPARILITIES
To analyze the expressiveness of the augmented features, we show that the hypothesis class not less expressive than linear readouts over the last hidden layer features. The necessary func- tion spaces for this analysis are defined with comprehensive foundational definitions and proofs in Appendix
Definition 1. The feature space Hfy and the composite RFF function space HRFF are defined as:
H == {g g(x) hw (z) 0, 0 E } 'REFF == {9 g(x) = <>(()) 0, 0 E IR2)).
(8)
where DD YD hw denotes the vector of composite RFF features defined in Equation
We will show that HaFF strictly contains Hy, and thus defines a more expressive hypothesis class. The argument constructs bridge between the two spaces using reproducing kernel Hilbert space.
Theorem 2. Let be any target function in L (St, (S2,TR) TR.. The projection error (see Definition 3 in D.1) achievable by the composite RFF Function Space HRFF is greater than the projection error achieved by the original Feature Space H, when the number of RFF features D goes to infinity.
The proof is given in Appendix D.2. This result establishes powerful theoretical assurance that RFF embedding offers better approximation capabilities. Theorem 2 yields the universal approximation corollary presented below, the proof of which is given in Appendix D.2.1
Corollary (Universal approximation). The projection error of the solution equation I onto HRFF can be made as small as desired, provided enough neurons and RFF features D.
5 RELATED WORK
Weight-balancing strategies These methods adapt the physics weight À in equation 3 during training. For instance, (Wang et al., 2021a) dynamically updates X to balance the gradients of data and physics losses, while the NTK framework (Jacot et al., 2018: Krishnapriyan et al., 2021) enforces equal decay rates, theoretically recovering feeuurryy solutions. Primal-dus methods (Goemans & Williamson, 1997; Barreau & Shen, 2025) instead compute À from the PDE residual. Although simple to implement, these approaches offer weak convergence guarantees and remain tied to single-level optimization. Nonetheless, they are complementary to our framework and could strategies within the upper-level problem. proble....
be integrated as
Resampling strategies A second line of work reduces the gap between the true loss Ex and its sampled counterpart Σ,. Examples include NTK-informed sampling (Lau et al., 2024), adversarial sampling (Tang al., 2024), and reinforcement learning (Song, 2025). While effective in reducing approximation error, these methods do not explicitly target spectral bias, which is the focus of our proposed method.

---

Published as a conference paper at ICLR 2026
Curriculum learning strategies Finally, new architectures and training schedules aim to bet- ter capture high-freet frequecyyy components. Attention mechanisms (Sirignano & Spiliopoulos, 2018), multi-stage networks (Howard et al.. 2025; Waheed, 2022; Chai et al., 2024; Mustajab et al., 2024; Eshkofti & Barreau, 2023; Wang & Lai, 2024), or finite-basis approximation (Moseley et al., 2023) have shown improved multi-scale resolution. However, their complexity often makes training slow and delicate, and they still lack dedicated optimization algorithms.
6 NUMERICAL EXPERIMENTS
Objective. In this section, we describe comprehensive experiments that establish four main ad- vantages of IFeF-PINN. First, improved approximation over PINNs and SOTA variants low- frequency PDEs. Second, higher accuracy on high-frequency and multi-scale linear PDEs, where standard PINNs typically show failure modes. Third, our framework exhibits strong generalization capabilities when integrated with advanced PINN variants. Finally, spectrum analysis experiment demonstrates that proposed method improves the network fitting accuracy for high-frequency signals.
Experiment setup. We will use four PDEs, namely the 2D Helmholtz equation (low and high frequency), ID convection equation (low and high frequency), ID convection-diffusio equation, and the viscous Burgers' equation. The baseline methods are Vanilla PINNs, NTK (Wang et al., 2022), PINNsformer (Zhao et al., 2023), and Inffrmee Gaussians (PIG) (Kang al., 2024), keeping their default settings for a fair comparison. Additional experimental comparisons with Multiple Fourier Features (MFF) (Wang et al., 2021b) are provided in Appendix G.1. For sim- plicity, we set X = 0.01 for the Vanilla PINNs in Equation 3. Detailed hyperparameters for our proposed methods are Appendix E. For low-frequency 2D Helmholtz and low-frequency ID con- vection equations, we adopt the uniform sampling strategy settings of Zhao et al. (2023). For the viscous Burgers' equation, we follow the setup of Raissi al. (2019). For the
Helmholtz equation, we employ Latin hypercube sampling (McKay et al., 2000) to improve do- main coverage. We evaluate two variants of our framework: IFeF (Vanilla training) and IFeF-PD (primal-dual weight-balancing proposed by Barreau & Shen (2025)). PDE definitions, datasets, and network architectures are provided Appendix We measure the relative L2-error after 1|uel== Each method is run five times with independent random convergence, defined as || seeds, with the best predictions for each approach. All models are implemented in Py PyTch and trained on a single NVIDIA GeForce RTX 4090 GPU. The code for all benchmarks are available at https: / / g thub. in Appendix G.2.
etw.rummm/eeeeeeeeeeeeeeeeeeeeee wwI--FFFFIII-- Computational aspects evaluated
6.1 RESULTS ON BENCHMARK PDES
We begin with three popular low-frequency benchmark PDEs: 2D Helmholtz equation, 1D convec- tion equation, and the viscous Burgers' equation. Figure 2 summarizes relative L2-errors across baseline methods; boxplots display medians and IQRs, and red diamonds denote means. Additional prediction and absolute error maps are provided in Appendix G.
I0R
10R
Relative 2. Error (log scale)
13k
108
B00250
ICR
10-2
ICR
000393
Method
000007
10-3
PiNllffomme
PIG
IFFFUPD
3000
0 3000)
Holmboltz (o)
d.
Convection (ff 51)
Burgers Equation
Figure 2: Boxplot of relative T.2-rrooss (log10 scale) for all methods three low-frequency bench- marks with median, inter-quartile range (IQR), and mean (red diamonds).
7

---

Published as a conference paper at ICLR 2026
Across all these problems, our proposed method attains the lowest median errors with reduced vari- ability. On Helmholtz, IFeF-PD achieves the best relative L error 3.5 x 10 On convection, IFeF achieves the best error of 4.3 10 Even in the nonlinear case of Burgers' equation, IFcF obtains the lowest median error. In addition, we conducted an ablation study where we discarded the RFF basis extension but performed similar iterative two-step optimization process, obtaining results that were similar but slightly better than those of the Vanilla PINN (1.4923 x 10 rela- tive L'-error) on the low-frequency convection problem. Figure 3 presents the predictions for the low- wwefreuuencc 2D Helmholtz case. On logarithmic scale, the gap between IFeF-PINN and other methods is consistent with the box plot summaries. These results highlight the strong approximation capability of the proposed method, especially for linear equations, underscoring its robustness for solving diverse PDEs.
Vonila
PIKNsfommer
NTK
IFeF
IF+F-PD
00
00
Abs. Eni. [Varila)
Abs Err, [PINNslomer]
Abs Err. [NTK]
Abs. Lo. (PTG)
Abs. [IFe-]
Abs. Erc. WD-F-FD
1012
10r4
1004
10-4
10+
10-4
10-1
04
10-4
10-4
20-4
3-
10
Figure 3: Low-frequency Helmholtz equation prediction solution (up) and absolute error log scale (bottom) of baseline methods.
6.2 MITIGATING THE SPECTRAL BIAS
To evaluate challenging cases of spectral bias, we study the failure modes of PINNs on high- frequency and multi-scale PDEs, where vanilla PINNs typically struggle learn rapidly oscillatory or widely separated frequency components. In particular, we study the high-frequence Helmholtz and convection equations, as well multi-scale convection-diffusion equation. Table 2 presents the mean and standard deviation of the relative -erors over baselines applied to these problems. Additional prediction and absolute error maps are provided in Appendix G.
Helmholtz
Convection (8 = 200)
Convection-Diffune
Baseline
(a) = 100)
(Noww 4m, hnggh = 60Tr)
Vanilla
0.9024 (0.0239)
0.0501 (0.0030)
PINNsformer
1.2278 (0.2010)
0.0525 (0.0001)
NTK
0.8685 (0.0318)
0.0526 (0.0001)
PIG
1.6884 (0.2775)
1.0009 (0.0003)
0.0560 (0.0010)
IFeF
0.0156 (0.0055)
0.0027 (0.0010)
0.0009 (0.0003)
IFeF-PD
0.0092 (0.0031)
0.0025 (0.0005)
0.0010 (0.0002)
Table 2: Average relative T.2-rrror with corresponding standard deviation for each baseline on three high-frequency PDEs. A dash "-" denotes that the bascline failed converge.
Figure depicts the high-freque frequeerreencyy Helmholtz solutions and the corresponding log-scale absolute errors. In the considered scenarios, all basclines exhibit clear failure modes. We also conducted similar ablation study as described in the previous section, removing the RFF basis extension, and the training did not converge for both the high-frequency Helmholtz and convection equations. In contrast, the proposed IFeF-PINN method effectively mitigates the spectral bias of neural networks. Moreover, when combined with the primal-dual method to adaptively balance the physics-based loss, our method achieves accurate solutions even under very high frequencies, which illustrates the flexibility of the proposed framework in incorporating advanced learning methods. A similar result holds for the multi-scale wwtrectioondiffusiii equation in Figure in Appendix G, clearly showing that only IFeF-PINN succeeds in learning both low and high frequency components of the solution.
8

---

Published as a conference paper at ICLR 2026
Vonila
PHkdNeformer
NTK
PIG
IFee
IFeF-PD
-2.
Als. (Vamila)
Alsx
10
1017
U0
10
20-1
Figure 4:
High-frequency Helmholtz equation prediction solution (up) and absolute error in log scale (bottom) of baseline methods.
In contrast, all baselines suffer from the spectral bias failure mode, where models prioritize learning low-frequency components and tend to ignore the high-frequency components.
6.3 SPECTRUM ANALYSIS
To quantitatively demonstrate our method's ability to mitigate spectral bias, employ the Fast Fourier transform to analyze the frequency-domain distribution of the network's prediction. We conduct spectrum analysis similar to Rahaman et al. (2019), designing challenging multi- scale convection equation with an initial condition composed of a superposition of ten sinusoids of different frequencies and unit amplitude. More details of the setup are in Appendix F.2.
During analysis, compare the performance of Vanilla PINNs against models where the basis is ex- tended with varying number of random Fourier features and carry out a one-step solution of the lower-level objective in Equation No additional training is performed for the upper-level problem.
Foature Number
3200
2900
2000
06
800
400
0.2
Vanila
We compute the magnitude of their discrete Fourier transform at frequencies k; denoted as l. Fig- ure 5 presents the average normalized magnitudes [fk,l A; over five independent runs. The results clearly illustrate the spectral bias Vanilla PINNs, which struggle to accurately capture high frequeryyynncy com- ponents. In contrast, by extending the network's ba- sis through RFF, the network fit high-frequency signals much more effectively, even without the sub- sequent bi-level training procedure of IFeF-PINN. Furthermore, observe that increasing the num- ber of random features enhances the network's 's ability to approximate high-frequency components, confirming the effectiveness of our basis extension strategy.
Frequency (1
Figure 5: Prediction of the network spec- trum with increasing number of Fourier features. The x-axis represents frequency, and the colorbar shows the normalized mag- nitude of the predicted solution at t = 0. The colorbar scaled accordingly from 0 to 1.
6.4 ABLATION STUDIES
In this section, present experiments demonstrate the effects of two-stage training in IFeF-PINN and the number of Fourier-enhanced features and the Gaussian sampling parameter
6.4.1 END-TO-END TRAINING
To validate the necessity of two-stage training in IFeF-PINN, we conduct an end-to-end ablation where both network parameters w and coefficients are jointly optimized. We keep the approxi- mation in Equation 5 but incorporate 0 as learnable parameters alongside and directly minimize Lx (x(M,,e) without the two-stage training. Unlike FFFF-PINN where 0 is always optimal under current features vip (z). 0 randomly initialized and updated simultaneously with w, losing the optimality guarantee. Table 3 presents the results for low- and high-frequency Helmholtz and Burgers' equa-

---

Published as a conference paper at ICLR 2026
tions. This ablation validates the necessity of two-stage training, IFeF-PINN significantly out- performs end-to-end training in linear PDEs through guaranteed lower-level optimality of 6, while showing modest improvements in nonlinear PDEs where the lower-level becomes non-convex.
Helmholtz
Helmholtz
Viscous Burgers 0.01
Ablation
(a1 1,
4)
(a) = 02 100)
End-to-End
0.0088
0.0049(0.0009)
IFeF
0.0003(0.0003) 0.00005(0.00002)
0.0156(0.0055) 0.0092(0.0031)
0.0024(0.0011)
IFeF-PD
0.0033(0.0004)
Table 3: Average relative L with corresponding standard deviation for end-to-end training and IFeF-PINN on three benchmarks. A dash "-" denotes that the baseline failed to converge.
6.4.2 HYPERPARAMTER ABLATION
We conduct an ablation on two key hyperparameters in IFeF-PINN: the number of Fourier features D and the Gaussian sampling parameter a. We evaluate their impact on performance using the low- and high-frequency Helmholtz equations, with results shown in Table 4. The ablation shows that too few features reduce expressivity while excessive features cause overfitting and may break the rank condition discussed in Appendix B.1. For o, larger values essential for high- high-frequency problems discussed in Tancik et al. (2020); Wang et al. (2021b). Low-free wow-frequnncy problems are robust to both hyperparameters, while high-frequency problems sensitive, especially to
Helmholtz (a1 1, 0g
D (a 1) Rel. 1,2 error
800
1200
3000
5.5 x 10
2.1 x 10
3.2 10
5.7 10
4.5 10
(D 800) Rel. L error
2
0.5
0.2
0.1
4.0 x 10
3.2 x 10
5.5 x 10
3.3 10
1.5 x 10
Helmholtz (a: 0u 100)
D (a 1) Rel. L error
800
1200
1600
2400
3000
7.11 x 10-2
5.40 x 10-2
3.09 10-2
1.56 x 10
2.22 x 10
(D = 2400)
20
10
0.2 1.05 x 10-1
Rel. L emrrr
4.6 x 10-3
3.0 10
5.7 10
1.56 x 10-2
Table 4: relativee -e-error for hyperparameter ablation for D and a on Helmholtz equations.
CONCLUSION
In this paper, we introduce IFeF-PINN, a novel iterative training method for Fourier-enhanced Fea- PINNs. By augmenting the network with Random Fourier Features mapping basis exten- sion with the bi-level problem, IFeF-PINN mitigates the spectral bias problem of standard PINNs when capturing the high- frehffryy and multi-scale components during training. Experimental sults demonstrate that IFeF-PINN consistently outperforms advanced baselines across various sce- narios, including popular low-frequency benchmarks and handling high-frequency and multi-scale PDEs. Furthermore, it has strong flexibility when integrating with different training strategies for PINNs.
Despite its strengths, IFeF-PINN faces challenges when extended to nonlinear PDEs. For nonlinear PDEs, the lower-level problem becomes nonconvex, precluding a one-step solve and requiring itera- tive two-stage gradient descent updates that can stall in local minima. Advancing principled bi-level optimization techniques to better handle the nonlinear lower-level problem remains a promising direction for future work.
10

---

Published as a conference paper at ICLR 2026
8 ACKNOWLEDGEMENT
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Pro- gram (WASP). funded by the Knut and Alice Wallenberg Foundation. It was further supported by the Swedish Research Council through the Distinguished Professor Grant 2017-01078, as well as by Wallenberg Scholar Grant from the Knut and Alice Wallenberg Foundation. The authors also gratefully acknowledge the support of Digital Futures.
REFERENCES
Jonathan F Bard. Some properties of the bilevel programming problem. Journal of optimization theory and applications, 68(2):371-378, 1991.
Matthieu Barreau and Haoming Shen. A control perspective on training PINNs. arXiv preprint arXiv:2501. 18582, 2025.
John Boyd. Chebyshev and Fourier spectral methods. Courier Corporation, 2001.
Xintao Chai, Wenjun Cao, Jianhui Li, Hang Long, and Xiaodong Sun. Overcoming the spectral bias problem of physics-informed neural networks in solving the frequency-domain acoustic wave equation. IEEE Transactions Geoscience and Remote Sensing, 2024.
Francisco Sahli Costabal, Simone Pezzuto, and Paris Perdikaris. 8-pinns: ral networks complex geometries. Engineering Applications of Artificial Intelligence, 127: 107324, 2024.
Physics-informed neu-
Vikas Dwivedi and Balaji Srinivasan. Physics informed extreme learning machine (pielm)-a rapid method for the numerical solution of partial differential equations. Neurocomputing, 391:96-118, 2020.
Katayoun Eshkofti and Matthieu Barreau. Vanishing stacked-residual pinn for state reconstruction of hyperbolic systems. IEEE Control Systems Letters, 2025.
Michel Goemans and David P Williamson. The primal-dual method for approximation algorithms and its application to network design problems. Approximation algorithms for NP-hard problems, pp. 144-191, 1997.
Amanda Howard, Sarah H. Murphy, and Stacked networks improve physics-informed training: Applications to neural networks and deep operator networks. Foundations of Data Science, 2025.
Zheyuan Hu, Khemraj Shukla, George Em Karniadakis, and Kenji Kawaguchi. Tackling the curse of dimensionality with physics-informed neural networks. Neural Networks, 176: 106369, 2024.
Arieh Iserles and Syvert Nørsett. Efficient quadrature of highly oscillatory integrals using deriva- tives. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 461 (2057):1383-133 1399, 2005.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Carl Jidling, Niklas Wahlström, Adrian Wills, and Thomas B Schön. Linearly constrained gaussian processes. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL https: http::///ooccceedeeeeeeee new.iieeiia.. cc/paper folapprffllee//// sapprr22017/ilee aldll http://cccccccecec11111440222210
Namgyu Kang, Jaemin Oh, Youngjoon Hong, and Eunbyung Park. Pig: Physics-informat gaussians as adaptive parametric mesh representations. arXiv preprint arXiv:2412.05994 2024.
G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physics-informed machine learning. Nature Reviews Physics, 2021.

---

Published as a conference paper at ICLR 2026
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics-informed neural networks. Advances in neural infor- mation processing systems, 34:26548- 2021.
Gregory Kang Ruey Lau, Apivich Hemachandra, See-Kiong Ng, and Bryan Kian Hsiang Low. PIN- NACLE: PINN adaptive collocation and experimental points selection. In The Twelfth Interna- tional Conference on Learming Representations, 2024. URL https://openrevieww
forum? Sommmiddeen Le=piddaaGVeeeeeg
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Arizzadenesheli, Burigede liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL
.eet (egerummidd-cPPPPNeeemmm http:://.ppparereeeeeeeeeeeerrra
Chensen Lin, Zhen Li, Lu Lu, Shengze Cai, Martin Maxey, and George Em Karniadakis. Operator learning for predicting multiscale bubble growth dynamics. The Journal of Chemical Physics, 154(10), 2021.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503-555 1989.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based the universal approximation theorem of operators. Nature machine intelligence, 3(3):218-229 2021.
Michael D McKay, Richard J Beckman, and William Conover. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 42(1))55-61, 2000.
Ben Moseley, Andrew Markham, and Tarje Nessen-Meyer. Finite basis physics-informed neural networks (fbpinns): scalable domain decomposition approach for solving differential equations. Advances in Computational Mathematics, 49(4)662, 2023.
Abdul Hannan Mustajab, Hao Lyu, Zarghaam Rizvi, and Frank Wuttke. Physics-ing Physics-informed neural networks for high-frequency and multi-scale problems using transfer learning. Applied Sciences, 14(8):3204, 2024.
Shaoxiang Qin, Fuyuan Lyu, Wenhui Peng, Dingyang Geng, Ju Wang, Naiping Gao, Xue Liu, and Liangzhu Leon Wang. Toward a better understanding of fourier neural operators: Analysis and improvement from spectral perspective. CORR, 2024.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning. pp. 5301-5310. PMLR, 2019.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations, 2017. URL https: //waxiv.org/abs///...
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential
equations. Journal of computational physics, 2018.
Zhenao Song. Rl-pinns: Reinforcement learning-driie driven adaptive sampling for efficient training of pinns. arXiv preprint 2ete22504..9999 2025.
12

---

Published as a conference paper at ICLR 2026
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kar Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Kejun Tang, Jiayu Zhai, Xiaoliang Wan, and Chao Yang. Adversarial adaptive sampling: Unify PINN and optimal transport for the approximation of PDEs. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: pettps://openreviiee 7Qr7tVrh2c.
cort/forum?dd
Umair Bin Waheed, Kronecker neural networks overcome spectral bias for pinn-based wavefield computation. IEEE Geoscience and Remote Sensing Letters, 19:1-5, 2022.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow patholo- gies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3055- A3081, 2021a.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature net- works: From regression to solving multi-scale pdes with physics-informed neural networks. Com- puter Methods Applied Mechanics and Engineering, 384:113938, 2021b.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:1 10768, 2022.
Yongji Wang and Ching- Yao Lai. Multi-stage neural networks: Function approximator of machine precision. Journal of Computational Physics, 504:1 12865, 2024.
Ching-Yao
E Weinan and Bjor Engquist. The heterognous multiscale methods. Communications in Mathe- matical Sciences, 1(1)877-132, 2003.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Zhi-Qin John Xu, Lulu Zhang, and Wei Cai. On understanding and overcoming spectral biases of deep neural network learning methods for solving pdes. Journal of Computational Physics, pp. 113905, 2025.
Shin Yeonjong, Darbon Jérôme, and Em Karniadakis, George. On the convergence of physics- informed neural networks for linear second-order elliptic and parabolic type pdes. Communica- tions in Computational Physics, 28(5):042-2074, 2020.
Zhiyuan Zhao, Xucying Ding, and B Aditya Prakash. Pinnsformer: A transformer-based framework for physics-iformedd neural networks. arXiv preprint a //83, 2023.
13


Reviews:
[{"review_id": "DiHnN3mqOW", "reviewer": "Reviewer_xJdr", "summary": "The paper introduces an iterative training method (IFeF-PINN) that mitigates the spectral bias of Physics-Informed Neural Networks, favoring low-frequency solutions. It separates training into two alternating stages: learning a latent basis through a standard PINN, then extending it with Random Fourier Features (RFF) to capture high-frequency components, followed by a convex regression step. The authors prove convergence for linear PDEs and show that the RFF extension expands the network’s expressive power. Experiments on benchmark PDEs (Helmholtz, convection, Burgers, etc.) demonstrate that IFeF-PINN consistently outperforms existing PINN variants, achieving accurate high-frequency solutions and reduced spectral bias.", "questions": "How stable is the iterative bi-level optimization when the Random Fourier Feature dimension D becomes large or when the feature distribution is poorly scaled? Does the alternating update between basis learning and convex regression amplify noise or cause overfitting to spurious high-frequency components?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper excels in its clear, well-motivated algorithmic innovation and solid theoretical–empirical balance. \n\nFirst, it provides a principled bi-level reformulation of PINN training that explicitly decouples feature learning from coefficient regression. This structure not only clarifies optimization dynamics but also leads to provable convexity and convergence for linear PDEs, a strong theoretical contribution. \n\nSecond, the integration of Random Fourier Features into the latent space is clever and effective. It directly targets spectral bias instead of relying on heuristics like adaptive weights or resampling. \n\nThird, the paper delivers rigorous ablation and benchmark comparisons (across Helmholtz, convection, Burgers equations) showing that IFeF-PINN achieves one to two orders of magnitude lower relative L2 error than strong baselines, especially under high-frequency and multi-scale regimes where PINNs typically collapse. \n\nFinally, the frequency-domain analysis using FFT provides quantitative evidence that the model indeed learns higher frequencies, reinforcing its claims with interpretable diagnostics.", "weaknesses": "A clear weakness of the paper is its limited treatment of nonlinear PDEs, where the proposed convex lower-level formulation breaks down. While the authors acknowledge that the lower regression problem becomes nonconvex in such cases (and suggest it to be promising direction), they did not offer quick remedies or convergence guarantee beyond gradient descent heuristics. This has limited its applicability as many practical PDEs of interest, such as fluid dynamics, reaction–diffusion systems, nonlinear elasticity, are inherently nonlinear. The experiments also mostly focus on linear or mildly nonlinear cases (e.g., Burgers’ equation), leaving open whether the method’s strong performance extends to more challenging nonlinear systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gc4DqDKBvk", "reviewer": "Reviewer_Coic", "summary": "The paper addresses the challenge of spectral bias in physics-informed neural networks (PINNs) by introducing an iterative training framework that utilizes random Fourier features (RFFs). The approach is based on two types of bases, nominal and extended, and applies a regression method on the extended basis coefficients to learn the PDE solution. The method is evaluated on canonical PDE problems. The paper claims that it is potentially generically applicable to different neural PDE solvers. However, that is not shown in the current version. Some aspects of the paper are unclear, for instance, the current write-up, particularly the detailed construction of the basis, and the motivation for the chosen network layer for RFF extension. Also, the empirical demonstration on nonlinear and highly oscillatory PDEs is a limitation.", "questions": "1. Could the authors clarify in detail how the nominal and extended bases are constructed and why the RFF extension is applied specifically at the last layer?\n\n2. What are the computational costs and convergence characteristics of the proposed method compared to [1] and [2]? How intensive is the iterative training in practice?\n\n3. How is the initialization of both upper- and lower-level optimizations handled, and how sensitive is the performance to this initialization?\n\n4. Can the authors include results on nonlinear PDEs (e.g., the KdV or Kuramoto–Sivashinsky equations) to demonstrate the method's generality beyond linear problems?\n\n5. Would an ablation on hyperparameter choices (especially for the physics-informed Gaussians and the proposed method) help in understanding performance sensitivity?\n\n6. If the approach primarily targets linear problems, could the authors explicitly position it in relation to [1] and [2], and clarify this limitation in the discussion?\n\nMinor comment:\n\nThe formula for computing the relative L2 error is not stated in the paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper addresses a well-known and important problem of spectral bias in PINNs.\n\n2. The proposed approach is conceptually general and can be integrated with various PINN frameworks.\n\n3. Provides theoretical convergence analysis and demonstrates improvements on canonical problems.", "weaknesses": "1. The paper is unclear in how the nominal and extended bases are created or selected, and lacks an explanation for the motivation behind modifying the last layer to improve spectral properties. The presentation of the methodology could be improved.\n\n2. The discussion of the bi-level optimization framework is limited, and different possible formulations and the rationale for the chosen structure are not justified.\n\n3. Computational aspects, such as training cost, convergence behavior, and initialization of the bi-level optimization, are not adequately discussed and compared with prior works.\n\n4. The numerical experiments primarily focus on linear PDEs; the method’s effectiveness for nonlinear oscillatory problems is not demonstrated.\n\n5. A comparison with related approaches, such as [1] and [2], is lacking, and no clear discussion is provided on computational trade-offs or advantages over these methods.\n\n[1] De Ryck, Tim, et al. \"An operator preconditioning perspective on training in physics-informed machine learning.\" The Twelfth International Conference on Learning Representations.\n\n[2] Moseley, Ben, Andrew Markham, and Tarje Nissen-Meyer. \"Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equations.\" Advances in Computational Mathematics 49.4 (2023): 62.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9by100kT2m", "reviewer": "Reviewer_JZcR", "summary": "To address the spectral bias of PINNs, IFeF-PINN is proposed. It is an algorithm for iterative training of PINNs with Fourier-enhanced features, enriching the latent space using high-frequency components through Random Fourier Features. This creates a two-stage training problem: (i) estimate a feature basis, and (ii) perform regression to determine the coefficients. For an underlying linear model, it is shown that the latter problem is convex and that the iterative training scheme converges. Numerical evaluation on classical benchmark problems shows the superior performance of IFeF-PINN over SOTA algorithms is shown.", "questions": "- Can the proposition 1 be extended to nonlinear problems?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* (Clarity) The paper is easy to follow. Contributions are well clarified and compared with other studies addressing the spectral bias.\n* (Technical contribution) The proposed methods effectively combines PINNs and RFFs in a novel way, addressing the limited approximation ability of RFFs in low dimensions.\n* (Broader impact & Technical contribution) An application to general neural architectures is straightforward, with some careful treatments described in Remark 2 in Section 2.\n* (Theoretical contribution) Uniqueness & existence of the solution of the lower-level problem (Proposition 1), Lipchitz continuity of the solution map (Proposition 2), convergence to a stationary point (Theorem 1), and the universal approximation (Theorem 2 & Corollary 1) are proved, ensuring theoretical soundness of the proposed method.\n* (Empirical contribution) The proposed method outperforms baselines.\n- (Reproducibility) Error bars are provided.", "weaknesses": "- Although the theoretical convergence is proved, the proposed method requires a bilevel optimization, which may cause training instability, necessitating warm-start training outlined in Section 3 (please correct me if I missed singing); that said, pretraining or warm-start is often required when training PINN's variants.\n- Currently, the code is not available, although it will be released upon acceptance (line 353). Could the authors provide the code? Just a snippet is acceptable.\n\n\n## Review summary\n\nThe paper is well-written and easy to follow. The contribution of the proposed method is clear. I am currently inclined to recommend its acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cQD48XR1Ru", "reviewer": "Reviewer_F9RY", "summary": "This paper proposes IFeF-PINN, an algorithm for iterative training of PINNs with Fourier-enhanced features. The core of IFeF-PINN is to enrich the latent space using high-frequency components through Random Fourier Features (RFF).  This work provides detailed theoretical derivations for several key conclusions. Experiments also show that IFeF-PINN outperforms other SOTA methods.", "questions": "Please refer to the weaknesses. The reviewer would consider raising the score if the first two weaknesses are adequately addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n- The claims of this paper are substantiated by multiple theoretical derivations.\n- The effectiveness of IFeF-PINN is validated on various problems.", "weaknesses": "- The two-stage training process is a key claim of this work, but it lacks sufficient justification and experimental evidence to demonstrate its necessity. The reviewer acknowledges the importance of decoupled representation and downstream tasks (e.g., large-scale pretraining). However, as the significance of this pipeline in the scope of PINNs is not yet established, the authors should provide a more thorough discussion. An ablation study comparing against an end-to-end training approach using RFF basis extension would help clarify this issue.\n- The paper lacks comparison with a basic baseline: the method described in Equation 4. The reviewer disagrees with the authors' claim in Line 155 (\"when the input is of low dimension, such as for PINNs, this could limit its asymptotic approximation capabilities\"). The method in Equation 4 has been successfully validated in NeRF (NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis), which also utilizes low-dimensional inputs (3D location and 2D viewing direction).\n- IFeF-PINN involves multiple training phases, potentially requiring more training steps. The authors mention in Line 344 that baseline methods are trained/evaluated with their default settings. The reviewer questions the fairness of comparisons and suggests that ensuring all methods use the same total number of training steps would provide a more equitable evaluation.\n- The impact of the number of Fourier-enhanced features (D) on performance is unclear. Is there a correlation between the value of D and downstream task performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)