Fourier Learning Machines: Nonharmonic Fourier-Based Neural Networks for Scientific Machine Learning
Authors: Mominul Rubel
Adam Meyers
Gabriel Nicolosi
Venue: Accepted by TMLR ()

Abstract: We introduce the Fourier Learning Machine (FLM), a neural network (NN) architecture designed to represent a multidimensional nonharmonic Fourier series. The FLM uses a simple feedforward structure with cosine activation functions to learn the frequencies, amplitudes, and phase shifts of the series as trainable parameters. This design allows the model to create a problem--specific spectral basis adaptable to both periodic and nonperiodic functions. Unlike previous Fourier--inspired NN models, the FLM is the first architecture able to represent a multidimensional Fourier series with a complete set of basis functions in separable form, doing so by using a standard Multilayer Perceptron--like architecture. A one--to--one correspondence between the Fourier coefficients and amplitudes and phase-shifts is demonstrated, allowing for the translation between a full, separable basis form and the cosine phase--shifted one. Additionally, we evaluate the performance of FLMs on several scientific computing problems, including benchmark Partial Differential Equations (PDEs) and a family of Optimal Control Problems (OCPs). Computational experiments show that the performance of FLMs is comparable, and often superior, to that of established architectures like SIREN and vanilla feedforward NNs.

Published in Transactions on Machine Learning Research (12/2025)
Fourier Learning Machines: Nonharmonic Fourier-Based Neural Networks for Scientific Machine Learning
Mominul Rubel" Department of Engineering Monagement and Systems Engineering Missouri University of Science and Technology, Rolla, MO
Adam Meyers Deportment Industrial and Systems Engineering University of Miamii Coral Gables, FL
Gabriel Nicolosi Department of Engineering Management and Systems Engineering Missouri University of Science and Technology, Rolla, MO
Reviewed on OpenReview: https: // op////veeweeww...m net/ formm? LPKt5vddy
Abstract
We introduce the Fourier Learning Machine (FLM), a neural network (NN) architecture de- signed to represent multidimensional nonharmonic Fourier series. The FLM uses a simple feedforward structure with cosine activation functions learn the frequencies, amplitudes, and phase shifts of the series trainable parameters. This design allows the model to cre- ate problem-specifii spectral basis adaptable to both periodic and nonperiodic functions. Unlike previous Fourier-inspired NN models, the FLM is the first architecture able to rep- resent a multidimensional Fourier series with a complete set of basis functions in separable form, doing so by using a standard Multilayer Perceptron- like architecture. A one to one correspondence between the Fourier coefficients and amplitudes and phaso-shifts is demon- strated, allowing for the translation between a full, separable basis form and the cosine shifted
Additionally, evaluate the performance of FLMs several scien- tific computing problems, including benchmark Partial Differential Equations (PDEs) and a family of Optimal Control Problems (OCPs). Computational experiments show that the performance of FLMs comparable, and often superior, to that of established architectures like STREN and vanilla feedforward NNs.
1 Introduction
Approximating functions as a composition or weighted sum of simpler basis elements is a fundamental approach in both classical analysis and modern machine learning. Two typical examples of this idea are the Fourier series and artificial neural networks (NNs). The Fourier series decomposes a periodic function (or, for nonperiodic functions, their periodic extension) into linear combination of fixed and predetermined set of sinusoidal (sines and cosines) basis functions and gives an interpretable spectral representation of it. While nonperiodic functions can also be handled by defining a periodic extension over a finite interval, this often introduces discontinuitics at the boundarics and lcads to oscillatory behavior, known as the Gibbs phenomenon. Moreover, these Fourier bases are fixed and not adaptable to the specifie features of the data. NNs, on the other hand, are computational models that approximate any function through compositions of parametrized nonlinear transformations of linear combinations of the data with (trainable) weights and biases. These parameters composing NN are learned and adapted to the data during training (the process of optimizing the network parameters) using nonlinear optimization techniques. However, these weights and biases are numbers with little interpretation, and the network itself works as a black box. Despite these
"These authors contributed equally.
i "Correpponding author. Email: gabrielnicalosii@ett.ee

---

Published in Transactions on Machine Learning Research (12/2025)
differences, both approaches share the basic underlying principle of building complex functions from simpler components.
Motivated by Joseph Fourier's early 19th century idea that any arbitrary function, regardless of its complexity or continuity, could be represented as a sum of sines and cosines, and by the successful application of NNs as universal function approximators since the late 20th century, we propose in this paper a new Fourier-inspired NN model, and we shall name the Fourier Learning Machine (FLM). Unlike typical black box NN models, the FLM's internal structure is built to directly represent an m m-dimensional nonharmonic Fourier series. It achieves this by relying solely the foundational components of classical multilayer perceptron (MLP); namely, affine transformations followed by nonlinear activation functions (in the context of this paper, these being pure cosines). This design choice eschews more complicated architectural features, such as multiplica- tion nodes, recurrent relations or any other customized components, previously found in predecessor models. With hardwired integer weights in its poput-to-hiddde layer, the FLM represents an m-dimeesionall, full- basis classical (finite) Fourier series as a special case. In order to tackle the approximation of nonperiodic functions, we relax the integrality constraints of the frequency components, thus configuring the frequencies as learnable parameters of the FLM. This allows for the search for the sinusoidal basis functions that best reprosent the target map. While maintaining interpretability, FLMs build function-specific spectral repre- sentation for a wide variety of functions, including nonperiodic ones. As such, our proposed model combines the interpretability of Fourier series with the driven nature of NNs (and, in particular, the simplic- ity of the MLP architecture), effectively representing a nonharmonic Fourier series (Young) 1981; Duffin & Schaeffer 1952) in which the basis functions are learned and adapted to the target function it is meant approximate.
The challenge of representing Fourier series with traditional MLP-like NNs can be understood by first considering a one-dimensional Fourier series on the interval [- (without loss of generality) in the cosine phase-shile shifted form, i.e., f(t) = ao +
N
Σ ==
An cos(nt on),
where An and on denote the amplitude and the phase shift for the th frequency component, respectively. For such a form of the Fourier series, it is straightforward to see that the approximation ʃ can be naturally represented by a single layer NN with cosine activation functions. Nonetheless, generalizing this idea to higher dimensions requires extra care since, as demonstrate in this paper, the NN structure then either departs too far from the MLP-like like archilecture or fails to represent actual, truncated nonharmonic Fourier series. As a result, certain degree of architectural ingenuity is required so that the model retains both a simple architecture and the integrity of the nonharmonic Fourier representation in higher dimensions.
In this paper, we show that the proposed FLM does not only combine an interpretable structure with adaptable bases, but also achieves approximation accuracy comparable to, and in some cases exceeding, that of established NN architectures. Although we focus on using FLMs to solve Partial Differential Equations (PDEs) and an Optimal Control Problem (OCP), this model can be applied to a wide range of problems, such as signal and image reconstruction, spectral analysis, and surrogate modeling for high-dinensiona systems, where both interpretability and adaptability are desired. The main contributions of this paper are:
1. We provide comprehensive review of all previous attempts to construct such a function approxima- tor, typically referred in the literature as "Fourier Neural Networks", departing from foundational works (Gallant & White, 1988) and covering up to recent architectures with trigonometric acti- vation functions (Sitzmann et al., 2020). Within this context, we demonstrate that the FLM is not only architecturally simpler than previous models but also computationally competitive, often outperforming established, traditional NN models.
2. By defining the m Lexi Sign Matrix, we show how the FLM, leedforward NN, can simultaneously represent a full basis finite Fourier series in m dimensions in both separable and cosine phase shifted forms, with a one-to-one correspondence between Fourier coefficients, frequencies and phaso-shifi and the NN weights and biases. To the best of the author's knowledge, this ficxibility to represent fall-basis classical Fourier series in ML.P-like architecture, is unique in the literature. In the
2

---

a monotonic and Published in Transactions on Machine Learning Research (12/2025)
nonharmonic case, it is shown that such representation feature is directly translated to more computationally efficient parameter optimization when compared to other traditional and Fourier- based NN models.
3. Through series of carefully designed computational experiments, empirically demonstrate the FLM's approximation capabilities for solving PDEs and optimization problems. In particular, we apply the FLM to the numerical solution of a multidimensional OCP arising from the evolutionary dynamics of a Rock Paper- Scissors circulan))) game, demonstrating its capability to solve complex, nonlinear problems.
The remainder of the paper is organized as follows: Section 2 reviews related literature and prior work on Fourier-related NNs. Section 3 introduces a few mathematical preliminaries and introduces the inherent challenges of representing a (nonharmonic) Fourier series with a MT.P-like NN architecture. Section [ constructs, formalizes and presents the proposed FLM architecture in detail. Dection&rrpone computational experiments and compares the FLM's performance with established NN architoctures. Finally, Scction concludes the paper and discusses potential future research directions.
2 Related Literature
The idea of a Fourier series inspired NN was first proposed in the late 1980s, just before NNs were shown to possess universal function approximation properties (Hornik et al. 1989). This pioneer architecture, called "Fourier Network" (FN) was proposed by Gallant & White (1988). In their work, they carefully constructed a Fourier series representation into NN by defining "cosine "cosine-squarl squases-quurrr activation function, a domain constrained, monotonic, cosine function. By fixing the input to hidden layer weights, this NN was demonstrated to mimic a finite multidimensional Fourier series. This ccosine-squsslee piecewise differentiable function, F TK + [0, 1], defined as
if net < -
F(net)= = [coss net + 3T + /2
2 S net S I
(1)
I net < Do.
While F(-) is not strictly periodic, the FN constructs the Fourier cosine bases by constraining and summing multiple hidden units within a group of neurons called "clumps". This activation function designed
fit the "squashing" paradigm of early NNs (Rumelhart al.) 1986) while allowing for the construction of
NN replicating the approximation properties of Fourier series. A single "clump" is depicted in Fig, [ As such, the output of each of these "clumps" forms one of the cosine or sine basis (by adding phase- shift) present in a Fourier series. To create a single cosine basis, the weighted sum of the inputs, i.e., the pre-activation net, = w. x + b; is preprocessod and activated using the cosine-squasher function. By constraining Sim- = 8'm = 20,, where = 2,..... K and m "e.....,,...... MM, the output of the FN is given by,
M
[20, F(-net; + m2x) 20, F(net; 2 m2r)
K Σ 00: 0, 2,2 1)) 1=1
Og cos(nels))
-
which is a classical Fourier series in cosine phase-shifted form when an extra term do = On + Σ [0 - 0, 22M + 1)] is added to the output. The role M, is to bound the range within which the output net, can vary. As the input æ varies in the (perturbed) hypereube [c,2x - dm", net, varies in the interval [-, 2n (M, + 1)]. Therefore, M, is inherently dependent on the "hardwired" parameters w; and by in the first. layer. Furthermore, M, increases with i, that is, the inclusion of higher frequency modes
3

---

Published in Transactions on Machine Learning Research (12/2025)
2n(-MM)
J1
nod, 24| M4,
P1
-ntt++ 2 2mr/A4 13
1ee W. x
Σ
4g-M)
-xad - Ir(-MM)
mnt, Jx 2n( (4--4,
2e1M
F11)
-nsl 24(26)
Figure 1: A "clump" of neurons that outputs the 8t single cosine basis of the Fourier Network appearing in Gallant & While (1988).
requires more neurons within "clump". For smaller network sizes, while the authors claim that by undoing the "hardwiring" (in w, and b:) and removing the constraint in Çim and 6im the FN could potentially perform better, doing so nullifies its exact correspondence with a finite Fourier series: making w; and b, trainable implies that one could not determine a specific value for each M; before training the FN. Thus, is unclear how to pick M, under the aforementioned construction. Lastly, by construction, removing the constraint fim = 6:m- 20; would hinder the 7th "clump" create a cosine basis. Therefore, such an attempt would not lead the network to represent nonharmonic Fourier series.
It is worth noticing that the requirement of "monotonic" activation function later dropped based the universal approximation (and convergence) result in Loshno et al. (1993), which established that non- polynomial and bounded functions (e.g., cosine) can be also serve as activation functions in NNs. Therefore, such an intricate design proposed in Gallant & White 1988) can be partially justified by the lack of such a result when their architecture was developed.
In the context of systems control and identification, Zhu Paul (9955 proposed the first NN capable of mimicking finite Fourier series by using pure sines and cosines, naming it the "Fourier Series Neural Network" (FSNN). Despite their NN capability to represent full-basis Fourier series, given fixed set of nontrainable "frequency weights", the architecture chosen relies on multiplicative nodes to compose each of the basis functions. The training process in this case is rather simple, given the linearity (with respect to the Fouricr cocfficients) of the output layer, thus yielding a lincar optimization problem. More recently, Benrabah et al. (2021) introduced an adaptive controller where a FSNN is used to compute and adjust the gains of PID controller online.
With emphasis similarities of its model with the brain neurophysiology, Silvescu 1999 proposed architecture relying multiplicative nodes that allempled to reproduce a finite Fourier series in cosine phase-shifted shifted form (optionally "squeezed" through sigmoid function). This network, named "Fourier Neural Network" (the first appearance of such a term in the literature) assumed the form
f(z f( Im) =
Σ ci II
cos(uujz + pp)))
(2)
j=1
However, it is implicit in Silvescu (1999) that the trainable parameters w and are not fixed constrained I to be either natural numbers (including zero) or odd multile of 2: respectively. Therefore, despite its similarities with a Fourier series, f()) fails to represent either a finite, classical Fourier series or a (typical) nonharmonic one. In its current form, a classical Fourier series could be attained from Equation 2 by fixing w's to a predetermined set of integers and making some ip E {(2k+ + 1) k Z}. Conversely, a nonharmonic series only be obtained in the form of sum of cosine-only basis functions, which would configure very
4

---

Published in Transactions on Machine Learning Research (12/2025)
specific (and arguably weak) form of nonharmonic Fourier series, allowing one to approximate even functions only. For depiction of this FN model, the reader is pointed to [Uteuliyeva et al. (2020).
In Yang et al. (2003), a single input, multiple output NN is proposed, wherein each output corresponds to that of an individual finite Fourier series. In this architecture, all the series share the same basis functions but each have their own set of (trainable) Fourier coefficients. Besides being of limited applicability (once it is design to approximate single variable functions only), Yang al. (2003) also fails to make any reference to previously proposed, Fourier-inspired NN already in existence in the literature at that time.
Halawa (2008) uses Discrete Fourier Transform to find the hidden-to-outpiit weights for the FSNN model from Zhu & Paul 1995). This neural network formulation, however, is unnecessary for the task, as the calculation provides little advantage over a direct computation performed outside of an NN framework.
In Liu (013)) another type of FNN proposed, this version utilizing a vector form of the multidimenniinaal, finite Fourier series. This NN assumes the form
N f(z)
Σ
a, sin (cf + b, cos d x))
(3)
where, the (real valued) numbers o₁ di, ag and b, are the learnable parameters of the corresponding NN. Unless the frequeney components are constrained to be equal position-wise, i.e., C; = d, this representation is a trigonometric expansion, but it is not strictly a nonharmonic Fourier series. This is because the use of different frequency vectors inside the sine and cosine prevents Equation 3 to be written its equivalent exponential or separable basis forms. Therefore, in its original form presented, Equation 3 represents neither a classical Fourier series nor a nonharmonic one.
A comparative study on the performance of the FN model of Gallant & White 1988), the FNN of Silvescu (1999) and the FNN of Liu (2013) is presented in Uleuliyeva et al. (2020), which concluded that the top performing model among these three is the early FN model of Gallant & White (1988). However, this FN model of Gallant & White (1988) was shown to underperform vanilla feedforward NN with sigmoid activation function. The reason behind this result, as hypothesized by the authors, is that, outside the interval [- x T. the "cosine- squasher" as written in Equation is constant, whereas the sigmoid employed in the vanilla feedforward NN is asymptotically constant outside of this same interval.
A comprehensive and critical investigation on the use of sine functions as activation functions in NNs has appeared in Parascandolo et al.] (2017). In this work, the authors claim that the wave-like behavior introduced in the loss function by the presence of sines introduces many local minima, making such models harder to train. However, the output of the NNs chosen for the experiments shown in Parascandolo el al.] (2017) are not meant to represent a Fourier series in any form. Furthermore, the performance of such activation function is not studied for a simple feedforward NN structure, but in the context of encoder- decoder architectures based on recurrent and long short term memory NNs. Related to this idea, Oznermer (2020) designs a "Sinusoidal Neural Network (SNN)", with a newly defined sine-based node operations inspired by information propagation in the brain. Even though empirical results presented are promising, no thoorctical background is provided to guarantce the convergence of this NN.
A (deep) NN with sine activation functions, called STREN (Sinusoidal Representation Network) is introduced in Sitzmann et al. (2020), tailored for applications in digital media processing. The authors customize the first layer of this NN in order to extract figh-frequency components of the target data (image, video or audio), doing so by multiplying the weighted sum of the input by a relatively large constant term wo (keeping the bias untouched), while all the other activation functions assume the form y + sin y. We note that a SIREN-1, that is, a SIREN with one single layer, for wi- = 1 can be shown, under certain trigonometric identities, to assume the mathematical form of
N
f(x)
Σ
a, sin (w.) x)) + be cos (w.. x)
(4)
Furthermore, we note that Equation A identical to Equation 3 for when c, = dj Thus, similar to the case of Liu 2013 a SIREN-1 represents a. Fourier series by fixing the components of integers (noted only
5

---

Published in Transactions on Machine Learning Research (12/2025)
later on by Benbarka et al.
(022)) ) and a nonharmonic series otherwise. It is worth noting that in Sitzmann et al. (2020) there is no mention to any potential relations between a SIREN NN and any form of a Fourier series. However, as it is also the case of the FNN of Liu (2013), this Fourier series cannot be represented in
a full-basis, separable form.
In the context of scientific machine learning, Ngom & Marin (2021) introduced a one-dimensional FNN as function approximators. Their FNN model assumes the form of a nonharmonic (phase shifted cosine) Fourier scries (with trainable frequency components). Nonetheless, all the numerical experiments reported considered only the approximation of periodic functions that can be fully decomposed as finite Fourier scries, thus not fully exploring the potential of their model in the approximation of nonperiodic functions.
In order to avoid confusion by the use of similar lerms and concepls, other scientific contributions that somehow involve the use Fourier analysis (but do not attempt to construct a NN that replicates a Fourier series) must be mentioned in order to avoid confusion. Rocently, the Fourier Neural Operator (FNO) gained tremendous altention in the scientific machine learning community (Li et al., 2020). The FNO primary goal is to approximate operator, that is, a mapping between function spaces. Thus, it is suitable to learn operators in the context of approximating the solution of entire parametric families of PDES. In this regard, the "Fourier" comes from the fact that the integral kernel is parameterized in Fourier space. The FLM model presented in this paper, instead, aims to approximate a mapping from a higher-dimensee dimension Fuclidean space to a real number. In Tancik et al. (2020), preprocessing imput vectors with a Fourier feature mapping allows this NN to overcome its inherent spectral bias, i.e., the tendency to learn low frequency components and therefore facilitating learning of high frequency functions. Lastly, by automatically determining the optimal sampling frequency and number of neurons, Fourier-basee extreme learning machines are applied in Ma et al. (022) for the solution of PDEs.
Some of the models described above move toward learnable spectral bases, while others are constrained by the integrality of their frequency components. Nonetheless, in either case, they often lack a systematic architecture allowing them to represent a complete, separable, and interpretable Fourier series of the target function (either classical or nonharmonic). Even though STREN-1 and the FNN in Ngom & Marin (2021) fulfill the aforementioned criteria for a single input, these properties don't hold true when considering the general, multidimensional case. As we will show later in this paper, FLMs fill this gap, being the first fourier-replicate NN that is capable of representing both finite and nonharmonic Fourier series in full, separable basis form in m dimensions by using only ALL--liee feedforward NN operations.
Lastly, even though explicitly outside of the NN domain (at least in the sense in which NNs are used in this work, i.e., for deterministic function approximation). it is worth mentioning that FLMs share a fundamental similarity with Sparse Spectral Gaussian Processes (SSGP), introduced in Lázaro-Gredilla
al. (2010)
This relationship draws from established results bridging NNs and Gaussian Processes (GPs) (Neal (1966)
Murphy (2023)). Both FLMs and SSGP rely learning frequencies of their underlying trigonometric basis functions. However, FLMSs rely on the usual gradient descent in the space of frequencies (and amplitudes) to find the set of frequencies that "best" describe the target function. As it will be shown in Section 5 this is not necessarily a data-driven approach, but instead based on the customized loss function that enforces the desired system dynamics. On the other hand, in SSGP, the frequencies are treated as hyperparameters and optimized by maximizing the marginal likelihood (thus data-driven) to approximate the spectral density of the covariance kernel. Therefore, SSGP falls into the framework of (efficient) probabilistic regression for large-scale, discrete data, not undermining the FLM's spectral adaptability, but rather reinforcing its importance and applicability.
Next, we present few mathematical preliminaries before introducing the proposed FLM architecture. Un- less the reader is vels-versed in Fourier analysis, reading the next section is strictly necessary in order to understand the architecture introduced later in Section
3 Mathematical Preliminaries
In this section, we establish the mathematical foundation for our approach, using the classical Fourier series departing point. Our goal is to generalize this representation by treating the frequencies as arbitrary
6

---

Published in Transactions on Machine Learning Research (12/2025)
veal-valued parameters, instead of fixed integer multiples of the fundamental frequency. In doing so, we transform the classical Fourier series into nonharmonic one.
Let f -- + R be a square integrable function and JH be the Fourier series expansion of its 2n periodic extension in each variable. We consider the domain [- =| x| for simplicity, however, this can be generalized to any domain [-h,, [n]x[-I1p T2]] x...... [- T.m: [m]] by scaling the input variables accordingly. The Fourier series of f in complex exponential form is given by
In(Ø)= = E
On 2
mEZ.
where a (x1, (#1, ....... 3m) E [-x, | n = (n1, .2,...,, Bm)) E 2 and a represents their dot product. The analytical expression for calculating the Fourier coefficients is
1 || f(x) e in= dx.
Cn (m
although in this paper we are interested in approximating them numerically, Alternatively, the same series Iu can be expressed using real separable basis functions (products of the fundamental sine and cosine functions)
(1( = Σ (a(n) cos(n101) cos(n222). ((s(m)2)))s((27))) cos(Am
MEN
(n) cos((1111 cos(n22)) ... sin +
(6)
agm sin(nn sin(n)22)) ...
for input = (I1, )m) E T} and frequency components n = (n1, 72,..... nm) E No = {0, 10,,,,.... This trigonometrie separable form has the advantage that the set of basis functions, associated with n, simultancously accounts for all possible 2n symmetric frequency components. Here, symmetric components refer to the sign-flipped versions of a frequency vector, i.e., +,,,,,-22.. tnm), which are always grouped together in the spectrum of real valued function. As a result, redundant terms that would otherwise appear in the complex exponential form in Equation 5 are implicitly bundled together, allowing real-valued functions to be expressed more efficiently while still spanning the complete set of available basis functions.
The classical Fourier series Equation and Equation 5 which are defined by a rigid, evenly spaced grid of frequencies, can be generalized into the nonharmonic Fourier series Duufii & Schaeffer 1952 Young, 1981). This concept provides a mathematical bridge to the Fourier Transform (FT), which uses a continuous spectrum of all possible frequencies. The nonharmonic series occupies middle ground: like classical series, it is discrete sum of sinusoids, but it inherits the flexibility of the FT by allowing its frequencies to be arbitrary real numbers. This transformation is formally achieved by replacing the fixed set of integer frequency vectors, No. with general finite (for practical implementation) set of real-valued vectors, N .....,,NN. This flexibility allows for the construction of an adaptive, task-specific Fourier basis
[n-,, nn, n?.... capable of efficiently capturing the dominant spectral features of both periodic and nonperiodic functions. From this point forward, unless specified otherwise, we will use the term "Fourier series" to refer to this more general nonharmonic form, where the frequencies are from the N. The classical harmonic series will be considered special case for when N [ N
With this change of frequency set just described, the classical series defined in Equation 5 and Equation t are generalized to their nonharmonic forms. The complex exponential form Equation is generalized using a frequency set N C m as
fNH(2)=
Σ neN
On e in.
(7)

---

phase-shifee shifted cosines, and Φ1, Φ2 are the respective phase shifts. The equivalence Published in Transactions on Machine Learning Research (12/2025)
and the real trigonometric separable form Equation 6 generalized using frequency set of nonnegative real vectors, N S 10 as
(N)
Σ (m)
cos(m131) cos (o8 ... (om
nEW
(m) cos(n1x1) (22222222))) ... (mm +
(8)
+ 02
+ a2m
(m) sin(n) (in(22a))
mi
To begin with, we set the frequencies to nonnegative vectors (W C R2))) for two reasons: (i) in practice, this constraint reduces the search space for the optimal frequencies from 2nd possible orthants a single one, which leads to more efficient and stable search process. This is justified since each frequency vector comes implicitly with its symmetrical equivalent in all the other 2 remaining orthants, and; (ii) in theory, this is justified because any sign flip in the components of frequency vector only affects the sign of the coefficients, not the basis functions themselves, thus keeping the representation intact.
While both Equation 7 and Equation represent standard forms of the Fourier scries, they are not readily compatible with NN architectures: in the exponential form Equation 7 the basis functions are complex expo- nentials, which require complex vvauud weights, activations, and gradient updates. Since traditional NNs are designed for real-valued computation, this would necessitate specialized handling into real and imaginary parts, increasing implementation complexity. Additionally, the trigonometric form Equation g exxreesess each multidimensional basis as product of one ded-dmiiinnnll sine and cosine factors. Thus, these bases become difficult to be represented by traditional NNs, which combine inputs additively through weighted sums before applying nonlinear activations.
explicit separation
To resolve the representation issues above, we opt for the cosine phase-shifted shifted form, where, by phase shift we mean the translation of a cosine wave along its direction of oscillation. By converting the Fourier basis into sum of cosines with phase shifts, we climinate the need for complex numbers in Equation 7 and avoid the multiplicative interactions present in the trigonometric form of Equation S This makes the resulting representation more compatible with the additive structure of traditional, feedforward NNs. Next, we rewrite the standard separable trigonometric Fourier series in a cosine phase shifted form, beginning with the two diredimnnioonal case before generalizing to m dimensions. We also show the relationship between the Fourier coefficients the separable trigonometric form and the amplitudes and phase shifts the cosine phasc-shifted form.
3.1 Cosine Phase-Shifted Basis for Tio-dimensional Input
Consider the two iio-dimnniional Fourier scries
(N)(01,22)= = Σ (n) cos(1Izz) cos(12222) + a!!)) cos(n1z1) sin(n222)
nEA
(9)
03 nn) sin(n (12())))))))))))))))))))()))())) a (m) sin(n1 sin sin (1222))
where n (21, Alternatively, Equation can be expressed the cosine phase-shift form
(2)( = Σ
(m) cos [112) + +2222 pfm)) + An)) cos (n|z) 12222 - 2 (m)
nEN
(10)
= Σ Hn:
nEN
where H, ((T)) cos [1121 + n222 ((()) + A!T) cos (1171 n2#2 - ((m) can be viewed as an implicit
linear combination of the four separable Fourier basis functions associated with A(T) and A!") are the amplitudes of the phase of Equation 9 and Equation 10 is shown in Appendix A with the detailed derivation of the relation that expresses the Fourier coefficients a 02 ag
(n) in terms of the amplitudes and phase shifts (n) (n) and a
8

---

Published in Transactions on Machine Learning Research (12/2025)
(/ = (( cos $1 = A cos 22
02
(m)) sin Φ1
(n)
A sin 02
(11)
9 A sin 0(()) + A sin 02 (n)
a - cos d( +
(m)
A!Th) cos 02
Notice how the linear combination of two phase shifted cosine bases in Equation 10 is implicitly generating the linear combination of the four separable trigonometric bases in Equation both associated with same frequency components. The same frequency pair (71,72) occurs inside both cosine bases with different amplitudes and phaso-shifts, and with the sign of the sccond clement flipped. To extend this idea of expressing separable trigonometric bases with phase shifted cosines dimensions, define the following matrix.
Definition 1 (m-Leex Sign Matrix) Let m 2 be positive integer and 2 The m-Lext Sign Matria (6m) is the x matria whose rows denoted by
(1) (6) (ei e2
( 1=
with c¹" = and (e)² .
! {1, 11mm11 i.e, the (m 1) -old Cartesian product of the set (1,-1}. The rows ( are arranged in lezicographic order with taken to have higher priority than -1. Erplicitly,
(2)
e
(22)
((3)
I
g(m)
e44)
1
-1
(12)
:
:
",
e(l-1)
-1
-1
-1
ell)
-1
-1
For instance, for the cases of m = 2 and m = 3, we have
1
((2)
1 1
1
5(3)
1
1
-1
1
-1
g(m) helps us construct the separable bases with phase-shifteee shifted cosines in m-dimensions,, which we discuss next.
3.2 Cosine Shist-Shifte Basis for dimensiiiaal Input
In similar fashion in the dimensional case, H1, represents the linear combination of = 2 phase shifted cosines, which is equivalent to linear combination of 2 separable Pourier basis in Equation X and can be written as
Hn = Σ A (m) cos ((e o p(m))
1=1
where (i)) is the ith row of the m-Lexi Sign Matrix S and ( @ n represents its Hadamard (element wise) product with the frequency vector n. The m m-dimensional Fourier series can be expressed in a single compact summation as
fNu(æ) Σ Hn. BEN
(13)
The relationship between the Fourier coefficients from Equation & and the amplitudes and phase shifts of this equivalent form Equation 13 presented below. Stating this relationship for general dimension requires

---

Published in Transactions on Machine Learning Research (12/2025)
more descriptive framework than the sequential indexing (e.g., a 02 ) suitable for the two swo-ddmmnsiinaal case. We aim to calculate the coefficient O
(n)
corresponding to the basis function
II PEZz
(Pp
II gEZ2
sin sin (ng
where the set Ia (1,...,m) and its complement I (1 (1....,,0) m} I& specify the indices of the sine and cosine terms, respectively, for the basis corresponding to the index k {l,..., 1.....,2'! 2n) The index set Ig is determined from the sequential index k using the m bit binary representation of k k-1- 1: an index is included in I& if and only if the jth bit from the left 1. Given the amplitudes A!") and the phase shifts (m)) the separable product basis Fourier coefficients " are related by
(1) Si cos d(
In = 0 (mod 4),
A 81,k sin m
I 1
(mod 4),
(n) dj
A(m) Si,k cos mm))
Ig = 2
(mod 4),
(m) Si,k sin (m)
I = 3
(mod 4),
where the sign factor 81,6 E 1-1,1} is defined
$1,6 II c
!
JETx
4 Model Construction and Architecture
In this section, we discuss the architectural construction of FLMs. We begin with the standard definition of single-layer feedforward NN followed by formal definition of the FLM.
Definition 2 (Single-Layer Feedforward Neural Network) single- hidden layer NN with an imput vector R" L hidden neurons, and single output f(x) E IR is function defined as:
f(x)= =
Σ uout owwf" bU)
where whth is the input-to-hidder weight vector, by is the hidden newron bias, o(-) is the hidden layer acti- vation function, w is the hidden corden-to-ouute to-output weight, and g(-) is the output layer activation function.
Definition 3 (Fourier Learning Machine (FLM)) The FLM is composed of N parallel sub networks, each consisting of complete single-layer NN conforming Definition g with its hidden activation function : : cos(x), and ils oulpul aclivalion, g + The weight applied to the inpul. the lop neuron in the hidden-les layer of each sub network is set to be a vector To each subsequent newron in the hidden-layer, the same weight vector is used, with the respective sign of its components determined by the Lexi Sign Matria. The final output of the entire FLM is the ordinary sum of the outputs from all these individual sub networks.
Next, we present three instances of possible sub- networss,,,,, first for the two- two-input and three- imput cases and then for the general m-input case.
10

---

Published in Transactions on Machine Learning Research (12/2025)
Sub-net 1
Sub-net 2
1
14
Σ
Bn
Ay 702
3/2 11g
Sub-net
Sub-net N
Figure 2: FLM with two imputs. The sub-network associated with frequency pair (Left) and the main network with N sub-networks (Right).
Example 4.1 (FLM Sub-networks with Two Inputs) An FLM with two imputs is composed of sub- networks, each parameterized by a learnable weight vector n, with input (x1, two hidden neurons with a cosine activation function and bias. The vector n = (M1. n2) contains the weights from the input-lager to the first hidden-newre newron. For the second hidden neuron,, the same weights are reased encept that the second element T2 has opposite sign. The output of the sub network is the weighted sum of outputs of the two cosine aclivaled hidden neurons, i.e.,
Hn A(m²) cos(n1x1 + n222 - ((()) A cos(m1 R2J2 -(()))
The output of the FLM is the ordinary sum of the sub network oulpuls
f(21,22) = Σ (x1,
(14)
mEN
where N is the set of learnable frequency vectors, one from each sub-network. Equation 14 has the same form of the Fourier series in Equation 10, The hidden-to-oudhe ouuput layer weights A(m) of the neb-network can be interpreled as the umplitudes (T)) and the biases ((()) as phuse shifls (!m), respeclively, for = 1,2. Fig. illustrates FLM sub network with two inputs and the FLM itself.
Additionally, FLM with 3 inputs is shown in Fig. alongside its underlying 5(3) matrix. Now we dis- cuss the structure of the FLM sub-networks in general for m m-inputs and explain the interpretation of the multidimensional FLM.
Example 4.2 (FLM NettoNessrss with m inputs) An m no-inpul FLM has N parallel sub nedworks, each of which associated with weight vector n (n1, ,,,...,;mm.. It has inputs F2., In and 2 hidden nodes with cosine activation function and biases ((m) fori=1....! The input-to-hidder layer weights for the ith node of the sub-net networkkk is the Hadamard (element wise) product e(t) o n where the row vector (i)) is the ith row of the m Lexi Sign Matria S(m) (Equation 12 An illustration of the m dimensional FLM sub network is presented in Fig. 4
...,,..,,
The same node operations to the 2 dimensionnl case are used and the weighted of the activation is calculated. Then the output of the sub-networt becomes
An
Σ A(m) cos [((e + e2 (0)
1222 + ... + e nm) ( (/m)
1=1
(n) cos (ee -b(rl)
11

---

Published in Transactions on Machine Learning Research (12/2025)
(a)
A
(1)
Iv
22
Σ
11
P6
Ag
nn)
23
3 Lexd Sign Matrix
A
Pd
S(3)
71
(c)
12/
(d)
77.
71
72.
713
771
772
77.
711
772
77:3
Figure 3: (a) FLM with imputs and the ((3) matrix defining the sign of the frequency components R1, 12 and ng. These components are illustrated, respectively, in (b) (d).
F1
P
(
Aj
12
P
A
(2) 1
Σ
H
E3
Pd
(3) m
A/
Lm
2 o
m inputs
I hidden-nodes
Single output
Figure m-dimensional FLM sub-network (biascs are not shown).
12

---

Published in Transactions on Machine Learning Research (12/2025)
Therefore, the output of the main network is given by
,m)) 02.,, 2m)) E nEN
RmL),
where N is the sinile set conlaining the learnable frequency veclor from each neb-nelwork. Notice that An matches the form of Hn and, consequently, f with the Fourier series fNH itself in Equation 13. The hidden- to-output layer weights A!m) of the sub-network b(m) as phose shifts ( respectively for i= == I...... We note that the 2m newrons per sub-network grow exponentially with m, which is an inherent consequence of representing an m -oddimnnsionll Fourier basis in separable form. Thus, each sub-network with 2 neurons explicitly generates composition of 2m-1 m dimensional phase shifted cosines or implicitly 2m seporable Fourier basis with a specific frequency vector. The relative contributions of each individual Fourier component are learned through the sub-network biases and oulpul layer weights. Although the nelwork does not perform explicil projection onlo the orthogonal bases, it learns both the amplitude and phase of each learned basis function, enabling structured and interpretable approzimation of the larget map.
be interpreted as the amplitudes A!m) and the biases
We have demonstrated in this section that a MLP-like Like, feedforward NN with traditional node operations can represent m mn-dimenioonal nonharmonic Fourier series. In the next section, we compare the performance of our FLM model against other NN models.
5 Computational Results
In this section test and compare the performance of our proposed FLM model in the computation of numerical solutions of four PDEs and an OCP arising from an odd-cireulant evolutionary game. Additionally, we design a series of computat ional experiments aimed at comparing the performance of our proposed model against other traditionally employed NN architectures.
The PDEs considered herein are: the onc-dimensional heat equation, the two-dimensional Poisson equa- tion, the generalized Black Scholes model, and the one dimensional inviscid Burgers' equation. Besides the emerging interest in machine lcarning tochniques for the solution of PDEs, demonstrated, for example, in Ngom & Marin 2202)); ; Ma et al. (2022), the solution of OCPs by such methods was also recently brought to attention (see, e.g.. Nicolosi et al. (2022]; Nicolosi & Griffin (2023): ; Nicolosi et al. (2025): Rubel & Nicolosi (2025 Lundqvist & Oliveira (2025)) ). Thus, we consider in this paper an OCP arising from the evolutionary dynamics of a Rock-Paper Papeesss--Seissooss
Scissors game, as studied in Griffin & Fan (022).. Its nonlinear dynamics along-
side its state space configuration make this OCP particularly challenging to be solved analytically, so our solution shall be tested against an (indirect) numerical solution based on the Pontryagin's 's Minimum Prin- ciple (PMP). While traditional techniques like Finite Elements and Finite Differences are well-ostablished methods, the recent rise in interest on machine learning- based techniques has introduced new class of powerful, mesh-free free solvers based on NNs (Raissi et al. 2019). In this section, we demonstrate the efficacy of our proposed FLMs in solving several benchmark PDEs and a class of OCPs.
In all experiments, the frequency vectors of the FLMs are initialized at a finite subset of integer grid points in NY,, chosen sequentially from the lattice: for example, (0, 0), (0, 1), (1,0), (1,1) 1) in the two-dimensional case with four sub networks. However, we do not constrain them to remain within Ry during the training, The reason for this lies in the inclusion of sign fign-fipped versions of the frequency vectors within each sub- network,, together with the even symmetry of the cosine function. If any component of the frequency vector becomes negative, the FLM automatically accounts for its symmetrical equivalent, thus adjusting the corresponding coelficients to miligate this elfect. In each sub network, biases (phase shifts) are randomly sampled from the normal distribution (0, 5) and hidden to tu-put layer weights (amplitudes) are initialized to 0. This initialization stratogy is primarily motivatod by empirical observations from our experiments and we keep it as an open question whether there is any mathematical way of determining better set of initial weights. Next, introduce each PDE considered, as well the customized loss function.
13

---

Published in Transactions on Machine Learning Research (12/2025)
5.1 Numerical Experiments for Partial Differential Equations
We consider general nonlinear PDE defined on the spatio temporal domain Q [O, where R=-1 and is given by
(a, t) 0, u(x,0) 0) uo(x)) (2, () g(2e,)) l);
(æ, Ω (0,7), E Ω (æ, E 0n x (0,
Here, 0n is the spatial boundary of Ω2 Fl-] and BI-] denote general (possibly nonlinear) differential operators, u(z, t) is the unknown solution, and uo (x) and g( t) define the initial and boundary conditions, respectively,
We approximate the solution u(æ, t) using a FLM uo(æ, parameterized by its weights and biases, col- lectively denoted as O. The network is trained by minimizing composite loss function ((O) (such in Cuomo et al. (002))) that penalizes deviations from the governing PDE and its associated initial and boundary conditions
(()) = Lie + LBC + LPDE,
(15)
where
Nic
1
Lic
(a 0) ux(x)))
Nic ==1
NBC
LBC
(oy, g(z,,, t,)²
NBC
j=1
Nppr
LPDE
Fluel 1x)
NPDE
Each component of the loss function Equation 15 is mean square error (MSE) computed over distinct set of collocation points, defining the training set. These points are randomly sampled from their corresponding domains: } from the spatial domain Ω at f = 0, {(E,,,;)) from the boundary on x [0, 7"), and {(*x, 4x) from the interior x (0, The integers Nic, NBC, and NPDE are the number of the training points used to enforce the initial, boundary, and PDE constraints in the loss function, respectively. The differential operators F[.] and B||] are evaluated on these points using automatic differentiation, and the loss function is then minimized with adaptive momentum gradient descent algorithm (ADAM) (Kingma & Ba 2014).
To test the model's final accuracy, separate test set is constructed from evenly spaced grid covering the entire domain. On this test set, we compare the FLM-approxinated solution to the exact solution using three slandard error metries. We have reported the MSE, which is the average of squared differences; the Mean Absolute Errors (MAE), the average of absolute differences; and the Max Error, which is the largest point wise absolute difference. Now we examine the effectiveness of FLMs to solve PDEs and see how their performance compared those of other popular NN architectures, listed in Table Each network in the table has Nhidden neurons per hidden-layer.
Table 1: Description of other NN architectures for performance benchmarking against FLMs.
Model
Description
SIREN 1
SIREN with hidden layer (wo = 1)
SIREN-2
SIREN with 2 hidden layers (wo = 1)
SIREN-3
SIREN with 3 hidden layers (wo =
Relu
Vanilla feedforward NN with Relu activation function and hidden layers
V LRelu
Vanilla feedforward NN with Leaky Relu activation function and hidden layers
V Tanh
Vanilla feedforward NN with Tanh activation function and 3 hidden layers
14

---

Published in Transactions on Machine Learning Research (12/2025)
We trained FLMs and diverse NN models with multiple hyperparameter configurations, selected the best using a hierarchical statistical testing procedure, and further trained the top configuration for 30,000 epochs with a stricter loss tolerance. Details are shown in Appendix C
Example 5.1 Dinenniinnsi Heat Equation) We consider the one-dimensional heat equation on the domain [0, 1], & [0, I], given by
du 8u 2x
E 1), t (0, 1].
at
u(x, 0) sin(TT)) x E [0. 1].
w(0, = 0, w 00 l LE 1]:
where the thermal diffusivity constant set a 0.1. The analylical solution this inilial boundary value problem u(x, 1) sin(xx) e aa
Fig. 5a illustrates the comparison between the exact solution and the FLM approximated solution. The performances of all models are summarized in Table 2a
Example 5.2 (Two-Dimensional Poisson Equation) Consider the Poisson equation on the unit square with homogeneous Dirichlet boundury condilions:
P2u 8u 2m²² sin((T)) sin())) (z,y) E Ω = (0,1)
dr
oy²
u( v) = O, (z,z) 8R.
The eract solution is u(x,y) sin(mx) (iy()
Fig. 5b shows the comparison between the exact solution and the FLM approximated solution. The results
are summarized in Table 26
Example 5.3 (Generalized Black-Scholes Equation) Consider the PDE extracted from Ma et al.
2022), with inilial and boundury condilion
du
82u
Ix + b(xx 1) du
a(xx 1)
+ cx f)u + d(a,t), (x,1) E (-2,2) x (0, 1],
01
dx
u(x, 0) = e", E
2,)) = 2-t u(2,1) = 2- LE [0, 1].
The coefficient functions are chosen as
a(x, f) 0.08 (2 + (1 1) sin(
b(x, f) = 0.06(1 + te ) 0.02c -t-e a(x, t),
c(x, t) -e
d(x,t) f) = 0.02e"
The exact solution is u(x, = e-=
The comparison between the exact solution and the FLM approximated solution is presented Fig 5c results are presented in Table 2c
The
Example 5.4 (Inviscid Burgers' Equation)
du
u du
+
0,
15r51,0Si51
01
dr
u(x, 1.0 0.35 sin
u(0,t) u(1,t)-
15

---

Published in Transactions on Machine Learning Research (12/2025)
This PDE has discontinuous solution and the boundary condition u(0,t) = =(1,t) induces discontinuity in the boundary well. In this case, we compare the approximated solution with the numerical solution obtained by Godunov's method. The results are illustrated in Fig. 5d and summarized in Table 2d
Table 2: Approximation Errors for Various PDEs Using Different Models
(a) Heat PDE
(b) Poisson PDE
Models
MSE
MAE
Max Error
Models
MSE
MAE
Max Error
FLM
6,24 x 10-
2.01 x 10
7.37 x 10
FLM
1.34 10
2.04 10 10"4
6.74 10"
SIREN-1
5.07 x 10-7
5.29 x 10
1.97 10
STREN-1
2.01 x 10 7
3.95 x 10
1.86 x 10 -3
SIREN-2
3.07 x 10-6
1.18 x 10
8.06 x 10-3
SIREN-3
5.40 x 10-5
3.90 x 10 10-3
2.00 x 10
SIREN-2
8.07 10
7.09 10-3
3.43 10-2
V Relu
6.39 x 10
2.03 10
6.72 10-1
Relu
2.45 10
3.97 10
9.99 10
V Tan
9.24 x 10-6 1.39 x 10-3
4.08 x 10-3
Tan
2.64 x 10-4
9.95 x 10-3
6,56 x 10-2
(c) Generalized Black Scholes PDE
(d) Burgers' PDE
Models
MSE
MAE
Max Error
Models
MSE
MAE
Max Error
FLM
3.80 x 10-7
4.44 x 10-4
4.53 x 10-3
FLM
6.55 10 3
4.07 10
3.73 10
STREN
2.13 10-2
1.15 10-1
3.89 10-1
SIREN-1
4.09 x 10-5
4.84 x 10-3
3.44 x 10-2
SIREN 2
9.80 10-3
5.86 10-2
3.59 10
SIREN-2 2
8.37 x 10-7
7.39 10-4
2.91 10-3
SIREN 3
6.96 x 10-3
4.34 x 10-2
3.68 x 10-1
V LRelu
6.53 x 10-2 -
1,75 x 10
9.85 x 10
Relu
3.09
10
1.46 10
3.74 10
V Tan
6,06 x 10- 2.02 x 10
5.54 x 10-3
V Tan
4.58 10-3
2.88 10
4.44 10-1
The numerical experiments demonstrate that FLMs consistently outperform other NN models across a of multidimensional PDFs. While STREN and vanilla feedforward NNs perform well when the target function contains similar mathematical behavior as the activation function they use, FLMss achieve significantly lower error metrics and faster convergence (see tables in Appendix C regardless of the PDE. In terms of the MSE and MAE metrics, FLMs were shown to perform better than all models considered for each PDE, with the exception of Burgers' PDE. This difficulty in approximating the solution of Burgers' equation was anticipated, mainly because of the presence of a discontinuity. However, the FLM was still able to attain an MSE of the same order of magnitude of the best performing model, the Vanilla Tan NN. In terms of Max Error, FLMs also showed an overall superior performance, even though not always dominating the other models. In general, out of all of the competing modcls, SIREN NNs are the models with the closest performance to that of FLMs. In the next section, apply the proposed FLM model to the solution of a family of OCPs.
5.2 Numerical Experiments for Optimal Control Problems
Both NN and Fourier series have been proven to be very effective to solve OCPs recent years (Fffati Pakdaman 2013 Nicolosi et al. 2022 Nicolosi & Griffin| 20233 Nicolosi et al.) 2025 Rubel Nicolosi 2025
To show the effectiveness of FLMs to solve OCPs, consider the optimal control of system governed by the dynamics of 3 s-srateegyy circueirullnt evolutionary game, namely, Rock- Sniper-Siisors game (Griflin & Fan 2022). The state of the system, vector u(t) = (u1, u2, M3) with 41 + u2 + 1, represents the proportion of population playing each strategy. The goal is to push these proportions toward the desired equilibrium point ueg = (1/3, 1/3, 1/3) representing an equally balanced population.
The system's evolution described by the replicator dynamics and influenced by an external control input, 7(). This control actively alters the game's payoff matrix, which is defined as As(r) La + MM,, where L3 is the standard payoff matrix and M3 is an actuating matrix and given by
0 -1
0
0
1
Lg =
1
0
-1
M3 =
0 0
-1
1
1 0
16

---

Published in Transactions on Machine Learning Research (12/2025)
Heat Equation
Exact Solution
Approximate Solution (FLM)
Absolute Error
x10
M|
0.5
0.5
0.5
0.5
0
8
L
(a)
Poisson Equation
Exact Solution
Approximate Solution (FLM)
Absolute Error
x10+
Iufa, v) Mafar, v)
0.5
0.5
0.5
0.5
0.5
0.5
0
0
æ
y
y
(b)
GBS Equation
Exact Solution
Approximate Solution (FLM)
Absolute Error
x
uala, t)
4
Iula, 8)
0.5
0.5
0.5
Burgers Equation
Exact Solution
Approximate Solution (FLM)
Absolute Error
0.3
1.2
We (a, t)
1.2
0.8
0.8
0.2
0.8
0.2 0.4 0.6 0.8
0.2 0.4 0.6
0.5 2:
t
æ
(d)
Figure 5: Solution surfaces of the Heat (a), Poisson (b), GBS (c), and Burgers' (d) equations, showing exact solution (left), FLM approximation (middle), and absolute error (right).
17

---

Published in Transactions on Machine Learning Research (12/2025)
The objective is to find an optimal control strategy (t) that minimizes cost functional over time horizon [0, T)- This cost has two components: a term that penalizes the squared distance of the state u(t) from the target equilibrium Way, and a term that penalizes the squared control effort 7())²
An important point to note is that the given OCP possesses a cyclic symmetry. If (t (uj (4), u5((), ( is the optimal state trajectory, (²()) is the optimal control and J the optimal objective value for an initial condition uo (wor, u02, uns) then any cyclic permutation of this initial state will yield the same optimal control solution and objective value. Consequently, the new optimal state trajectory will be the corresponding cyclic permutation of the original trajectory, u (t). For instance, if the initial state is permuted to (Moz, Mo3, Mo1), the optimal trajectory becomes (t)(t), (t(t), (?))), while the optimal control and objective value remain unchanged. The numerical approximation of the solution of the OCP with our proposed model is presented in the next example.
Consider a class of the OCP described above, where the initial condition uo varies withing a given range. written
min J(u,1) 7) = 1/" (+ uegll (/²²)) dt
s.t.
ú = F(u) + (()
(16)
u()) = Ug E Uo:
where ull) E A2 lies on the 2- simplex. The vector fields, for i 1,2,3, are given by
Fu( u, ((e: u1" Lzu): G u, ((e, - -
(17)
Let (t, ug: ..,)) be the FLM that learns the approximation of state variable (t, uo), d.e., ū (ū1, 02 ūs) and (t. uo: Q4) be the FLM that learns the control variable ytt, un) for any trajectory departing from Uo, where O = (O.,, Ouz Oug: θ,} is the collection of all network parameters of the respective networks. Given the complex nature of the OCP above, the training process is different than it was for the PDEs and so is the construction of the training loss function. We train all the FLM simultancously by minimizing the training loss using a penalty method (Ch. 9 in Bazaraa et al. (006)). We define an unconstrained loss function that incorporates the objective functional and penalties for dynamics and initial condition violations as
((0) = NIC 20
1
Σ
J
J-
S Vayn / j
Vinit
(18)
with
((9)= = /''
T (1 (21ā uegl' + F 772) dt,
V dym (+) (/' i F(ū) - GG(( 3 F(u) (()))-G((@) dt,
T
0
(+) (ū(0) uo) ② (0 un)
where 11: μl are vectors of positive penalty coefficients. Minimizing the total loss L with respect to O, and O. forces the FLM to approximate the optimal state trajectory and control signal that solve the original OCP. Next, we consider two cases: a fixed initial condition case, for given uo, and a for range of initial conditions Uo A2.
5.2.1 Fixed Initial Condition OCP
First we solve the OCP for fixed initial condition with a one dimensional FLM, which is a NN with one hidden-layer and cosine activation function. We consider three such FLMs, two for the state variables u and u2 and one for the control 7. The third state variable is calculated using the relation us u1 ug. The training is performed with default. learning rate 0.001 and betas &1 0.9 and B2 0.999 with the ADAM
18

---

Published in Transactions on Machine Learning Research (12/2025)
optimizer. To test the accuracy of our model, compare its performance against numerical solution obtained via. PMP to formulate a two point boundary value problem, which was then solved numerically to a high precision (tolerance of 10= using the solve_bye function from Python's SciPy library. The FLM approximated solutions and the PMP-based numerical solutions for three different, fixed initial conditions are shown Fig. G The approximated optimal objective function values have absolute percentage errors of 0.47%, 0.37%, and 0.34% respectively. Fig- 7 shows the state trajectory for the initial point up = (0.20, 0.20, 0.60) The parameters of the OCP, as in Griffin & Fan (0022), are chosen as T = 6.0, r 0.2.
We note that the PMP-based method we compare our results against is an indirect method, thus it is contingent upon the existence of a Hamiltonian function associated to that particular system. On the other hand, our FLM based method is a direct one (í.e., it translates the OCP to a discrete, nonlinear optimization problem), neither based on the existence of a Hamiltonian nor on any convexity assumptions, thus applicable to solve both convex and nonconvex OCPs.
Control for Different Initial Conditions
11g = (0.20, 0.20,
1k = 0.217,.22,00611
11g = (0.15, 0.68)
0.4
FLM Apprex.
0.5
.PMP-baaee Approx.
0.3
0.4
0.2
0.2
0.3
0.2
0.1
0.1
0.1
4
4
4
Figure 6: The PMP-based numerical solution vs the FLM approximation for different initial conditions with N = 5 neb-networ.s.
State Space Trajectory
PMPP-assed Appeox.
FLM Approx.
Inltial Stace
Terminal State u(T)
0.8
0.6
0.4
0.2
0.2
0.4
0.6
0.8
01
U2
Figure 7: Optimal state trajectories for wo = (0.20, 0.20, 0.60), T= and r r= 0.2.
5.2.2 Varying Initial Condition OCP
We solve the same OCP considered thus far, but now varying the initial condition uo in a set Uo, defined as a disk on the 2-simplex, centered at (0.20, 0.20, 0.60), with radius 0.15. Three FLMs with with N = 27 and three inputs (1, u01, 402) (we recall that the third stale variable and its initial value is dependent on the other two), are used to approximate the two state and one control variables. The three FLMs are trained on 250 random points inside the disk, and then tested in 100 newly generated random points on the disk with center at (0.20, 0.20, 0.60). Given the cyelic symmetry of the s-simllex and the underlying payoff matrix, two other disks, centered at (0.20, 0.60, 0.20) and (0.60, 0.20, 0.20), provided sampling of 200 extra points
19

---

Published in Transactions on Machine Learning Research (12/2025)
used initial conditions for testing. This choice of testing points is mere illustration how could leverage the inherent cyelic symmetry of this OCP in order to reduce the training space required to learn the approximated optimal control law. This is attained by informing the controller on what disk it is actuating on, allowing it to use the appropriate coordinate control learned at the disk used for training. Nonetheless, such a training choice does not undermine the results reported herein had we chosen to focus on a single disk in the 2 s-mipee.x.
The hyperparameters are tuned and learning rate of 5 x 10 and beta values of 0.95 and 0.97, respectively, are selected for training. To quantify the approximation accuracy, define the Mean Absolute Percentage Error (MAPE) on the objective function as follows:
1 MAPE
JELM JPMP
JAMP 100%
(19)
100
Percentage Error in the jth test point
where JFLM and J.MP are the objective function values obtained from the FLM approximaled solution and the PMP based solution, respectively, for the jth test point. Over the test points each of the three disks, the MAPEs are calculated to be 0.81%, 0.85%, and 0.72% respectively, while the respective MAEs are calculated to be 2.54 x 10-3 2.74 x 10-3 and 1.81 x 10-3 Fig. S demonstrates the percentage errors and the absolute errors (point wise) in the test points the three disks. These numerical results represent a substantial improvement from the ones initially reported Nicolosi (2023), where a finite Fourier series failed to approximate the optimal control law of this OCP in this case of varying initial condition.
Optimal Objective Function Approximation Errors
Percentage Error
Abolute Erroz
3
3
1.5
0.5
0.5
0
0.5
Mg
Mg
Figure 8: (Left)) Absolute percentage errors and (right) absolute errors in the objective function value on test points within the circular disks on the simplex plane centered at (0.2, 0,2, and its cyclie permutations with a radius of 0.15.
5.2.3 Varying Initial Condition 5-dimensional OCP
To demonstrate the scalability of FLMs, we solve the strategy exlension of RPS, known RPSSL (Rock Scissors Scissors-Spockk Spock-Lizard). We extend the OCP in Equation 16 by replacing the matrices L3 and M3, respectively, in Equation 17 with
1
1
-1
0
0
0
1
0
-1
-1
1
0
0
1
0
-1
1
0
-1
1
M,
0
1 0
0
1
-1
1
-1
1
0
-1
0
For this experiment, 4 FLMs with 5 inputs and N 16 sub-networks were used approximate 4 states and FLM with inputs and N 32 webb-newwrks employed to approximate the optimal control. In this
20

---

Published in Transactions on Machine Learning Research (12/2025)
case, the equilibrimm point is Ueg (0.2, 0.2,0.22 All of the FLMs were trained and tested following the same method discussed before with a learning rate of 10-1 and beta values of 0.95 and 0.97. Training the models with initial conditions (uniformly) sampled from a hyperdisk in the 4-simplex AS of radius 0.045 centered at (0.11, 0.11, 0.11, 0.56), we have achieved an MAPE of 0.67% with MAE of 2.10 x 10-3.
Our results show that FT.Ss are capable of numerically solving PDEs with the same level of accuracy of other traditional NN models, often outperforming them in terms of error metrics considered. The OCP example also illustrated how FLMs can approximate the state trajectory and the optimal control of a high digeesinnssional family of OCPs.
6 Conclusions and Future Work
We have introduced the FLM, a neural architecture that learns optimal frequencics directly while systemati- cally representing a multidimensional nonharmonic Fourier series, thereby bridging the gap ween classical spectral methods and modern NN architectures. By jointly training frequencies, amplitudes, and phase shifts, FLMs overcome the restrictions of fixed harmonic Fourier series and adapts naturally to both periodic and nonperiodic problems. Its implementation optimization tools, such as backpropagation, automatic differentiation and state-of-the-a solvers. Com- putational experiments on benchmark PDEs demonstrate that FLMs can match or outperform established architectures such as SIREN and vanilla feedforward NNs. Additionally, FLMs solve a family of OCPs with high accuracy, demonstrating its generalization capability and its potential for high dimensional, nonlinear problems.
NN allows it to leverage scientific computational and
Future work will investigate an improved initialization scheme for the FLM weights (Trequencies and am- plitudes), potentially important for when the target function is dominated by Additionally, one can consider applying FLMs to other domains, such as image and audio processing, medical signal processing, and time series analysis (a brief time series imputation study is presented in Appendix B Finally, deeper theoretical relations, in particular with SSGP (Lázaro-Gredill et al. (2010)) can be further investigated, potentially strengthening and better understanding the underlying principles of the spectral adaptability of both FLMs and SSGP models.
components,
References
Mokhtar S Bazaraa, Hanif D Sherali, and Chitharanjan M Shetty, Nonlinear programming: theory and algorithms. John wiley & sons, 2006.
Nuri Benbarka, Timon Höfer, Andreas Zell, et al. Seeing implicit neural representations fourier series. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2043-2242 2043-2050, 2022.
Mohamed Benrabah, Kamel Kara, Oussama AitSahed, and Mohamed Laid Hadjili. Adaptive fourier series neural network pid controller. International Journal of Control, Automation and Systems, 19(10):3388 3399, 2021.
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics-informed neural networks: Where we are and what's next. Journal of Scientific Compuling, 92(3):88, 2022.
Wenjie Du, David Côté, and Yan Liu. Saits: Self-attention-batt imputation for time series. Expert Systems with Applications, 219:119619, 2023.
Wenjie Du, Jun Wang, Linglong Qian, Yiyuan Yang, Zina Ibrahim, Fanxing Liu, Zepu Wang, Haoxin Liu, Zhiyuan Zhao, Yingjie Zhou, et al. Tsi-bench: Benchmarking time series imputation. arXiv preprint ar 2rv::22406. 12747, 2024.
This choice of radius length such that the density of sampled initial conditions the same as the RPS case.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Richard Duffin and Albert C Schaeffer. A class of nonharmonic fourier series. Transactions of the American Mathemalical Sociely, 72(2):341-38) -666, 1952.
Sohrab Effati and Morteza Pakdaman. Optimal control problem via neural networks. Newral Compuling and Applications, 23:2093 2100, 2013.
A. Ronald Gallant and Halbert White. There exists a neural network that does not make avoidable mistakes. In IEEE 1988 International Conference Newral Networks, pp. 657-664. IEEE, 1988.
Christopher Griffin and James Fan. Control problems with vanishing lie bracket arising from complete odd circulant evolutionary games. Journal of Dynamics & Games, 2022.
Krzysatof Halawa. Fast and robust way of learning the fourier series neural networks on the basis of mul- tidimensional discrete fourier transform. In International Conference on Artificial Intelligence and Soft Computing, pp. 62-70. Springer, 2008.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Newral nelworks, (6)):59-336) 1989.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ar preprinl arXiv:1412.6960 6980, 2014.
Miguel Lázaro- Lázaro-Gredilla, Joaquin Quinonero-Candela Carl Edward Rasmussen, and Aníbal R Figueiras- Vidal. Sparse spectrum gaussian process regression. The Journal of Machine Learning Research, 11: 1865-1881, 2010.
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural nefworks, 6(6):861-867, 1993.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadencsheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkunar. Fourier neural operator for parametric partial differential equations. arXiv preprint Xiv:20010 08895, 2020.
Shuang Liu. Fourier neural network for machine learning. In 2013 international conference on machine learning and cybernetics, volume 1, pp. 295- IEEE, 2013.
Oliver GS Lundqvist and Fabricio Oliveira. Was residual penalty and neural operators all we needed for solving optimal control problems? arXiv preprint arXiv:2506.044 04742, 2025.
Mingjie Ma, Jianhui Yang, and Ruobing Liu. A novel structure automatic-determinee fourier extreme partial differential equation. Knowledge-Ban Systems,
learning machine for generalized 238:107904, 2022.
Kevin P Murphy.
Probabilistic machine learning: Advanced topics. MIT press, 2023.
Radford M. Neal. Bayesian Learning for Neural Networks, volume 118 of Lecture Notes in Statistics. Springer- Verlag, New York, 1996.
Marieme Ngom and Oana Marin. Fourier neural networks as function approximators and differential equation solvers. Statistionl Analysis and Data Mining: The ASA Data Science Journal, 14(6):647-66) 661, 2021.
Gabriel Nicolosi. Fowrier Appronimations of Optimal Control Surfaces and Their Applications. PhD thesis, The Pennsylvania State University, 2023.
Gabriel Nicolosi and Christopher Griffin. Approximation of optimal control surfaces for the bass model with stochastic dynamics. In IISE Annual Conference and Erpo. IISE, 2023.
Gabriel Nicolosi, Terry Friesz, and Christopher Griffin. Approximation of optimal control surfaces for 2x 2 skew-symmetric evolutionary game dynamics. Chaos, Solitons & Fractals, 163:1 12535, 2022.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Gabriel Nicolosi, Terry Friesz, and Christopher Griffin. Approximations and bounds for optimal control via finite fourier series. Plysioo Scripla, 2025. In press.
Tekin Evrim Ozmermer. Sinusoidal neural networks: Towards ann that learns faster. Complex Systems Informatics and Modeling Quarterly, pp- 44-57, 2020.
Giambattista Parascandolo, Heikki Huttumen, and Thomas Virtanen. Taming the waves: sine as activation function in deep neural networks, 2017.
Maziar Raissi, Paris Perdikaris, and George E. Karniadakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Compulational physics, 378:686 707, 2019.
Mominul Rubel and Gabriel Nicolosi. An initial condition-dependent neural network approach for optimal control problems. arXiv preprint or Sil:2502.08000 2025.
David E Rumelhart, GeofTrey Hinton, and Ronald Williams. Learning representations by back- propagating errors. nature, (6088))553 536, 1986.
A. Silvescu. Fourier neural networks. In IJCNN'99. '99. International Joinl Conference on Newral Nelworks. Proceedings (Cat. No.99CH86339), volume 1, pp. 488 491 vol.1, 1999.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in neural information processing systems, 33: 7462 7473, 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Malika Uteuliyeva, Abylay Zhumekenov, Rustem Takhanov, Zhenisbek Assylbekov, Alejandro Castro, and Olzhas Kabdolov. Fourier neural networks: A comparative study. Intelligent Dato Analysis, 24(5): 1107-1120, 2020.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34: 22419-22430, 2021.
Xuhua Yang, Huaping Dai, and Youxian Sun. Simo fourier neural networks research. In Proceedings of the 2003 IEFE International Conference on Intelligent Transportation Systems, volume 2, pp. 1606-1609. IEEE, 2003.
Jinsung Yoon, William R Zame, and Mihaela Van Der Schaar. Estimating missing data in temporal data streams using multi-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering, 6675):1477-1490 2018.
Robert M Young. An introduction to nonharmonic Fourier series, volume 93, Academic press, 1981.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang, In- former: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fiftt Fifth AAAI Conference Arlificial Intelligence, AAAI 2021, Virlual Conference, volume 35, pp. 11106 11115. AAAI Press, 2021.
C. Zhu and F. W. Paul. fourier series neural network and its application to system identification. Journal of Dynamic Systems, Measurement, and Control, 117(3):253 261, 1995.
23

---

Published in Transactions on Machine Learning Research (12/2025)
Appendix
A Detailed Derivation of The Relation Between Separable Coefficients and Phase-Shifted Cosine Parameters in Two-Input Case
Hn = A cos NI + 2222 - 01 (m)) + A!T) cos RI - n222 - 022
(12)
= A cos o) cos(n1 + n242) + sin の (m) sin(n1 + n22222
+ A (oSs (m) cos(m121 - n222) + sin 02 sin(n1 - 122222)
= =4 (")
cos ofT!! (cos(m1z1) cos(ng22) - sin(n] sin + sin o/ (sin(1I cos(12212) cos(n1 sin (
+ (m) A
cos (m) (os((1 ( + sin (1) sin (222n2)
+ sin m! (sin(n1Z1)) cos()) cos( (m sin(22z2)))
(m(m) cos p(m))
(m)
AAmm cos ob²²) (m)
(m)
cos(n1 cos
(m sin Φ1 (n)
A
(m) sin d5 (m)
cos sin (22
+ sin Φ
(n)
m sin 02 (n)
(m)
sin(n (12322)
+ A COs Φ + A cos (m)
(1)
(m)
(in sin(22
Comparing with Equation 9 we get,
(n) = A(m) cos (m) + A cos (m)
(n) = A (m) sin 01 (n) AA(r) sin "
Q2
3 (m) = A(m) sin Φ1 + (n) sin 02
(n)
(n) A ((m) cos 11 (m)
(n)
+ (m) cos 02
B FLM For Time Series Imputation
To demonstrate the potential of FLMs for time series imputation tasks, we present in this Appendix some numerical results derived from brief set of computational experiments using data from the Flectricity Transformer Temperature (ETT) dataset Zhou et al. (2021) This dataset is widely used in time series forecasting and impulation benchmarks due to its strong quasi periodic patlerns and significant fluctuations, making it good testbed candidate to evaluate the frequency learning capabilities of our model. We compare the performance of FLMs to the models presented in Du et al. (0044) also evaluated for the ETT data set.
B.1 Dataset and Experimental Setup
We utilize the ETThl (Electricity Transformer Temperature hour) subset, which contains hourly records of oil temperature (OT) and other load features from electricity transformers over a period of two years. For this experiment, we focus on the univariate imputation of the OT channel only, which exhibits both daily and yearly scasonality alongside high-frequency oscillations.
We replicate the data preprocessing and experimental protocol present in Du et al. (024)):
The data is normalized using standard scaler (zero mean, unit variance). The time index t is mapped to the domain [-.,
Preprocessing:
Splitting: We employ a chronological split of 60% for training, 20% for validation, and 20% for testing.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Masking: To simulate missing values, we apply random point-wise masking at two different miss- ingness rates: 10% and 50%.
The model is trained to minimize the MSE on the observed (non masked) data points of the training set. After hyperparameter tuning, we utilize the Adam optimizer with learning rate x 10 and a hidden layer size of 8. The training runs for 10,000 epochs with early stopping based on validation loss.
Training:
B.2 Results
We report the FLM approximation error the held-out masked values of the test set and the performance is evaluated using MAE. Table 3 summarizes the results averaged over 10 independent runs with different random seeds. For comparative baselines, we refer to the comprehensive results established in Du et al.) (2024), noting that the FLM achioved a competitive performance (although not superior) with a significantly lewer number of parameters and much simpler architecture than most of the models considered in [Du et al. (2024) (mostly Deep NN models). In particular, referring the performance of models in Du et al. (2024) for the dataset considered, FLMs would rank 26 and 24 out of 29 for the cases of 10% and 50% of point wise missing data, respectively. We emphasize that such performance was attained with single-input FLM only. These preliminary results indicate that FLMs have great potential to be explored in this domain, and perhaps can be further studied or adapted for the specific task of time series imputation.
Table 3: FLM imputation results on ETThl (Mean + SD over 10 seeds) against some of the performance of state-of oftte-tf- the-art models considered in Du et al. (0244) (** refers to the top performing and to the lowest ranked one).
MAE
Model
Mean (Standard Deviation)
10% Missing Rate
FLM
0.57 (0.15)
MRNN
Yoon et al. (018))
0.79 (0.02)
Mean
Du et al. 2024
0.74 (-)
Median
Du et al 2024
0.71 (-)
SAITS**
Du et al
2023
0.14 (0.01)
50%
Missing Rate
FLM
0.66 (0.18)
AutoFormer
Wu et al. (0211
0.98 (0.01)
Mean
Du et al. 2024
0.74 (-)
Median
Du et al. 2024
0.71 (-)
SAITS
0.22 (0.01)
C Numerical Results for Hyperparameter Selection
When numerically solving each PDE, we trained FLMss and other NN models for 18 27 different hyperpa- ramoter configurations, with a maximum of 10,000 epochs and a training loss tolerance of 10-4, All error metrics (MSE, MAE, Max Error) and the number of epochs required to reach the specified training loss tolerance were averaged over 10 runs with different random soeds. The performances of all tested models applied to each PDE are reported in Tables 5 266] Table 4 the number of trainable parameters in each model for different number of hidden units. The effect of the number of nub-networks in approximating the solutions of the PDEs is shown in figure Fig.
To select the best hyperparameter combination for each model applied to each PDE, we performed a hierar- chical statistical testing procedure. For each metric (MSE, MAE, Max Error, and number of epochs, in that order), we first tested for normality using the Shapiro Wilk test. If all hyperparameter sets were normally distributed (p > 0.05) in their respective metric, we applied one-way ANOVA followed by Tukey's HSD test for pairwise comparisons; otherwise, we used the nonparametric Friedman test followed by the Nemenyi test.
25

---

Published in Transactions on Machine Learning Research (12/2025)
Table 4: Number trainable parameters for different models. For FLM, N denotes the number of sub networks. For all other models, Nhiden denotes the number of neurons each hidden layer. For each row, parameter counts are reported in the same order as the listed N or Muidden values.
Model FLM
Hidden Units
Formula 6N
Trainable Parameters
N
{04, 16, 25, 49, 61)
{24, 96, 150, 291, 381)
SIREN
Nidden {06, 24, 37, 73, 93)
4Nhidden +
(25, 97, 149, 293, 373)
SIREN-2
Nhidden {03, 08, 10, 15, 17)
N hidden + 5Nhädden +
{25, 105, 151, 301, 375]
SIREN-3, V-ReLU,
Nhidden {02, 06, 11, 12)
2N hidden 6.hiideen
{21, 109, 141, 309, 361)
V-LReLU, V-Tanh
Heat Equation
Poisson Equation
10-3
10a
10-2
MSE
MSE
10
10
10"
10-7
10
149
36
196
N
N
(b)
GBS Equation
Burgers Equation
10a
0.035
0.03
0.025
10:2
AISE
MSE
0.015
10-
149
16
25
36
-1
64
81
100
121
144
169
196
14
9
16
25
49
81
100
121
144
160
196
N
NV
(c)
(d)
Figure 9: Effect on approximation MSE for the PDEs considered in Section 5 (shown in vertical axis in logarithmic scale) as N increases. All FLMs were trained using default learning rate and beta values of ADAM optimizer in PyTorch with maximum epochs of 10, 000 and loss tolerance of 10 10-4
We identified the with the lowest respective metric mean (or average rank in the nonparametric case) for the current metric and grouped hyperparameter configurations not significantly different (p 0.05 within the critical difference) from it. This top group was carried forward to the next metric, repeating the process sequentially. If multiple groups remained statistically equivalent after all metrics had been evaluated, we selected the one with the smallest network size as the tiebreaker.
With the best performing hyperparameter configuration ion (highlighted in each table below) selected by follow- ing the aforementioned procedure, we train the NN models for 30, 000 new additional epochs with a training loss tolcrance of 10 and report the error metrics for each PDE in their respective Tables 2a 2d
26

---

Published in Transactions on Machine Learning Research (12/2025)
Table 5: Heat Equation with FLM
N
LR.
Betas
MSE 1 SD
MAE SD
Max Error
SD
Epochs 1: SD
[0.9, 0.999)
(5.59 + 1.
[1.88 I 0.23)E-03
(8.54 = 1.58 1.58) 13-03
8611 1087
0.001
(0.95, 0.97)
(1.05 1: 0.26)E-05
(2.54 1 0.38)
(1.39 1 0-32)
2941 627
(0.97. 0.95)
(1.05 0.23)
(2.57 + 0.28)E-03
(1.34 = 0-23)
2802 275
[0.9, 0.999)
(5.03 1.36]E-06
[1.76 [1.76 E-2
2.70)2.08)E0
3870 12541
4 0.005
(0.95, 0.97)
(8.71 4..08) E-06
(2.36 +: 0.29)
(1.13 = 0.30
1173 + 649
(0.97, 0.95)
(7.42 2.88 2.88 E-06
[2.12 + 0.39)E-03
(1.13 = E-38)
1129 334
10.9, 0.999)
(1.89 1.39)
(1.75 E.22)
(8.41 = 2.09)
3233 1807
0.01
(0.95, 0.97)
(7.28 1: 2.43)E-06
[2.13 1: 0.44)E-03
(1.03 1: 0.33)
927 1 609
(0.97, 0.95)
(7.98 3.49) FF-06
(2.23 +: F.59
(9.66 F-29
1097 502
[0.9, 0.999)
(1.69 1: 0.46 E-06
[9.84 1: E.80
(6.72 : E.97)
2793 1: 592
0.001
(0.95, 0.97)
(2.22 0.72)
(1.13 0.18)
(6.34 E.34))-03
594 115
10.97, 0.95)
(2.86 + 1.10) E-06
[1.28 + 0.27)
(7.57 + E-19)
598 + 68
(0.9, 0.999)
[183 0.63]E-06
[1.04 0. 0.11)
(6.67 1 2.14]E-03
1005 253
16
0.005
(0.95, 0.97)
(2.81 + F-25
(1.31 + F.33
(6.29 + F-03
348 41
(0.97, 0.95)
(4.16 + 2.32) E-06
(1.60 + 0-19)
(8.01 = 2-07)E-03
381 - 37
(0.9, 0.999)
0.88)
[1.09 E-17)
1E-03
139
0.01
(0.95, 0.97)
(5.55 1: 3.13)E-06
(1.93 1 0.62)E-03
(6.87 1: 1.58 E-58)
295 1 29
(0.97, 0.95)
(5.71 + FF-0
(1.87 + 0.43)
(8.35 + F-94)
338 31
[0.9, 0.999)
(1.34 6.62)
[8.76 + 1.23)E-04
(8.09 E-70
2509 428
0.001
(0.95, 0.97)
0.67) E-06
(9.32 + 2.02)
(6.94 = 2.65 EE-03
191 + 45
(0.97, 0.95)
(1.44 E-47)
(9.79 L.54)E-04
(7.79 E-93)
25
[0.9, 0.999)
(1.05 1.370.0.))000
[1.89 1 1.100
(7.18 1 1.8
860 1 188
0.005
(0.95, 0.97)
(2.49 1.70) E-06
(1.19 E-41
(7-33 - E-75)
336 52
(0.97, 0.95)
(3.54 # 2.90) E-00
E-0G
[1.48 + E-003
(7.77 = E-00)E-03
353 + 35
10.9, 0.999)
[1.08 : 0.38) 16-06
[8.00 + 1.11/E-04
(6.80 = 4.21)E-03
568 : 158
0.01
(0.95, 0.97)
(6.16 E-96
(2.02 = 0.53)E-03
(8:27 = E-03
292 60
[0.97, 0.95)
(8.26 + 4.61) E-66
(2.30 + 0.71)E-00
(7.92 = 1.69) 1E-03
319 + 39
Table 6: Heat Equation with SIREN-1
Naiddon
LR
Betas
MSE 1 SD
MAE 1 SD
Max Error 1 SD
Epochs 1 SD
[0.9, 0.999)
[2.83 = E-58)
[1.01 = 1-1
(2.32 = 0.90)8-02
9999 0
0.001
(0.95, 0.97)
[1.11 0.27)
(2.60 + 0.34)E-33
(1.59 + 0.44)
6269 : 2156
(0.97, 0.95)
(9.97 1: 2-77)
(2.49 1 F-38
(1.35 1 0.24)E-02
5255 1 1420
(0.9, 0.009)
[114
[2.47 0:02
9444 1148
6
0.005
(0.95, 0.97)
(9.18 1.181...22)E.00
(2.39 1 0.33)E-03
(1.27 1: 0.30)EB-02
3904 1 2088
(0.97, 0.95)
(8.88 2.96
(2.37 F.43)
(1.12 0.26)
1019
[0.9, 0.999)
9.233
(2.11 = E-1
(1.22 = 0.44)E-00
8132 = 1932
0.01
(0.95, 0.97)
[8.64 4.76)
(2.27 + 0.74))-03
(1.12 = 0.32)B-02
3803 = 1919
(0.97, 0.95)
[1.06 1 0-2
(2.64 1 E.35)
(1.17 1. 0.26)EE-0
1938 753
(0.9, 0.999)
(2.08 + 1.19)
[3.43 + L.01/E-03
(2.26 - 0.60)E-02
9879 = 379
0.001
[0.95, 0.97)
(1.18 E-10
(2.67 + E-19
(1.84 + 0.335)
6588 = 1250
(0.97, 0.95)
[1. 0.13)
(2.61 + 0-27
(1.95 = 0.34)E-02
5990 = 811
(0.9, 0.999)
99.12 5.900 E-06
(2.23 1 E.74)
(1.52 - 0.37]
7008 : 2507
0.005
(0.95, 0.97)
(9.81 1 2.20 E-06
(2.33 1 F.
(1.85 1. 0.34)EE-02
5093 1462
(0.97, 0.95)
[LOI + 0-01)
(2.24 + F-20
(2.06 + 0-22)
3288 + 929
0.999)
1.02)-000
2,66 + E-03
(1.27 0.4888-8B11
6689 2821
0.01
(0.95, 0.97)
[1.07 1 0.29) B-05
(2.61 + E-55)
(1.54 + 0.34]B-02
4371 + 1796
(0.97, 0.95)
(1.20 1 E.05
(2.63 0.
(1.61 0.45)BB02
(0.9, 0.999)
(1.53 F-05
(2.96 F.03
(1.92 1 0-69)
9617 743
0.001
(0.95, 0.97)
(1.27 + E-02
(2.82 + 0. E-00
(1.83 + 0.34)E-02
7915 = 1354
(0.97,
[1.27 E.12
(2.79 E-25
(1.80 0-41)
6116 746
(0.9, 0.999)
(9.35 1 4.03)E-06
(2.43 I 0.55)E-03
(1.42 0.4618-22
7341 1 2075
0.005
(0.95, 0.97)
(1.13 1 E.05
(2.62 0.74)
(1.65 0-37)
2289
[0.97, 0.95)
(1.03 + 0-36)E-05
(2.45 + E-49
(L.6 + 0-52)E-02
4219 = 1018
(0.9. 0.999)
(1.00 1 E.50
(2.51 1 0.
(1.31 0.444]6-02
7143 1 2049
0.01
(0.95, 0.97)
(1.00 E-4
(2.59 - F-77)
(1.45 + 0.55)F-022
5468 2671
(0.97, 0.95)
(1.45 + 0.57)E-05
(3.01 - 0.79)E-03
(1.27 = 0.29)E-02
2810 = 1014
27

---

Published in Transactions on Machine Learning Research (12/2025)
Table 7: Heat Equation with SIREN 2
Miidenn
LR
Betas
MSE SD
MAE + SD
Max Error + SD
Epochs 1: SD
(0.9, 0.999)
[2.35 1.70
(3.65 1 1.4
(1.89 1 0.98)E-02
9999 +:0
0.001
(0.95, 0.97)
(5.87 E.70)
(1.79 1 0.80)E-03
(1.15 1 0.57)
7297 3557
(0.97, 0.95)
(4.12 + 2.24)
(1.55 0.47)
(1.01 + 0.32)
6144 + 2903
(0.9. 0.999)
[1.73 11.+1.11 1.18)--05
(3.25 4
(1.43 0.59)
9218 + 1647
3
0.005
(0.95, 0.97)
(7.03 + 6.13)E-06
(1.93 0.93
(9.41 + 3.58) E-03
5834 + 3126
(0.97, 0.95)
(5.84 6.57)E-06
(1.69 +.96
(1.11 + 0.60)E-02
5149 3420
(0.9. 0.999)
(7.23 80)80)
(3.82 6.09)
(1.38 + 1.44)
9959 + 126
0.01
(0.95, 0.97)
(9.54 8.34)E-06
(2.26 0.86)
(1.24 1: 0.82)E-02
6205 3561
(0.97. 0.95)
(4.86 1.75)F-06
(.31
(9.53 4: FF-03
5585 3913
(0.9, 0.999)
(1.90 + 0.99)E-06
(9.79 1 E.
(8.73 1 3.20 E:03
7476 1 1894
0.001
(0.95, 0.97)
(2.27 E.37)
(1.12 0.33)E-03
(7.86 E-00)
408
(0.97. 0.95)
(1.83 1.05)E-06
(1.02 + 0.29)
(6.76 + E-07)
1506 + 305
(0.9, 0.399)
(1.99 1.05JE-06
(9.85 1 2.31
[1.22 0.36 12.02
4380 1 1700
8
0.005
(0.95, 0.97)
(2.92 F-37)
(1.32 1 0.36]
(9.41 + F-78)
891 +: 300
(0.97. 0.95)
(2.97 + 1.11)E-00
(1.35 + 0-27)
(8.62 + 3.00 (00000)E-1 E-03
810 + 206
(0.9, 0.999)
(3.10 E-07
E-02
0.59) E-02
2569
0.01
(0.95, 0.97)
(7.20 1 E.06)
(2.25 1 E.36
(8,21 1 E-82)
853 1. 423
(0.97, 0.95)
(5.43 F-
(1.86 + 0.33
(9.76 +: F.30
652 +: 172
(0.9. 0.999)
[2.07 0.92]E-06
0.21))-0
(9.84 2.88)
7124 + 1546
0.001
(0.95, 0.97)
(2.67 = E-06
(7.16 + E-00)
1/887 + 398
(0.97, 0.95)
(2.32 0.72)E-06
0.24)
(8.62 E-05)
(0.9, 0.3999)
(2.70 1.701120 2.47)1:06
(1.7 1.177
(9.60 1 2.58)
4110 1110111 1315
10
0.005
(0.95, 0.97)
(4.48 E.57)
(1.76 0.41)
(9.67 + E-03
1051 353
(0.97. 0,95)
(4.16 + 1.71)E-06
(1.60 + 0.44))-03
(1.01 + 0.20)
901 + 235
(0.9, 0.399)
(2.01 0.84)E-06
(1.05 H 0.25]E-03
[8.77 1 2.49)E-03
2917 902
0.01
(0.95, 0.97)
(1.05 E-30)
(2.77 + 0.56]E-03
(1.11 = 0-29)
919 + 143
(0.97, 0.95)
(6.22 + 2.93)E-06
(2.06 + 0.62)
(8.96 + 1-66)
761 + 172
Table 8: Heat Equation with SIREN-3
Nyiddnn
LR
Betas
MSE SD
MAE 1 SD
Max Error 1 SD
Epochs 1 SD
(0.9, 0.999)
(1.16 3.58)E-05
(192 + E-90]--03
(3.08 + E.12)
9999 +
0.001
(0.95, 0.97)
(1.19 1 0.76)E-05
(2.65 1 E.81
0..7)
9729 + 853
(0.97, 0.95)
(5.48 14.99)F-05
(3.66 1 4.79)E-03
(1.89 1 2.52)
9080 1 1963
(0.9. 0.399)
[4.35 7.18)E-00
(4.51 13.8))63)
[2.10 1.13)-222
9999 t
2
0.005
(0.95, 0.97)
(6.72 1: E7-2
(4.41 5..11..1))..0
(2.21 1 E.02
9640 1 1135
(0.97, 0.95)
(6.41 F--4
(3.95 5.26
(2.00 + 2.67 2.07
8225
(0.9. 0.999)
(6,25 = 16. 12)E-06
(1.28 4.77)E-03
(2.12 E-02
9901 + 301
0.01
(0,95, 0.97)
(8.64 21-
(5.03 5.85)
(2.43 + E-78)
9576 1339
(0.97, 0.95)
(5.08 16.63)).05
(5.29 1 E-01
(2.68 1 3.14)
9507 1146
(0.9, 0.999)
(3.67 = E-0
(1.38 E-07
(1.17 + 0.39 E-39 E-02
6117 4 1818
0.001
(0.95, 0.97)
(3.80 1.34))-06
(.30
(1.11 + 0-40)--02
1233 + 519
(0.97. 0.95)
(2.49 1.14)E-06
(1.66 + E-23
(9.21 + 1-33
1199 335
(0.9, 0.999)
(5.09 4.76)E-00
(1.64 E-03
[1.04 1 0.31)
1714
6
0.005
(0.95, 0.97)
(5.26 2.47)E-06
(1.75 1 E.4
(1.00 1 0.29)
922 1 596
(0.97. 0.95)
25.24 + E-08
(183 + 0.57)
(9.33 + F.94
929 550
(0.9. 0.999)
(1.29 3.06
+ B-08
(1.05 0.33 E-02
2123 +
0.01
(0.95, 0.97)
(9.77 + E-94)E-06
(2.22 1 E.63
(1.08 + E-35
884 + 623
(0.97, 0.95)
(1.09 0.72)
(2.72 1 E.03
(1.11 1 0.24)
(0.9. 0.999)
(3.85 F-36
(1.48 FF-1
(1.18 1 F.02
7256 1: 1718
0.001
(0.95, 0.97)
(4.17 + E.72)E-06
(1.50 0.40)--03
(1.27 + 0.24)E-02
1415 + 621
(0.97, 0.95)
(3.31 E-06
(131 B-25
(1.22 + E-27
212
(U.9, 0.399)
(2.77 1.73)E
121770
(9.68 1 E.03
3584 1 1547
0.005
(0.95, 0.97)
(7.23 3.91
22.12 0.75
(1.28 1 F.02
436
(0.97. 0.95)
(6.76 + 4.71)E-06
(2.05 + 0.92)E-03
(1.05 + 0-40
1072 + 364
(0.9, 0.399)
(3.94 1 1.42)8.06
(1.49 1 0.32]E.03
(1.11 1 0.30
2740 L 1005
0.01
(0.95, 0.97)
(1.50 0-45
(3.28 1 0.65)
(1.38 + F.42
1201 734
(0.97, 0.95)
(1.23 + 0.82)E-05
(2.84 + 1.08)E-03
(1.16 + 0-29)
900 + 694
28

---

Published in Transactions on Machine Learning Research (12/2025)
Table 9: Heat Equation with Vanilla Relu
Naidoon
LR.
Betas
MSE 1 SD
MAE 1 SD
Max Error 1 SD
Epochs 1 SD
(0.9, 0.999)
(7.14 1.11)-22
[2.11 = 0.11)E-01
(7.00 = 0.74]]-001
10000 =
0.001
(0.95, 0.97)
(7.17 1.14)
[2.13 1 0.13)E-01
(6.84 1 0.-1)
10000
(0.97, 0.95)
(6.27 + 0.88)
(2.02 2 0.15
(6.58 4 0.51)
10000 +
(0.9, 0.999)
1.1)42.
0.1))=0
0.61E06)))-0
10000
2
0.005
(0.95, 0.97)
[2.08 + 0.11)E--0
(6.67 : 0.18)B-01
10000 +
(0.97, 0.95)
(6.42
2210)
(6.49 = B-51)
10000
(0.9, 0.999)
[6.91
(2.09 + 0.12)
(6.82 = 0.62)
10000
0.01
(0.95, 0.97)
(6.43 1.09)
(2.03 1 0.16)E-01
(6.50 1: 0.50)
10000 1:
(0.97. 0.95)
(6.44 1.10
(2.03 16)
(6.50 0-50)
10000
(0.9, 0.999)
(8.03 0.65)
[2.26 + E.09
(6.51 = 0.16]B-01
4973 3 3984
0.001
(0.95, 0.97)
(7.80 1 0.83)E-02
(2.23 0.10)
(6.52 0.18)E-01
5508 4738
(0.97, 0.95)
(8.17 + 0-15
(2.25 + 0.07)
(6.60 + 0.53)
4297 + 4030
(0.3, 0.999)
T.34100.86
(2.23 1 0.11)E001
(6.56 1 0.14]E-01
6297 1 4786
0.005
(0.95, 0.97)
88.04 F-62)
(2.25 + F.07)
(6.59 + 0.24)F-01
6181 4929
(0.97, 0.95)
(8.03 = 0.63) E-02
(2.21 = 0.07)
(6.59 = 0-26)
6852 = 4256
(0.9, 0.999)
(7.68 B-8
[2.20 0.13)E-01
(6.61 B-4
4753
0.01
(0.95, 0.97)
(8.19 1 0.17)
(2.26 1 0.07)E-01
(6.68 1 0.52)E-01
4958 1 4232
(0.97, 0.95)
18.28 + 0.26
(2.25 0.07)
(6.91 0.73)
7183 + 4536
[0.9, 0.909)
0.07
[2.29 + 0.02
(6.46 0.10)--0
5614 3938
0.001
(0.95, 0.97)
(8.23 0.05)
(2.29 - 0.01)
(6.12 1.120.100 0.16)
917 + 329
(0.97, 0.95)
(8.20 E-0
[2.28 0.01)E-01
(6.43 0.16)B-01
2790
(0.9, 0.999)
(8.21 0.08)
[2.28 1 0.02)E-01
3628 1. 3880
0.005
(0.95, 0.97)
(8.24 E-14
(2.29 + 0.02)
(6.42 0.18
1921 3169
(0.97, 0.95)
(7.97 + 0.72)
(2.21 + 0.11)E-01
(6.63 = 0-19)--01
4461 = 4786
[0.9, 0.999)
(7.98 1 0.70
(2.25 H 0.08)E-01
(6.39 0.11]E-01
2533 = 3213
0.01
(0.95, 0.97)
(8.13 E-17
(2.26 + 0.05)E-01
(6.49 0.19
3401 4438
(0.97, 0.95)
(8.20 + 0.18)E-02
(2.26 + 0.07)E-01
(6.60 = 0.59)B-01
4811 + 4720
Table 10: Heat Equation with Vanilla Tan
Naiddon
LR
Betas
MSE 1 SD
MAE 1 SD
Max Error 1 SD
Epochs
SD
(0.9, 0.999)
(8.63 = 20-28)E-03
[5.32 1 5.69)E-02
(2.14 1-62]E-01
9999
0.001
(0.95, 0.97)
(9.75 :: E1-4
(2.06 1:06:1.40 1.41
(9.93 4 E-48)E-02
9999 +
(0.97, 0.95)
(1.09 1: 1.20)
1.1331..33
(1.07 1 0.75)
9999 1
(0.9, 0.008)
(8.79 20.22)E-00
6.444
(2.22 1.69]--01
9009
2
0.005
(0.95, 0.97)
(1.79 1: E.48)
(3.00 1.63)E-02
(1.43 0.777)--
9999 1 0
(0.97, 0.95)
(1.25 + F-16)
(2.46 1.23
(1.25 0 0.70
9909 +
(0.9, 0.999)
(2.03 = 1,43)E-03
(3.29 1-4)
9999 = 0
0.01
(U.95, 0.97)
(1.22 + E-49))-U3
[2.33 + E-69)E-02
(1.12 2 0.73)E-01
9999 + 0
(0.97, 0.95)
(7.94 1: 10.03)E-04
(1.85 1.24
(1.03 0.66)E-01
9999
(0.9, 0.999)
(5.81 2.61EEE6
(1.76 E-03
(1.31 2
5322 - 1491
0.001
(0.95, 0.97)
(5.86 - 3.11EE000
[1.75 O.
(1.47 0.49)
1628 - 697
(0.97, 0.95)
(6.80 = 2.3
(1.95 0.
(1.52 1 0-41)
1733 - 9419
(0.9, 0.999)
(9.86 11.34)-6
[2.32 1.
11.99
3072 : 1540
0.005
(0.95, 0.97)
(9.88 1 6-36)
(2.47 E.0)
(1.46 0-30
1410 1 884
(0.97, 0.95)
88.06 - 3.56)
(2.18 0.64)E-03
(L.66 + 0.37)
1299 + 1080
(0.9, 0.999)
2-3
[1.61 E-00
(1.35 1 E-02
2110 1195
0.01
(U.95, 0.97)
(1.67 - E-9
(3.41 + E3-33
(1.64 1 U.54)E-02
1089 + 589
(0.97, 0.95)
(2.57 1: E.
(4.17 E.03
(1.71 0.19)
921
(0.9, 0.909)
(5.70 1: F-72)
(1.80 1.60+00.30 0.31
(1.38 F-36
4635 1 2253
0.001
(0.95, 0.97)
(6.35 2.42
(1.82 - 0. 044
(1.53 + 0-35)
1539 = 726
(0.97, 0.95)
(8.15 E-0
(2.12 14/E-03
(1.66 E-28
1340
(0.9, 0.999)
662211877E8
(1.66 1 0.34)
(1.40 U.36]E-02
1771 1 463
0.005
(0.95, 0.97)
(1.07 E-50
(2.59 F.91
(1.57 0.4
797
(0.97, 0.95)
(1.31 = 0.47))-05
(2.93 + E.87
(1.70 1.70+000 0.24)B-02
717 + 143
(0.9. 0.909)
(4.84 1 1.141E-06
[1.58 L 0.33)
(1.41 1031]E
1565 1. 763
0.01
(0.95. 0.97)
(1.74 0.71)E-05
(3.54 1.02/F-03
(1.57 0.38)F-02
745 + 304
(0.97, 0.95)
(2.13 = 0.59)E-05
(3.85 + 0.77)E-03
(1.81 + 0.38)E-02
597 = 157
29

---

Published in Transactions on Machine Learning Research (12/2025)
Table 11: Poisson Equation with FLM
N
LR
Betas
MSE 1 SD
MAE + SD
Max Error 1 SD
Epachs 1: SD
(0.9, 0.999)
(9.38 :: 1446)
(2.88 = E.20)
(5.59 1 0.67]E-02
10000 1
0.001
(0.95, 0.97)
11.71 E-32)
(3.23 1.45)E-03
(1.30 1 0.22)E-02
3817 2113
(0.97. 0.95)
(.36
2.50+0.433)--0
(1.52 2 0.25)FF-2
3307 + 619
(0.9, 0.999)
2.9)-55
1.7)-
(1.67 L.6780/1))B-0
8981 1277
4 0.005
(0.95, 0.97)
0.49
814 + 93
(0.97, 0.95)
0.
(3.85
E-0
(1.50 0.14)
874 + 49
(0.9. 0.999)
(1.37 E-82)E-05
1.104=144
(1.28 + 0.31)E-02
6194 + 2428
0.01
(0.95, 0.97)
(1.86 1: 0.50)E-05
(3.34 E36
(1.43 1: 0.17)E-02
630 1: 60
(0.97, 0.95)
(2.21 FF-05
(3.67 F-92)
(1.33 1: 0.33)
667 24
(0.9, 0.999)
(9.28 1: E.97)
(2.27 - E.8
(1.34 1: 0.56]E-02
9988 + 32
0.001
(0.95, 0.97)
6.16.1
(2.03 E-0
(1.14 0.47)
1785 528
(0.97, 0.95)
(T.96 + E.61)E-06
(2.15 = E-89)
(1.12 + 0.43)
1279 + 229
(0.9, 0.399)
4.53
(1.5 0.35)
(9.67 1 2.88JE.0
5865 365
16
0.005
(0.95. 0.97)
(3.67 2.58)
(1.36 F-4
(7.35 + F-80)
1069 154
(0.97. 0.95)
(1.93 1.22) E-06
(1.77 = E-28)
(8.12 1-68)E-03
957 - 181
(0.9, 0.999)
2.29)
E-04
(9.72 E.02]E-03
4471 363
0.01
(0.95, 0.97)
(6.38 3.11)E-0
(2.04 1 E-50)
(6.95 1. 1.81
1129 1. 151
(0.97. 0.95)
(8.11 4.31)
(2.34 - F-7
(8.72 4: 2.76
+
(0.9, 0.999)
2.01)
(2.08 E-3
(1.23 0.25)
8751 672
0.001
(0.95, 0.97)
(1.58 = 0.32)E-03
(9.56 + E-03
1412 - 314
(0.97, 0.95)
(1.78 0.46))-03
(1.20 E-32
1508 429
(0.9, 0.3999)
(5.10 1.93119300.00
(1.68 1 0.23)E-03
(1.02 1 0.16]E..2
4796 1 532
0.005
(0.95. 0.97)
(5.08 2.47
(1.71 - F-50)
(8.31 + 2.58)E-03
1491 493
(0.97, 0,95)
(5.51 2.89)
(1.74 0. E-46)
(9.85 + 3.09)E-03
1632 + 632
(0.9, 0.3991
[4.99 2.283-06
(1.65 = 0.38)E-DS
(9.83 1 2.67JE-0
3632 1 337
0.01
(0.95, 0.97)
(2.91 0.94)E-03
(8.42 E-68)
2132 626
(0.97, 0.95)
(8.97 + E.42)
(2.40 = 0.86)E-03
(9.77 + E-26
2039 + 780
Table 12: Poisson Equation with SIREN-1
Nyiddnn
LR
Betas
MSE SD
MAE 1 SD
Max Error 1 SD
Epochs 1 SD
(0.9, 0.999)
(2.07 = 1.77)E-03
(3.55 + 1-62]
(1.70 + 0-82)
10000 + 0
0.001
(0.95, 0.97)
(4.33 10. 44)-45)
(3.57 + E.1)
(1.79 1 1.85)
8115 + 2465
(0.97, 0.95)
(9.23 1: 10.75)F.006
(1.99 1 1.26)E-03
(1.01 1 0.60)
6335 2371
(0.9. 0.939)
[7.71 6.69)]E-0
0.6333337)6-0
(3.30
E-38
10000 + 0
6
0.005
(0.95, 0.97)
(2.14 3.11)E-00
(2.96 1.331)EE3
(1.49 1: E.12)
8946 1 1519
(0.97, 0.95)
(9.50 F-04)
(2.37 0-64
(1.12 F.02
2573 1848
(0.9. 0.999)
(7.35 = 5.61)E-05
66.37 +3.1)-633
(3.27 = E-61)
9996 + 14
0.01
(0,95, 0.97)
(1.04 100+110 E.14)E-05
1.275
(1.05 + 0.71)E-02
8550 + 2006
(0.97, 0.95)
(1.25 0.65)
1.966
(1.09 1 E-31)
1553 861
(0.9, 0.999)
(2.29 1.99122 1.27)E-01
1123
(6,26 + 1.95) E-96) E-00
10000 + 0
0.001
(0.95, 0.97)
(9.00 + 6.54)E-06
(2.11 1.00)E-03
(1.16 0-4))E-02
10000 0
(0.97. 0.95)
(1.95 1.05+1.0 E.33)E-06
(9.99 + E.43)
(3.38 + E-72)
10000 + 0
(0.9, 0.999)
(3.66 1.811338
(4.53 22722E--0
[1.65 1 0.60)
10000 0
24
0.005
(0.95, 0.97)
(7.38 E.09)
(2.18 1 0.53)E-03
(8.38 1 E.03
10000 0
(0.97. 0.95)
(2.26 + 2-04)
(1.9 1 0.40)
(4.92 + 2. F-00
10000 +: 0
(0.9, 0.999)
E-58)
5.07 E-00
(1.T1 0.50))-02
10000 0
0.01
(0.95, 0.97)
(1.00 + E-05
(2.11 + E-02
(9.41 : 1.78) E-03
10000 1: 0
(0.97, 0.95)
(5.09 3.03)
(187 1 0.68
(5.29 E-43
9256
(0.9. 0.999)
(1.30 F.45
(9.53 1 1--0)
(4.63 + F.02
10000 0
0.001
(0.95, 0.97)
(1.03 + 0-92)
(2.44 + 1.30)E-03
(1.30 + 0.56 E-002
10000 + 0
(0.97, 0.95)
(2.80 14)E-06
(1.13 B-8
(5.86 E-73
10000
(U.9, 0.399)
(2.69 1 60)0
(591
(1.89 1 0.77
10000 1:0
37
0.005
(0.95, 0.97)
(6.45 3.84)
(2.06 0.82]E-00
(6.91 1 1.62)
10000
(0.97. 0.95)
(2.40 + 1. E-00
+ (123 0.40)6-03
(5.01 + 1.74)E-03
10000 + 0
(0.9, 0.399)
(1.99 1.44)..005
(3.44 1.34JE.03
(1.65 1 0.48)E.02
10000 1
0.01
(0.95, 0.97)
(1.86 2 0.88
(3.67 4 1.09)
(9.49 + F.03
10000
(0.97, 0.95)
(1.15 + 1.24)E-05
(2.77 + 1.50)E-03
(6.54 + 2.96)
9618 + 508
30

---

Published in Transactions on Machine Learning Research (12/2025)
Table 13: Poisson Equation with SIREN 2
Miideen
LR
Betas
MSE SD
MAE + SD
Max
Error 1 SD
Epochs 1: SD
(0.9, 0.999)
(2.00 1.53)E-03
(3:20 1 1.39]E-02
(1.58 1 0.78)E-01
10000 + 0
0.001
(0.95, 0.97)
(2.56 E-46
(1.13 0.57)
(5.68 1 2.89)E-02
10000 0
(0.97. 0.95)
(2.84 + 2-30
1.27:0.455)-00
(5.78 + 1.92]
10000 + 0
(0.9, 0.999)
6.00)
0.72)--2)
(8.87 E-02
10000 0
3
0.005
(0.95, 0.97)
(2.48 + 2.81)E-04
(1.10 E.0
(5.61 + 2.6)).66))E44
10000 + 0
(0.97, 0.95)
(1.63 = 0.06))-04
(9.78 + 2-40]
(4.92 + E-53)E-02
10000 0
(0.9. 0.999)
(6.81 = 11.74)E-01
(1.74 E.39)E-02
(7.27 + E-02
10000 + 0
0.01
(0.95, 0.97)
(1.78 0.81)E-04
(1.05 1: 0.28)
(5.04 +: 1.76)E-02
10000 1: 0
(0.97. 0.95)
(2.89 2.75)
(1.20 1 0.70
(5.46 +: F.08)
10000 0
(0.9, 0.999)
(1.61 + 0.
99.70 1 3.16]E-03
(5.49 1 2.05 E.05) 02
10000 1: 0
0.001
(0.95, 0.97)
(9.87 6.17)
88.14 2.69)
(4.29 E-01
10000 0
(0.97. 0.95)
(9.33 + 5.82)E-05
7 7,86 + E.
(3.94 + 1. 28 E-02
9818 + 575
(0.9, 0.399)
[1.4 0.83)
9.61 1 2.97
[4.97 1.49)..02
10000 1:
8
0.005
(0.95. 0.97)
[1.08 F-01
(8.58 + 2.22]
(4.51 + 1.11
10000 0
(0.97. 0.95)
(1.04 = E-50)
8850 + E-44)
(4.16 + (.32)--02
10000 + 0
(0.9, 0.999)
(1.19 1-22)-001
E-91
(3.63 1.46)
10000
0.01
(0.95, 0.97)
(1.09 1 E-52)
(8.50 1 2.69)
(4.28 1: 1.59)E.02
10000 1 0
(0.97, 0.95)
(9.18 F-45
(7.62 3.64)
(3.56 +: F.49
10000 0
(0.9. 0.999)
2..77
(5.00 0.97)
10000 0
0.001
(0.95, 0.97)
(1.12 0.68)
(8.61 E-04
(4.55 : 1.04)E-02
10000 + 0
(0.97, 0.95)
(8.47 4.48)E-05
(.48
(4.09 0.74)
10000
(0.9, 0.3999)
(1.77 1 0.98)1.04
1.04000.116.12
(5.54 1.54111.35J...00 1.35)E.02
10000 1: 0
10
0.005
(0.95, 0.97)
(9:95 6-78)
3..4443444)
(4.33 + E-20)
10000 0
(0.97. 0,95)
(7.46 + 5. 48)E-05
(6.91 + 3.05))E-03
(3.59 + 0-91)E-02
10000 + 0
(0.9, 0.999)
(1.29 1. 191---1
(7.75 4.85JE-03
[3.92 1 1.91)E-02
10000 3 0
0.01
(0.95, 0.97)
(8.01 E-90)E-05
(7.00 2-77
(3.65 + E-40)E-02
10000 0
(0.97, 0.95)
(5.53 + E-36)
(5.65 E.26)
(2.91 + E-22)
9718 + 891
Table 14: Poisson Equation with Vanilla Relu
Naiddon
LR
Betas
MSE 1 SD
MAE 1 SD
Max Error :: SD
Epochs
1 SD
(0.9, 0.999)
(2:45 0.00) E-00
(3.97 0.00)
(1.00 = 0.00]B++00
10000
0,001
(0.95, 0.97)
(2.45 E-00
(3.97 + 0.00)E-01
(1.00 : 0.00)E+000
10000
(0.97, 0.95)
(2.45 0.00
(3.97 1 0.00)E-01
(1.00 1 0.00)E++00
10000
(0.9, 0.009)
0.000
(3.97300 0.00)E-01
(1.00 2 0.00]8:-00
10000
2
0.005
(0.95, 0.97)
0.00
(3.97 1 0.00)E-01
(1.00 1 0.00)EE+00
10000 1
(0.97, 0.95)
0.00
(3.97 + F.00
(1.00 + 0.00
10000
[0.9, 0.999)
0.045 E-01
[3.97 = E-01
(1.00 = 0,00]E+000
10000 0
0.01
(0.95, 0.97)
0.00)
(3.97 + 0.00))E01
(1.00 = 0.00)B++00
10000 +
(0.97, 0.95)
(2.45 1 0.00) E-01
(3.97 1 0.
(1.00 0.00)E++0
10000
(0.9, 0.999)
[3.97 2 E-00
(1.00 = 0.01)E+000
10000 + 0
0.001
[0.95, 0.97)
0.00
(3.97 + E-00)
(1.00 = 0.01))++00
10000
(0.97, 0.95)
(2.45 0.06)
(3.97 + 0.05)E-01
(9.99 = 0.11))-011
10000
(0.9, 0.999)
2.45
(3.97 1 0.05)E-01
(9.99 = 0.13]]B-01
10000
0.005
(0.95, 0.97)
(3.97 1 0.05)E.00
(9.99 0-
10000
(0.97, 0.95)
22.45 1.45+0.10 0-14
(3.97 + O. 12)
(9.99 + 0.36)
10000 +
(0.9, 0.999)
0.00
(3.97 + 0.07)
(1.00 0.077
0.01
(0.95, 0.97)
22.45 1 0.01)
(3.97 + 0.D4)E-01
(1.00 = 0.08)B+-00
10000 + 0
(0.97, 0.95)
22.45 1 0.13)
(3.97 1 0.11)E.001
(9.99 0.32)BB-01
10000
(0.9, 0.999)
(2.45 1 0-02)
(3.97 1 0.01)F-011
(1.00 1 0.03)TE+00
10000
0.001
(0.95, 0.97)
(2.45 + 0.05)
(3.97 + 0.04)E-01
(9.99 = 0.12)E-01
10000 + 0
(0.97, 0.95)
(3.97 0.05)E-01
(9.99 0.19)E-01
(0.9, 0.999)
0.066
(3.97 I 0.05)E-01
(9.99 = 0.12)B-01
10000 1
0.005
(0.95, 0.97)
(2.45 0.01
(3.97 1 0.02)
(1.00 0.
10000
[0.97, 0.95)
(2.45
(3.97 + 0.01)E-01
(1.00 + 0.0
10000 + 0
(0.9. 0.999)
(2.45 0.,01-.0
(3.97 1 0.01)E.011
(1.00 1 0.01]E+000
10000 1
0.01
(0.95, 0.97)
2
(3.97 - 0.02)E-01
(1.00 - 0.02
10000
(0.97, 0.95)
(2.45 + 0.00))--1
(3.97 = 0.02)E-01
(1.00 = 0.02)E+000
10000 - 0
31

---

Published in Transactions on Machine Learning Research (12/2025)
Table 15: Poisson Equation with Vanilla Tan
Niideen
LR
Betas
MSE 4 SD
MAE = SD
Max Error = SD
Epochs + SD
(0.9, 0.909)
2.17
(7.79
(2.15 = 0.67)E+00
10000 =
0.001
(0.95, 0.97)
(9.87 E-00
(8.33 E-2
(2.30 0.00)
10000
(0.97, 0.95)
(8.91 ---1)
(2.02 = 0036)8+000
10000 + 0
(0.9, 0.999)
1.11
[7.65 0.300)00
(2.00 0.443]
10000
2
0.005
(0.95. 0.97)
(8.85 2.13)_01
(7.91 + 0.87)FF00
(2.21 + 0.09))
10000 + 0
(0.97. 0.95)
(9.01 +: E-00))
(7.94 : 0.84)E-01
(2.35 1: 0.76)B++00
10000 +
[0.9. 0.999)
3.2111.211J.1
[7.63 1 0.38)
(2.07 1 0.56]
10000
0.01
(0.95, 0.97)
(8.12 = 2.17)-01
(7.61 + E.99)E-01
(2.01 = 0.45)E+000
10000 + 0
(0.97, 0.95)
(7.89 2.21)
(7.58 E-03)E-01
(1.87 0.19)
10000
[0.9, 0.909)
(3.29 + E-03
[4.93 - E.-)
(2.39 = 0.22)
10000 - 0
0.001
(0.95, 0.97)
(1.14 E-0
(2.68 + E-08)--02
(1.50 = 0.58)E-01
10000
(0.97. 0.95)
6.43
(1.81 F-05
(1.14 + 0.69)
10000
(0.9, 0.999)
(3.58 - E.00)
(1.93 = 0.46]E-01
10000 = 0
0.005
(0.95, 0.97)
[.211
(2.74 E-3
(1.45 0.64)
10000
(0.97, 0.95)
(6.72 1 4.79)
(2.00 1 0.70)E-22
(1.26 1 0.51)B801
10000 1
(0.9, 0.509)
2.0231.133
[3.69 - 1.20
(1.30 0.57
10000
0.01
(0.95, 0.97)
[1.27 + 0.81)
(2.85 - E-02
(1.63 = 0.39)E-01
10000
(0.97. 0.95)
[6.35 + E-62)
(1.93 - B-85)
(1.08 = 0.68)B-01
10000 +
(0.9, 0.999)
0.17
[4.66 1 0.71)E.02
(2.19 0.22)B-01
10000
0.001
(0.95, 0.97)
[1.18 1 0.72)
(2.76 1 1.01)E-02
(1.46 1 0.59)E-01
10000 1 0
(0.97, 0.95)
(4.18 F-46
(1.45 + F-06
(8.28 F-0
10000
[0.9, 0.999)
(2.47 = E-61)
(1.18 0.07
(2.19 = 0.27)E-01
10000
0.005
(0.95, 0.97)
(9.37 + E-04
(2.26 + 1.34))-02
(1.23 = 0.74)B-00
10000 +
(0.97, 0.95)
(5.58 1 E-94
(1.59 1 1.28)
(1.00 i 0.56)E-01
10000
(0.3, 0.309))
(3.52 1 0.96))2-02
(1.92 2 0.39)
10000
+ 0
0.01
(0.95, 0.97)
(9.58 1 E.35)
(2.40 1.20)
(1.30 0.53)B-01
10000
(0.97, 0.95)
(4.72 1 4.84)/-00
(1.57 1 0.97)
(9.99 + 5.85)
10000 + 0
Table 16: GBS Equation with FLM
N
LR.
Betas
MSE + SD (8.34 1 2.24)E.06
MAE SD
Max
Error SD
Epochs + SD
(0.9, 0.999)
(1.86 1 0.27)
(2.41
1 E-02)
10000 1 0
0.001
(0.95, 0.97)
0.64)
[9.28 + F.57)
(1.28
R-52
4781 1334
(0.97, 0.95)
0.50
(8.92 E-07)
(1.19
+ E-38
3208 619
(0.9, 0.999)
(6.39
[1.66 0.98)E-03
(1.36 - 0-57)
9965 111
25
0.005
(0.95, 0.97)
(1.75 1: E.46)
(1.24 1 0.48)
5116 2025
(0.97, 0.95)
(1.57 1.32)-6
(1.72 + 0.28)
(1.11 + 0.36)
3985 + 1380
[0.9, 0.999)
3.226)
[2.52 E.40
(1.56 0.49)E-02
432
0.01
(0.95, 0.97)
(2.31 2.21)
(3.62 1: F.58)
(1.25 1 0.40)EE-2
6768 2400
10.97, 0.95)
(1466 E-00
[3.03 + E-08
(1.11 - 0.31)
5732 1903
(0.9, 0.999)
(9.73 8.01)
(1.87 1 E.68)
(2.05 1 0.57)
10000 1 0
0.001
(0.95, 0.97)
(2.59 + 10.11.15))... F--06
(1.01 + 0.24)F-00
(1.22 + 0-30
4739 833
(0.97, 0.95)
(2.19 E-42)
(9.50 2-28
(1.03 0.24)E-02
3252 433
[0.9, 0.399)
[2.20 E 1:80
(1.26 1 0.34
9473 1 960
0.005
(0.95, 0.97)
(6.91 2-499
(2.09 + E-40
(1.04 - 0-30)
5493 1427
(0.9T, 0.95)
2.000 E-06
(1.92 + 18)
(1.16 = 0.49)
3947 1313
[0.9, 0.999)
(2.27 1.11)E-6
[9.54 1: 2.37)
(1.06 1 0.40)
7291 935
0.01
(0.95, 0.97)
(1.64 1: FF91)
(3.14 F.94
(1.14 0.27)
8876 1423
(0.97, 0.95)
(2.48 + 1.45)E-05
(3.98 + 1.27)E-03
(1.16- - 0.21)
7146 - 1947
Table 17: GBS Equation with SIREN
Miiden
LR
Betas
MSE = SD
MAE SD
Max Error + SD
Epochs = SD
(0.9, 0.999)
(2.38 + 1.53))-004
(1.17 0.43)
(8.47 +: F.20
10000 1 0
0.001
(0.95, 0.97)
(5.76 1.92)E-00
(5.65 0.86)
(5.07 1: E-06
10000
(0.97, 0.95)
(8.13 +: 4.64)F-05
(6.78 2.00)
(5.22 4: F-32)
10000 + 0
(0.9. 0.999)
[2.24 = 2.30)E-04
(..6440.60)
[6.02 - 2-22
10000 0
37
0.005
(0.95. 0.97)
(9.91 = 1.64)E-05
(7.63
(5.13 + E-02
10000 + 0
(0.97, 0.95)
(7.80 2.61)
(6.82 1 1.25)
(4.91 +: 0.79)E-02
10000 0
(0.9. 0.999)
(1.55 1.45:22000 2.62
(5.36 1. 1.
10000 1 0
0.01
(0.95. 0.97)
(1.86 + 1.10)
(1.08 0.37)
(5.37 + E-97)
10000 + 0
(0.97, 0.95)
(1.12 0.44)
(8.61 1.93
(4.81 1: E.2
10000
(0.9. 0.999)
(1.74
(9.92 1 3.0
(6.11 + 1.68)
10000 : 0
0.001
(0.95, 0.97)
(6.89 1.74)E-05
(6.25 1 0.79)
(4.35 1 0.52)E-02
10000 1: 0
(0.97. 0.95)
(6.24 1.53)F-00
(5.92 1 F--0
(4.11 + F.02
10000 0
(0.9. 0.999)
(2.48 3.87)
(1.06 + 0.98)6-02
(4.06 + 1-47)--02
10000 + 0
73
0.005
(0.95, 0.97)
(1.20 1.11
E-06
(1.61 + E-02
10000
(0.97, 0.95)
(8.68 + 4.94)E-05
(7.10 1 E-03
(3.91 1 0.63) 0.63)E-02
10000 1 0
(0.9. 0.999)
(8.81 = 17
(1.98 + E-09
(5.62 + 2.97
10000 + 0
0.01
(0.95, 0.97)
(2.58 + 1.16)
(1.32 1 0.28)E-02
(5.93 : E-06)
10000 1 0
(0.97, 0.95)
(1.59 1 1.08)E.04
(9.51 1 3.68)E-03
(4.32 1 0.59)E-02
10000 1 0
32

---

Published in Transactions on Machine Learning Research (12/2025)
Table 18: GBS Equation with SIREN-2
Naidden
LR
Betas
MSE 1 SD
MAE : SD
Max Error
SD
Epochs 1 SD
(0.9, 0.999)
(9.49 = E-000
[1.71 1-7
(1.47 = E.31
9866 = 302
0.001
(0.95, 0.97)
(2.05 1 E-80)
[1.08 + E.25)E-03
(1.05 0.62)
3179 : 1260
(0.97, 0.95)
(2.56 +: 0.68)
(1.21 + 0.11)E-03
(102=+.59))-00
3176 + 718
(0.9, 0.999)
(2.04 E-9
[1.04 0.24))E055
4.561:445211-03
7790 1604
10
0.005
(0.95, 0.97)
(2.35 1 E.10)
(4.09 1: 1.28)E-03
0.30100300
3765 1 1829
(0.97, 0.95)
(2.07 +: 0.89)
(3.67 + 1.15)
(1.59 0.48)
3491 + 1829
(0.9, 0,999)
(.09
[1.41 = E.93)E-03
7108 = 2632
0.01
(0.95, 0.97)
(7.58 + E-12)
(7.01 + 2.52)E-03
0.65)-02
5043 + 2902
(0.97, 0.95)
(7.96 1: 3.28)E-05
(7.63 1 1.00)E-03
1.1110.51
3415 1 2026
(0.9, 0.999)
(4.00 1: F-04
(1.29 1 E-16)
6.21)
9856 1 295
0.001
(0.95, 0.97)
(3.1 + E-30
(1.38 + 0.28)E-03
(9.06 5.31)
3452 + 2239
(0.97, 0.95)
(291+ 1.62) E-0G
(1.28 + B-3
2576 = 883
(0.9, 0.999)
(2.29 1 1.68)
(1.09 I 0.44)E-
(7.43
7266 : 1114
15
0.005
(0.95, 0.97)
(3.14 1 2.61)
(4.49 1 L.89)E-03
1.33
3254 1 1314
(0.97, 0.95)
(4.05 + 1.33)
(5.70 - 1.17)E-03
(1.55 1.03
3061 - 1707
(0.9. 0.999
(1.82 70
[1.38 01000E-003
(1299
7723
0.01
(0.95, 0.97)
(0.16 + 1-)
(8.03 + 2.07)
(2.05 0.35)
3608 + 1437
(0.97, 0.95)
(1.17 = 0.46)E-04
(9.12 = 2.27)
(2.26 0.35)E-02
2772 = 1414
Table 19: GBS Equation with Vanilla Leaky Relu
Miiden
LR.
Betas
MSE + SD 66.43 E-02
MAE + SD
Max Error =
Epochs + SD
(0.9, 0.999)
0.
2.166-111
10000
0.001
(0.95, 0.97)
(6.39 : B-52)
E.10)E-01
+.24)
10000 = 0
(0.97, 0.95)
(6.71 E.96
(1.80 0.10)E-01
(1.02 1:2::00 0.22)
10000
(0.9, 0.909)
6.31
[1.22
2..6222.40)1--
10000 1 0
0.005
(0.95, 0.97)
0..23
(1.70
(9.88 2.59)E-01
10000
(0.97, 0.95)
(6.31 0.81)
0.10)
(8.91 = 2-0))E-01
10000
(0.3, 0.999)
[1.74 0.08)E-00
(8.26 1 1.16]E-01
10000 1
0.01
(0.95, 0.97)
(1.76 0.12)E-01
(8.13 0.98)
10000
(0.97, 0.95)
(6.51 + 0.69)
(1.72 1 0.11)E-01
(1.03 + 0.31)8+-00
10000 + 0
[0.9, 0.999)
G.47
[1.67 -.11
(1.03 = 0022)BB-000
10000
0.001
(0.95, 0.97)
(1.61 + E.0)
(1.13 = 0.26)E--00
10000
(0.97, 0.95)
(7.06 1.05)
(1.73 1 0.11)E-01
(1.23 1. 0.14)B++00
10000
(0.9, 0.008)
(0.64 0.
(1.74 0.12)00001
(1.01 0.23)
10000
11
0.005
(0.95, 0.97)
(6.65 1 0.84)E-02
(1.72 1 0.12)E-01
(1.05 1 0.29)B++00
10000 1 0
(0.97, 0.95)
(6.69 + F-02
[1.70 + 0.14)
(1.12 0.29)
10000
[0.9, 0.909)
[1.78 + 0.14)E-01
(0.39 = 1.69]6-01
10000
0.01
(0.95, 0.97)
(6.87 + E-02
(1.78 - 0.18))-01
(1.03 = 0.31)
10000
(0.97, 0.95)
(6.43 : E-02
(1.73 1 0.12)E-01
(2.07 : 3.33)E+000
10000
Table 20: GBS Equation with Vanill Tan
Nniddon
LR.
Betas
MSE + SD
MAE + SD
Mnx Error
SD
Epochs 1 SD
(0.9, 0.999)
(3.76 + 2.32) E-06
[1.44 : E.57)E-03
(2.30 : 2.17]E-22
7482 1 446
0.001
(0.95, 0.97)
(4.24 2.81
[1.50 1 0.58)
(2.14
1402 136
(0.97, 0.95)
(7.57 + 3.89)
(2.13 0.58
(L144
1184 + 67
[0.9, 0.999)
1.10
[2.12 2.01)-33
(2.37 = 1.97E--2
3321 119
0.005
(0.95. 0.97)
1-08
1.27)
(1.54 0.97))-002
618 98
(0.97, 0.95)
(2.85 0.89)
(1.30 = 0.54)E--0
626 + G1
(0.9, 0.999)
2.03
[1.38 0.40)E-33
(1.92 + 1.88)B-02
0.01
(0.95, 0.97)
(3.68 2.433)E-05
(4.96 1 2.24)E-03
(1.77 1 0.66)E-02
540 1: 126
(0.97, 0.95)
(3.79 F-34)
(5.33 + 1.30)
(1.50 0.34))F-02
536
(0.9, 0.999)
(1.26 + 0.42)E-03
(1.80 1 1.05)E-02
5829 1 656
0.001
(0.95, 0.97)
(5.37 + 1.69)
(1.81 + 0-32)E-03
(1.37 + 1.02)E-02
931 + 15
(0.97, 0.95)
[1.03 E-0
(2.56 0.61)E-03
(8.88 1.61)B-03
926
(0.9, 0.999)
1 1 L.13)--0
[1.35 1 0.37)
(1.56 1 0.96115-12
1926 1 393
0.005
(0.95, 0.97)
(1.73 + E-05
(3.41 + E-45)E-03
(1.11 0.22)
436
(0.97, 0.95)
(2.31 + E-05
(1.05 + E-25)
(1.21 = 0-25)
551 + 31
[0.9, 0.999)
(4.80 - 3.32) E-06
(1.67 = 0.62)E-03
(1.30 = 0.72)BB00
1163 + 352
0.01
(0.95, 0.97)
(4.08 1 E-05
(5.49 1 2.08)
(1.81 1. 0.44)E-02
396 51
(0.97, 0.95)
(6.38 + E-83)
(6.86 + 3.20)E-03
(1.80 + 0.38)B-02
513 + 58
33

---

Published in Transactions on Machine Learning Research (12/2025)
Table 21: Burgers' Equation with FLM
N
LR
Betas
MSE 1 SD
MAE 1 SD
Max Error 1 SD
Epochs 1 SD
(0,9, 0.999)
(8.62 = 1-45/--33
6.33
(3.54 = 0.13)E-01
10000
0.0005
(0.95, 0.97)
(8.58 1 B-3
(5.33 0.23)
(3.49 : 0.10)E-01
10000
(0.97, 0.95)
0.30
(5.25 0.10)
(3.51 0. 0010)--01
10000 = 0
(0.9, 0.999)
0..80
0.17)-00
(3.59 0.15)E-01
10000
25
0.001
(0.95, 0.97)
1.333
(5.10 0.23)E-02
(3.58 1 0.10)E_001
10000 1 0
(0.97. 0.95)
0.36)
(5.17 0.23
(3.61 + 12)
10000
(0.9, 0.999
0.44)--3
(5.30 E.31)E-02
(3.55 0.15)--01
10000 = 0
0.005
(0,95, 0.97)
(8.39 B-24
(5.11 0.21)E-02
(3.57 = 0.10)E-01
10000
(0.97, 0.95)
(8.52 0-3
(5.19 E.23)E-02
(3.62 1 0.12)E-01
10000
(0.9, 0.999)
0.35)
(4.38 F-22
(3.70 + E-29)
10000 0
0.0005
(0.95, 0.97)
0.35)
(4.38 E-26
(3.80 + 0.37)
10000
(0.97, 0.95)
(148 E-30
(3.70 = 0.23)--01
10000 0
(0,9, 0.999)
0.36
(4.25 0.
(3.68 = 0.21)E-01
10000
64
0.001
(0.95. 0.97)
0.27)
(4.19 0.16)E-02
(3.67 1 0.21)E-01
10000
(0.97, 0.95)
(6.63 6.
(4.23 + 0-26)
(3.70 0.27))-01
10000 0
(0.9, 0.999)
0.33
11:6 0.10)E-02
(3.33 0:23JE-01
10000
0.005
(0.95, 0.97)
(6.59 + F-2
(4.18 0.11)
(3.74 F.27)
10000 + 0
(0.97, 0.95)
(6.52 = 0.24)E-03
(4.10 0.13)E-02
(3.73 0.25)E-01
10000 =
Table 22: Burgers' Equation with SIREN-1
Naidden
LR.
Betas
MSE + SD
MAE
Max Error SD
Epochs + SD
(0.9, 0.999
(3.26 1-116-02
(1.23 = 0.32]E-01
10000 = 0
0.0005
(0.95, 0.97)
0.07)-2
(1.24 0.03)E-01
(3.86 + 0.13)
10000 = 0
(0.97, 0.95)
(2.22:10.077
[1.24 0.04)E.01
(3.87 0.14)E-01
10000
(0.9, 0.908)
[2.0310.100..02
(116100005):000
(3.57 2 0.08):-01
10000 1 0
37
0.001
(0.95, 0.97)
(.002
(3.74 = 0.16)E-01
10000 0
(0.97, 0.95)
(2.11 E-07)
(1.20 0.01)
(3:77 = 0.15)EE01
10000 0
(0.9, 0.999)
(2.07 1: 0.08)E-02
[1.14 0.03)E:01
(3.79 = 0.20JE-01
10000 0
0.005
(0.95, 0.97)
(2.10 E-07
(1.14 0.03)E-01
(3.88 0.19))-01
10000
(0.97, 0.95)
(2.10 1: 0.06
(1.14 + 0.02)E-01
(3.89 + 0.18)
10000 0
(0.9, 0.989)
(2.33 = (2.33 0.631--02
[1.27 0.18)E-01
(3.80 0.34))-01
10000 0
0.0005
(0.95, 0.97)
(2.21 + 0.09) E-02
[1.21 0.03)
(3.80 = 0.11))-01
10000 + U
(0.97, 0.95)
(2.23 1 E.08)
(1.25 1. 0.03)E-01
(3.83 1. 0.10)E-01
10000 0
(09, 0.000)
(2.03 0.08
(1.15 0.04)
(3.67 1 1.14
10000 0
93
0.001
(0.95, 0.97)
(2.10 1 0.08)
(1.19 1 0.04)E.01
(3.60 1 0.12))--01
10000 1 0
(0.97, 0.95)
(2.11 + 0-08)
(1.20 + 0.03)
(3.64 13
10000
(0.9, 0.999)
(2.06 + 0.0916-02
[1.14 + 0.04)E-01
(3.83 = 0.22)E-01
10000 = 0
0.005
(0.95, 0.97)
(2.14 + 0.04)
(1.15 = 0.02)E-01
(3.96 = 0.13)
10000 0
(0.97, 0.95)
(2.13 + 0.02)
(1.14 + 0.02)E-01
(3.95 = 0.12)E-01
10000 0
Table 23: Burgers' Equation with SIREN-2
Nuidden
LR.
Betas
MSE + SD
MAE SD
Mnx Error
SD
Epochs +: SD
(0.9, 0.999)
(2.03 1 0.11)
(1.10 1 0.
(3.71
: U. 0. E-11
10000 1 0
0.0005
(0.95, 0.97)
1.42210.18)..00
(7.99 1 0.86
(3.63
0.14)
10000
(0.97, 0.95)
(1.28 0.09)
(7.27 + E-50
(3.63 - 0-07)
10000 0
(0.9, 0.999)
(1.52
[8.11 1.07)E-02
(3.57 = 0.08)E-01
10000 0
10
0.001
(0.95, 0.97)
1.24
(7.19 F.04)
(3.59 : 0.
10000 0
(0.97, 0.95)
(1.17 0.15)=0
(6.82 + 0.71)E-02
(3.57 - 0.16)E-01
10000 0
(0.9. 0.999)
(1.31 E-19
[T.67 E-88
(3.57 E-09)
10000 0
0.005
(0.95, 0.97)
(1.12 1 0.17)
(6.60 1 0.79)E.00
(3.56 1 0.09)E-01
10000 1 0
(0.97, 0.95)
(9.57 + 0.08)
(5.77 + F.32
(3.56 + 13)
10000 0
(0.9, 0.999)
(1.84 0.34)
(9.76 1: E.29)
(3.75 1 0.24)E-01
10000 0
0.0005
(0.95, 0.97)
(1.30 -: E-02
(7.29 + E-30
(3.68 + 0.24)E-01
10000 + 0
(0.97, 0.95)
(1.25 0.08]E-U2
(6.99 0.31)E--2
(3.66 0-22)
10000
(0.9, 0.999
(1.1 1011102211 0.2716.2
[8.36 1. 1.04)E.02
(3.70 0.23]E-01
10000 1 0
17
0.001
(0.95, 0.97)
(1.21 + 1-2
(6.85 + F-45
(3.67 - 0.21)E-01
10000 0
(0.97, 0.95)
(1.14 E-12
(6.51 - E-0
(3.67 = 0.24)E-01
10000 0
(0.9, 0.999)
(1.23 : E-22
[7.13 - 0.98)E-02
(3.67 = 0.24)E-01
10000 U
0.005
(0.95, 0.97)
(1.01 1 E-02)
(5.95 1 0.48)E-02
(3.69 1 0.24)E-01
10000 0
(0.97, 0.95)
(9.70 + 0.10)
(5.74 + 0.46)
(3.67 - 0.22)
10000 + 0
34

---

Published in Transactions on Machine Learning Research (12/2025)
Table 24: Burgers' Equation with SIREN-3 3
Naiddon
LR
Betas
MSE 1 SD
MAE = SD
Max Error 1 SD
Epochs
SD
[0.9, 0.999)
(1.67 0-39)
(3.65 8.18)--01
10000
0.0005
(0.95, 0.97)
(9.86 +:::1..00 E-03
0.56)-E2
(3.61 4 0.15
10000 +
(0.97, 0.95)
(9.19
0.29
(3.60 0.15)
10000 +
[0.9, 0.999)
1.13)-013
7
0.001
(0.95, 0.97)
(9.51 1.677 E-03
15.3
(3.60 1 1.112)E--
10000 1
(0.97, 0.95)
(8.83
(3.62 4 1.10
10000
[0.9, 0.999)
(1.05 -
0.186-00
(3.60 = 0.16)E--
10000 =0
0.005
(0.95, 0.97)
(8.86 B-0))
[5.35 + E-24)
(3.63 0.18)E-01
10000
(0.97, 0.95)
(7.91 E.92
(4.87 0.49)2-02
(3.67 0.19)E-01
10000
(0.9, 0.999)
0.1
(7.09 1 E-00
(3.69 1 0.16)E-01
10000 +:
0.0005
(0.95, 0.97)
(8.83 + 0.58)
(5.30 E-20)
(3.65 0818)-011
10000
(0.97. 0.95)
(8.26 0.61
(5.01 0.00
(3.66 + 0.16)
10000 +
[0.9, 0.999)
(1.07 E-17)
(3.68 0.
10000
0.001
(0.95, 0.97)
(8.77 1 E-08
(5.22 1 E-50
(3.67 0.16]
10000
(0.97, 0.95)
(8.01 + F-00
(4.83 + 0-40
(3.08 + 19))E-01
10000 +
[0.9, 0.999)
(9.09 E-82]E-03
1 0:00
0.005
(0.95, 0.97)
(7.84 + F-31)
(4.79 + F-01
(3.67 + 0.20)
10000 +
(0.97, 0.95)
(7-40 = E-08)
(4.53 + 0.53)E-02
(3.72 0.31)B-01
10000 =
Table 25: Burgers' Equation with Vanilla Relu
Naideen
LR.
Betas
MSE + SD
MAE
Max Error SD
Epochs + SD
(0.9, 0.999)
(3.22 + 6.06]6002
(1.49 0.28)
(3.72 = 0.24]E-01
10000 = 0
0.0005
(0.95, 0.97)
+ (3.11 E.17)E-02
: (1.51 0-31)
(3.66 = 0.13)E-01
10000 = 0
(0.97, 0.95)
(3.48 1.28]E-02
[1.55 4. 0.33)E-01
(3.76 0.25)E-011
10000
(0.9, 0.909)
(3.69 1.3611222
(3.80 1 0.32)
10000 1 0
0.001
(0.95, 0.97)
(3.30 - 1-18)
(1.51 + 0.31)E-01
(3.64 = 0.14)E-01
10000 0
(0.97, 0.95)
(3.93 E-75)
(1.61 0.39)
(3.93 = 0.67)
10000 0
(0.9, 0.999)
(3:29 1: 1.17]E-02
[1.50 1 0.31)
(3.67 = 0.17)E-01
10000 0
0.005
(0.95, 0.97)
(3.24 1.20) E-02
[1.48 0.32)E-01
(3.70 0.17)
10000
(0.97, 0.95)
(4.17 +: E-06
(1.70 + 0.39)
(3.95 + 0.66)
10000 0
(0.9, 0.989)
(2.82 = 5-80
[1.38 0.20)1-01
(3.03 = 0.28)E-01
10000 0
0.0005
(0.95, 0.97)
(2.59 + 0.09) E-02
[1.32 + 0.01)
(3.58 + 0.10))-01
10000 + U
(0.97. 0.95)
(3.04 1 E.40)
(1.42 1 0.29)E-01
(3.72 1 0.58)E-01
10000 0
(09, 0.000)
(2.83 1 0.73)
(1.38 1 0.000
(3.65 1 0.21)
10000 0
12
0.001
(0.95, 0.97)
(2.60 1 0.13)
(1.32 1 0.05)
(3.59 1 0.07)E-01
10000 1 0
(0.97, 0.95)
(2.90 + F-69
(1.4) + 0.18)
(3.66 0.20)
10000
(0.9, 0.999)
(2.64 + 0.08JE-02
[1.34 + 0.03)E-01
(3.58 = 0.09)E-01
10000 = 0
0.005
(0.95, 0.97)
(2.84 + 0-70)
(1.39 + 0.19)E-01
(3.63 = 0.19)E-01
10000 0
(0.97, 0.95)
(3.16 + 1.01)
(1.47 + 0.25)
(3.74 + 0.35)E-01
10000 0
Table 26: Burgers' Equation with Vanilla Tan
Nuidden
LR.
Betas
MSE + SD
MAE SD
Mnx Error
SD
Epochs +: SD
(0.9, 0.999
(1.16 1 0.161E-02
[6.84 1 0-
(3.63
: 0.18 E-
10000 1 0
0.0005
(0.95, 0.97)
(6.82 1 0-4
[4.38 1 0.26
(3.60
F.1
10000
(0.97, 0.95)
(6.02 0.50)
(3.87 + E-02
(3.88 - 0-60)
10000 0
(0.9, 0.999)
(8.33 1:07)-333
[5.08 0.58)E-02
(3.03 = 0.11)--01
10000 0
7
0.001
(0.95, 0.97)
0-0
(4.12 F.34)
(3.68 : F-29)
10000 0
(0.97, 0.95)
(6.81 0.57)
(3.73 = 0.35)E-02
(3.85 = 0.59)
10000 0
(0.9. 0.999)
0.89
(1.55 0.53)
E-29)
10000 0
0.005
(0.95, 0.97)
(6.16 1 0.53)E.03
(3.94 1 0.30
(3.73
0.36)
10000 1 0
(0.97. 0.95)
0.51)
[3.68 + F.35
(3.86 0-52)
10000 0
(0.9, 0.999)
(8.95 1: 2.13)JE-0
(5.40 1: 1.14)
(3.61 1 0.18)
10000 0
0.0005
(0.95, 0.97)
(5.17 (
(3.26 + E-36
(1.14 + E-80)
10000 + 0
(0.97, 0.95)
(4.82 0.42]E-3
0.17)E-02
(4.15 0.93))-011
10000
(0.9, 0.999
(7.88 0.9316..0
[4.89 1 0.56 E2002
(3.64 1 0919
10000 1 0
12
0.001
(0.95, 0.97)
(5.00 +: E-01
[3.18 F-32
(4. - 0.60)
10000 0
(0.97, 0.95)
- (1.75
E-37)
(2.99 + 0-32)E-02
(1.57 - 0.95)
10000 0
(0.9, 0.999)
- (5.93 B-7
[3.84 - 0.45)
(3.81 = 0.45)E-01
10000 U
0.005
(0.95, 0.97)
(5.00 1 0-44)
(3.22 1 0.28)
(4.23 1 0.88)E-01
10000 0
(0.97, 0.95)
(4.42 1 0.45)
(2.82 + 0.31)E-02
(1.75 - 0.92)E-01
10000 + 0
35


Reviews:
[{"review_id": "4izNRsLgYc", "reviewer": "Reviewer_Kxbw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose the Fourier Learning Machine, a neural network architecture designed to emulate the Fourier series representation of an unknown target function. The network employs shifted cosine functions as activation functions, and its weighting parameters are optimized to eliminate redundancies for improved efficiency. Unlike existing methods, this approach enables accurate construction of both standard Fourier series and their non-harmonic extensions. Simulation results demonstrate that the proposed architecture outperforms comparable Fourier network models and conventional multilayer perceptrons.", "claims_and_evidence_yn": "Yes", "claims_explanation": "All claims are supported with data and or proofs.", "audience_yn": "Yes", "audience_explanation": "The proposed architecture is particularly interesting as it bridges the well-known Fourier series approximation with conventional neural network structures. Preliminary results indicate that there are cases where this architecture is superior to standard networks in terms of accuracy.", "requested_changes": "Major:\nIt seems natural that a Neural Network with this type of smooth basis functions outperforms \"standard\" MLPs on problems with rather smooth solutions (as shown in Fig. 4-6. It would be interesting to swap one of these examples to a PDE with a discontinuous solution (e.g. https://towardsdatascience.com/can-neural-networks-formulate-shock-wave-147bb75d0715/)\n\nMinor:\nPage 7: \"in Equations Equation 5 and Equation 6\"\nPage 13: \"These points are randomly sampled\" Please specify what kind of distribution was used (assume uniform)", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EjWbVpZIIo", "reviewer": "Reviewer_Zt2w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Fourier learning machines (FLMs), a type of neural network motivated by Fourier analysis, and in particular, non-harmonic Fourier series. This essentially entails a Fourier series with a finite set of learnable (non-integer) frequencies in an FLM \"subnetwork,\" which are then summed together. A comprehensive comparison to other neural network architectures using sinusoidal-like activations is undertaken to contextualize the FLM. Then two sets of numerical experiments are presented. The first is with PDE learning, using a PINN-like loss, where FLMs compare favorably to SIREN and MLP networks in several examples. The second set of experiments are in optimal control problems, where FLMs learn well-performing polices.\n\n### Strengths\n- The paper is relatively clear, and the literature review is extensive.\n- There is strong empirical performance among the methods tested in PDE learning, and the optimal control applications are interesting.\n\n### Weaknesses\n\n- There are a few claims I find overly broad or inaccurate, though I think these can mostly be fixed by tempering claims. I detail these below. \n- The SNN is criticized as \"no theoretical background is provided to guarantee the convergence of this NN,\" which is fair. But the FLM also lacks such a formal convergence analysis, which is left implicit. \n- While there is a hyperparameter ablation in Appendix B, further investigation of hyperparameters would strengthen the paper. The reader is left with fairly limited understanding of how many sub-networks are typically beneficial, or how performance behaves with a increasing number of frequencies.\n- I have a lot of trouble following the definition of the $m$-Lexi sign matrix. Revising for clarity would be much appreciated.\n- The examined PDEs are rather simple, including the heat equation, the Poisson equation, and a Black-Scholes-like PDE with a simple analytic solution, $u(x, t) = \\exp(x-t)$. Examining on more difficult PDEs would greatly improve the perceived impact in scientific ML. \n- The basis representation blows up with the input dimension $m$. This is not explicitly acknowledged, to the best of my knowledge, and illustrating scaling would improve the paper.", "claims_and_evidence_yn": "No", "claims_explanation": "There are a few claims I find to be lacking in evidence, though I think they can mostly be ameliorated via tempering claims.\n\n1. In my opinion, \"an investigation on the use of sine functions as activation functions in NNs first appeared in Parascondolo et al. (2017)\" can be misread as implying Parascondolo et al. were the first to attempt/benchmark use of sine activation functions. I think this should be reworded to something like \"Parascondolo et al. (2017) undertook a comprehensive and critical investigation on the use of sine activation functions in deep NNs.\"\n\n2. The novelty claim (\"the FLM is the first architecture able to represent a complete, separable Fourier basis in multiple dimensions using a standard Multilayer Perceptron–like architecture\") is rather strong, and completeness is not formally stated. Plus, the FLM is formally capable of finite-dimensional representations, which cannot be complete.", "audience_yn": "Yes", "audience_explanation": "Yes, improved architectures in scientific ML are clearly of interest to a subset of the TMLR audience.", "requested_changes": "Critical:\n- See evidence section above.\n- Improve upon the definition of the $m$-Lexi matrix.\n\nSuggestions:\n- Include more difficult PDEs for learning and comparison.\n- Comment on the cost with respect to the input dimension $m$, and present experiments accordingly. Compare this to something like SIREN.\n- More directly experiment and comment on the affect of $N$, beyond the hyperparameter ablation. For example, a plot would be very helpful for the reader.", "broader_impact_concerns": "N/A", "additional_comments": "I think theoretical comparison to the sparse spectrum GP [1] would also be interesting, especially the \"optimized\" version where the frequencies are learned.\n\nSome additional minor typos:\n- Above equation (7), \"in Equations Equation 5 and Equation 6\"\n- Below equation (17), \"are the the objective\"\n\n[1] Lázaro-Gredilla, M., Quinonero-Candela, J., Rasmussen, C. E., & Figueiras-Vidal, A. R. (2010). Sparse spectrum Gaussian process regression. The Journal of Machine Learning Research, 11, 1865-1881.", "extra_scores": {}, "extra_text": {}}, {"review_id": "rsd64g3EqI", "reviewer": "Reviewer_6H1i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Fourier learning machine, a feedforward neural network that uses cosine functions as activation layers. \nThe main idea is to decompose the input signal into its spectral basis and learn coefficients to combine these for the end task.\nTo do this, initially the Fourier decomposition of the input signal is written as a combination of cosines and sines. Secondly, the trigonometric decomposition is written in a cosine phase-shift form that can be expressed in the form of a neural network free of complex numbers. In this form, the Fourier coefficients combine the cosine and sine of the input in different linear manners. \nTo generalize this representation to higher dimensions, the authors define the m–Lexi Sign Matrix, a structured matrix that systematically encodes all possible sign combinations of frequency components across the input dimensions.\nThis matrix allows the model to efficiently generate the full set of separable Fourier basis functions (the various sine–cosine products) in a compact cosine-only representation.\nOverall, by allowing these frequencies to be arbitrary and trainable, the model learns its own spectral basis that addresses the end task. This makes the model capable of capturing both periodic and nonperiodic behaviors in data using a simple multilayer perceptron–like architecture.\nThe FLM architecture operationalizes this spectral formulation through a collection of parallel sub-networks, each representing one frequency component of the learned Fourier basis. Each sub-network corresponds to a particular learnable frequency vector and produces a sum of phase-shifted cosine activations whose amplitudes and phases are trained from data. By summing the outputs of all sub-networks, the FLM reconstructs the target function as a weighted superposition of learned spectral components, thereby realizing a neural analogue of the multidimensional nonharmonic Fourier series.\nThe authors apply it to the numerical solution of several benchmark partial differential equations and an optimal control problem. In these experiments, the FLM serves as a mesh-free spectral neural solver, where the network’s learnable frequency components replace fixed basis functions traditionally used in numerical schemes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The methodology is novel, the explanation is clear and the results are promising.\nAcross all PDE benchmarks, the FLM consistently achieved superior accuracy compared to standard neural architectures such as SIREN, ReLU, and Tanh networks. In both the heat and Poisson equations, the FLM attained errors one to two orders of magnitude lower than its competitors, demonstrating its ability to capture smooth and oscillatory solutions with high spectral fidelity.\nIn the nonlinear optimal control problem arising from the evolutionary dynamics of a Rock–Paper–Scissors game, both the state trajectories and the control policy were approximated using coupled FLMs trained via a penalty-based loss formulation. The model successfully learned cyclically symmetric solutions consistent with the problem’s odd–circulant structure and closely matched the reference trajectories\nThese experiments collectively illustrate that FLMs can serve as general-purpose spectral solvers for both forward PDE problems and optimal control formulations.", "audience_yn": "Yes", "audience_explanation": "It is an interesting methodology that relates directly to neural representation learning and time series forecasting.", "requested_changes": "1. To demonstrate the applicability of FLM, the model should be compared on a real-world time-series forecasting problem to demonstrate its ability to learn dominant frequencies and trends from empirical data, and compare with the SOTA.\n\n2. It would also be beneficial to add an experiment on a high-dimensional OCP. To my understanding, all current examples are ≤3-dimensional. Since the FLM architecture scales naturally with the number of inputs via the m–Lexi Sign Matrix, testing higher-dimensional dynamics (e.g., a 5-strategy evolutionary game or a 3D diffusion equation) would demonstrate scalability and generalization.\n\n3. The paper would benefit from an experiment with time-dependent PDEs that have non-periodic boundary conditions. Current PDE examples (heat, Poisson, Black–Scholes) hase smooth periodic structure. Since FLMs are inherently spectral, testing them on non-periodic or discontinuous boundary conditions would show robustness.\n\n\nMinor:\n\n'To begin with, we set the frequencies to nonnegative vectors...' I assume the authors mean they set them to zero.", "broader_impact_concerns": "There is no evident ethical concern from this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)