FAN: Fourier Analysis Networks
Authors: Yihong Dong
Ge Li
Yongding Tao
Xue Jiang
Kechi Zhang
Jia Li
Jinliang Deng
Jing Su
Jun Zhang
Jingjing Xu
Venue: NeurIPS 2025 poster ()
Keywords: Foundation model architecture
Primary area: deep_learning

Abstract: Despite the remarkable successes of general-purpose neural networks, such as MLPs and Transformers, we find that they exhibit notable shortcomings in modeling and reasoning about periodic phenomena, achieving only marginal performance within the training domain and failing to generalize effectively to out-of-domain (OOD) scenarios. Periodicity is ubiquitous throughout nature and science. Therefore, neural networks should be equipped with the essential ability to model and handle periodicity. In this work, we propose FAN, a novel neural network that effectively addresses periodicity modeling challenges while offering broad applicability similar to MLP with fewer parameters and FLOPs. Periodicity is naturally integrated into FAN's structure and computational processes by introducing the Fourier Principle. Unlike existing Fourier-based networks, which possess particular periodicity modeling abilities but face challenges in scaling to deeper networks and are typically designed for specific tasks, our approach overcomes this challenge to enable scaling to large-scale models and maintains the capability to be applied to more types of tasks. Through extensive experiments, we demonstrate the superiority of FAN in periodicity modeling tasks and the effectiveness and generalizability of FAN across a range of real-world tasks. Moreover, we reveal that compared to existing Fourier-based networks, FAN accommodates both periodicity modeling and general-purpose modeling well.

FAN: Fourier Analysis Networks
Yihong Dong'', Ge Li Yongding Tao', Xue Jiang', Kechi Zhang', Jia Li ,
Jinliang D Jing Jun Zhang'. Jingjing Xu² School of Computer Science, Peking University
The Hong Kong University of Science and Technology *ByteDance dongyh @ stu. stu.pku.edu.cn lige @ pku.edu.cn
Abstract
Despite the remarkable successes of general-puroos neural networks, such as MLPs and Transformers, we find that they exhibit notable shortcomings in model- ing and reasoning about periodic phenomena, achieving only marginal performance within the training domain and failing generalize effectively to out-of-domain (OOD) scenarios. Periodicity is ubiquitous throughout nature and science. There- fore, neural networks should be equipped with the essential ability to model and handle periodicity. In this work, we propose FAN, novel neural network that effectively addresses periodicity modeling challenges while offering broad applica- bility similar to MLP with fewer parameters and FL.OPs. Periodicity is naturally integrated into FAN's structure and computational processes by introducing the Fourier Principle. Unlike existing Fourier-basee networks, which possess particular periodicity modeling abilities but face challenges in scaling to deeper networks and typically designed for specific tasks, approach overcomes this challenge enable scaling to large-scale models and maintains the capability to be applied to more types of tasks. Through extensive experiments, ity of FAN in periodicity modeling tasks and the effectiveness and generalizability of FAN across range of real-world tasks. Moreover, we reveal that compared existing Fourier-based networks, FAN accommodates both periodicity modeling and general-purposs, modeling well.
demonstrate the superior-
Introduction
The flourishing of modern machine learning and artificial intelligence is inextricably linked to the revolutionary advancements in the foundational architecture general- purpose neural networks. For instance, multi-layer perceptron (MLP) [Rosenblati 1958 Haykin 1998) plays a pivotal role in laying the groundwork for current deep learning models, with its expressive power guaranteed by the universal approximation theorem [Hornik et al. 1989]. Recent claims about the impressive perfor- mance of large models on various tasks are typically supported by Transformer architecture [Vaswani] et al. 2017 Touvron al. 2023 OpenAl 2023) In this context, the community' enthusiasm for research on neural networks has never diminished. Some emerged neural networks demonstrate
notable capabilities in specific fields [Gu and Dao 2023
Liu al. 2024 sparking widespread
discussion within the community.
Beneath the surface of apparent prosperity, uncover a critical issue that remains in existing general-purpose neural networks: they struggle to model the periodicity from data, especially OOD
"Equal Contribution
"This work was supported by cooperation project between Peking University and ByteDance Company. During this time, Yihong was also an intern at ByteDance. "The code is available at atthttps://githuu wtt.aluu.ooommm//////e/eeee wtt.lccco///o///e wwwm/Y/hoogDogg//A...
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

sin(x)
Pooe Fresktero
MLP
KAN
Transformer
FAN (Ours)
The performance of different neural networks within and outside the domain of their training data for the sine function, where a scalar variable.
Figure I:
scenarios. We showcase this issue through an empirical study as illustrated in Figure T The results indicate that existing neural networks, including MLP [Rosenblati] 1958), KAN [Lu et al. [024]] and Transformer (Vaswani et al.) 2017], face difficulties fitting periodic functions, even on simple sine function. Although they demonstrate some proficiency in interpolation within the domain of training data, they tend to falter when faced with extrapolation challenges of test data. This signifies that their generalization capacity is primarily dictated by the scale and diversity of the training data, rather than by the learned principles of periodicity to perform reasoning.
Periodicity is an essential characteristic in various forms of reasoning and generalization, as it provides basis for predictability in many natural and engineered systems by leveraging recurring patterns in observations. Besides periodic phenomena, non non- con-periodic phenomena can also be contextualized or explained within some larger or more macro-ppriodle framework. Although some Fourier-based networks exhibit particular periodic modeling abilities, they are primarily tailored for specific tasks
[Silvescu 1999 Liu 2013] and do not work well as the networks deepen [Liu et al. [020]: which limits their applicability to the general task such as language modeling [Uteuliyeva et al. 2020).
However, our goal is to exploit periodicity to benefit broader range tasks including language modeling. To achieve this, we aim to develop neural network that accommodates modeling and reasoning capabilities for periodicity while maintaining the capability to applied to more types of tasks.
In this paper, we propose Fourier Analysis Network (FAN), a novel neural network built upon the principle of Fourier Analysis. By leveraging the power of Fourier Series, we enable the neural network model periodic patterns and extrapolate beyond them, offering the network way to model the general principles from the data. FAN follows two core principles, the first ensures that its periodic modeling capacity scales with network depth, while the second guarantees periodic modeling is available throughout the network. These principles allow it to scale to deeper networks, capability where existing Fourier neural networks fall short. As a result, FAN exhibits exceptional capabilitics in periodicity modeling, while maintaining broad applicability to the general task, which holds great potential as a substitute for MLP, with fewer parameters and FLOPs.
To verify the effectiveness of FAN, we conduct extensive experiments from three main aspects: 1) For periodicity modeling, FAN achieves significant improvements fitting both basic and complex periodic functions, compared to existing neural networks (including MLP, KAN, and Transformer), particularly in OOD scenarios. 2) FAN shows superior performance in various real-world tasks, such as symbolic formula representation, time series forecasting, and language modeling- Using FAN outperforms the representative models various tasks, including MI., KAN, LSTM, Mamba, and Transformer. 3) Compared to existing Fourier-based networks, FAN accommodates both period- icity modeling and general-purpos modeling well. The advantageous characteristics and promising results indicate that FAN has the potential to become basic component for building fundamental large models.
2 Preliminary Knowledge
Fourier Analysis (Seein and Weiss 1971 Duoandikoetxea 2024 a mathematical framework that decomposes functions into their constituent frequencies, revealing the underlying periodic structures within complex functions. At the heart of this analysis lies Fourier Series [Tossoov] [012], which expresses periodic function infinite sum of sine and cosine terms. Mathematically, for

---

function f(x), its Fourier Series expansion can be represented as:
2TnT
f(x)
+
Un COS
+ bn sin
(I)
T
T
where T is the period of the function, and the coefficients On and bn, are determined by integrating the function over one period:
1 an - - [(x) cos
2TTN T
bn 7 /'- f(x) sin
22Tnr T
dx,
dx.
The power of Fourier Series lies in ability to represent a wide variety of functions, including non-periodic functions through periodic extensions, enabling the extraction of frequency components. Building on this math foundation, FAN aims to embed the periodic characteristics into network architecture, enhancing generalization capabilities and performance various tasks, particularly in scenarios requiring the identification of patterns and regularities.
( dBBm W
(x)
cos( 11 sin(W,))
o(B, W3x]]
Activation Function
Cosine
Sine
Activation Function
7
Leamable Weights
(a) MLP Layer
(b) FAN Layer
Figure 2: Illustrations of FAN layer ((x) vs. MLP layer ((x).
3 Fourier Analysis Network (FAN)
In this section, we first construct naive neural network modeled by the formula of Fourier Series. Then, by modifying and improving it, we design FAN adhering to two core principles. Finally, we discuss the difference between the FAN layer and MLP layer.
Consider a task involving input-output pairs {x;, J: with the objective of identifying a function f(x) : Rd. + Rd that approximates the relationship such that yı x f(x;) ) for all 2, where d, and dy denote the dimensions of x and y, respectively. We first construct shallow neural network fx) that represents Fourier Series expansion of the function, specifically F()()) }, as described Eq.
express fa(r) follows:
N
12TTEe T
22 T
fs(x) Og
an cos
+ bn sin
i
C cos (w)) + w sin (w:))r))
(3)
B [wf wh w cos( (w=te] |لسيم ... [/
+ [wf wh, w sin Sin([urj ([w)') | الار ... [|
B + W. cos(WW,.......
W. sin (hna.)
(II)
B + Wone [cos W
(n xn) ssnssnnnnnni (innr)],
3

---

Table I: Comparison of FAN layer and MLP layer, where dp is hyperparameter of FAN layer and defaults to douuppu in this paper, disutt and doupput denote the input and output dimensions of the neural network layer, respectively. In our evaluation, the floating point of operations (FLOPs) for any arithmetic operations considered 1, and for Boolean operations as O.
MLP Layer
FAN layer
Formula
(x)) BB W Wm))
x(I)
|cos( W p (l + War)
Num of Params
(duput x doutput) + douput
(1
ty x ((dmput x doutpus) + doupuu)
2 X (dinput x doutpor)
(1 durma
do
x 2 x (dinpux x doutpor)
FLOPs
+FLOP WLLOPSpon-linnee Spon-lineer X doutput
+ Ppooliaael doutput
where B Rdx, Wi E R-xxd, and out E Rd,x2N are learnable parameters, (I) follows that the computation of an and b, computed via Eq. (2) is definite integral, (II) and (III) follows the equivalence the matrix operations, [-Il-] and [: denotes the concatenation along the first and second dimension, respectively.
To fully leverage the advantages of deep learning, we can stack the aforementioned network fs(x:) to form a deep network fn fI)),, where the i-th layer, denoted as (r)), retains the same structural design as fa(r). Therefore, (D( can be formulated as:
fD(x) 11 o 1
o
where I x denotes the application of the left function I to the right input æ; that is 1 (z). However, discover that the direct stacking of fs(x)) results in the primary parameters of the network fD (a) focusing on learning the angular frequency (wi, = 2mm) thereby neglecting the learning of the Fourier coefficients (a,, and b,)), as follows:
fD(x) = 11111-1 o 1T-2 ..... .0.....4000 ox)
(5)
B² + Wil (cossW (1.. x)) sin (l1. o x))]
where h:2 o x: is defined as L o 2/2-2 00 ................................ ol. th x. Whh (11. ox) is used to approximate the angular frequencies, and W is used approximate the Fourier coefficients. We can find that the capacity of fo( to fit the Fourier coefficients is independent of the depth of fu(x), which is an undesirable outcome. It will limit the network's 's representation ability, hindering to address the complex tasks.
To this end, we design FAN based on the following principles: 1) the capacity of FAN to represent the Fourier coefficients should be positively correlated to its depth; 2) the output of any hidden layer can be employed to model periodicity using Fourier Series through the subsequent layers. The first one enhances the expressive power of FAN for periodicity modeling by leveraging its depth, while the second one ensures that the features of FAN's intermediate layers are available to perform periodicity modeling.
Suppose we decouple fs(x) follows:
fs(x) = Sout o sin x.
(6)
where
J ( = [cos( ( [n))l (W,.)))
(7)
foua(1) = W T.
(8)
To satisfy both principles, the inputs of the intermediate layers in FAN necessitate to employ fia and fout simultaneously, rather than applying them sequentially.
Finally, FAN is designed on this basis, with the FAN layer ((x) defined as below:
((x) A [cos(W
w,r) sin (inWW>))la + Wor)),
where W, E Rd,x W E Rd. xdp, and B, E &d are learnable parameters (with the hyperpa- rameters dp and dp indicating the first dimension of W, and W,, respectively), the layer output Φ(x) pdd,+dp and denotes the activation function. Under this definition, the MLP layer can be

---

2 mod
traiting Plct trrdttora
Iranstormer
MAN
40.00
Daman of Tra
oos (2x)
Froditiory
The performance of FAN in periodicity modeling compared MLP, KAN, and Transformer (Part D) where the green line represents the test data within the domain of training data, while the blue line represents the test data outside the domain of training data.
Figure 3:
regarded as special form of Eq. ( when W, are learned to be zero metrics, which provides way for FAN maintain general-pee purpose modeling abilities as MLP.
The entire FAN defined as the stacking of the FAN layer o(x) as follows:
FAN(2) = dr 00.. oz,
(10)
where
<ι(T) = [cos( (cos(WWa))) ( + Wsa)). ifl L, if L,
= B + W W'
(11)
The difference between FAN and MLP. The illustrations of FAN layer ((x) vs. MLP layer + (a) are shown in Figure
Note that the FAN layer ((z) computed via Eq. (9) can seamlessly replace the MLP layer & computed via Eq. (12) in various models with fewer parameters and FLOPs. achieved by sharing the parameters and computation of Sin and Cos parts. The number of parameters [ The reduction which set to 1 by default in this paper.
2
and FLOPs of the FAN layer compared the MLP layer are presented in Table
ratio parameters and FLOPs is about dp
desppe
Experiments
In this section, we first verify the superiority of FAN in periodicity modeling tasks (Section 4. Second. we demonstrate the effectiveness and generalizability of FAN across range of real- real-world tasks (Section 4.2). Finally, we conduct further analysis of FAN (Section 4.3). including comparisons with Frourierbasee networks, rumning time, hyperparameter impact, and more. See Appendix for more experiments and the experimental details can be found in Appendix
4.1 Periodicity Modeling
Setup. In periodic modeling tasks, we select periodic functions with practical significance and compare the models' performance in learning the underlying principles of periodicity. Specifically, we generate data from periodic functions over large domain, using a portion of this domain as training data and the entire domain as test data, i.c., part of test data would be out of the domain of

---

7 cotta? cca20 antes
13-
0:311
CCN
615
0003
1690
6,9
2000
1230
19000
2002
Racch
Reach
Spack
Figure 4: Comparison of training and test losses for different models on the tasks of learning complex periodic functions.
training data. We compare FAN and its variant FAN (Gated $ with MLP, KAN, and Transformer. The input of this task scalar.
well as Figure Appendix, show the performance of FAN and other baselines in periodicity modeling. The results indicate that existing neural networks, including MLP, KAN, and Transformers, exhibit notable deficiencies in their ability to model periodicity. Although they attempt to these periodic functions, their ability limits their performance in modeling a large domain of periodicity, including the test data within and outside the domain of the training data. In contrast, FAN significantly outperforms baselines all these tasks of periodicity modeling. Moreover, FAN performs exceptionally well on the test data both within and outside the domain, indicating that our specialized design of FAN can effectively model and understand periodicity rather than merely memorize the training data.
Results. Figure
We also compare the training process of different models on the tasks of learning complex periodic functions, as shown in Figure 4 which leads to the following findings. 1) FAN far exceeds the other basclines in both convergence speed and final effects. 2) FAN (Gated) often achieves faster convergence than FAN, but the final performance remains comparable. 3) Although the baselines show stabilization or gradual reductions in training loss as the number of epochs increases, their modeling may have diverged considerably from the distribution of the test data, resulting in sharp increase in test loss. This phenomenon further demonstrates the shortcomings of these models in capturing periodicity.
4.2 Application of Real-worid Task
1) Symbolic Formula Representation a common task in both mathematics and physics. We follow the experiments conducted in KAN's paper [Liu et al. 2024]. adhering to the same tasks, data, hyperparameters, and baselines. In addition to the original baselines, we also include Transformer for comparison in this task.
Results. Figure 7 in Appendix shows the performance of different models applied to common functions in mathematics and physics. We can observe that while KAN remains competitive with FAN when the number of parameters small, its performance declines clearly as the number of parameters increases, which exhibits U-shaped trend [Liu et al., 2222)). In contrast, as the number of parameters becomes large, FAN consistently outperforms the other basclines, including MLP, KAN, and Transformer, in fitting these functions, despite many of these functions being only partially periodic or even implicitly periodic. This may be attributed to FAN's ability to capture and model both
FAN (Gated) is variant of FAN that adds gates to control the tendency of the layer, with the formula detined as (x) = [g cos(W,=)]lg sin(W, g) oBB + Wyr)), where g is learnable parameter.

---

periodic and non non-periodic features and the advantages of fewer parameters. These results indicate that although FAN enhances its ability model periodicity, it does not compromise its capacity to fit non-periodic functions.
2) Time Series Forecasting plays a critical role in various real-world applications. We employ four public datasets of this task to assess the model performance on time series forecasting, including Weather [Wu al. 2021]. Exchange [Lai et al., 2018), Traffic (Wu et al. 2021], and ETTh [Zhou et al. 2021] datasets. For each dataset, we input 96 previous time steps and forecast the subsequent time steps of (96. 192, 336, 7203. In this task, we choose the sequence models as baselines, including LSTM, Mamba, and Transformer.
Results. As shown in Table 2 (See GJ Appendix for com- plete results), we compare the performance of Transformer with FAN and other baselines for time series forecasting tasks. The re- sults indicate that Transformer with FAN outperforms other representa- tive sequence models these tasks, The improvements of Transformer with FAN and FAN (Gated) over the standard Transformer are notable, with the average relative improve- ments ranging from 15.0% to 15.6% for MSE and from 7.6% to 8.4% for MAE It suggests that incorporating explicit periodic pattern encoding within neural networks improves time series fore- casting performance in real-world applications.
Table 2: Average performance on different public datasets and output lengths in time series forecasting tasks, where Input Length 96 and the bold value indicates the best performance.
Average
Model
Num of Params
MSE 1
MAE 1
LSTM
12.51M
1.083
0.726
Mamba
12.69M
1.002
0.668
Transformer
12.12M
0.994
0.689
w/ FAN (Gated)
11.07M
0.845
0.637
FAN
11.06M
0.839
0.631
Improvements
1.06M
15.6%
8.4%
3) Language Modeling is a fundamental task in natural language processing. We conduct language modeling using the SST-2 [Socher et al., (2033) dataset and evaluate the model's performance on its test set, as well on the related datasets such as IMDB [Maas al. 2011], J Sentiment (Sahni it al., (017], and Amazon Reviews [Linden et al. 2003). These four classic datasets all belong to the field of sentiment analysis. The comparisons are between Transformer with FAN and FAN (Gated), along with the classic sequence models, including LSTM, Mamba, and Transformer.
Table 3: Performance of different sequence models on language modeling tasks, where the models are trained on the training set of SST-2 and evaluated on the other datasets, the bold value indicates the best performance on each column, the bold italic indicates the second-best performance, and the improvements represent relative improvements of using FAN based on standard Transformer.
Model
Num of Params
SST-2 (test)
IMDB
centiment140
Amazon Reviews
Loss
Acc 1
Loss 4
Ace f
Loss
Acc f
Loss
Ace f
LSTM
120. 14M
0.4760
80.60
0.6449
64.38
0,8026
59.79
0.5791
71.52
Mamba
129.73M
0.4335
79.59
0.6863
62.03
0.7871
58.74
0.6163
67.19
Transformer
109.48M
0.4297
81.19
0,5649
69.94
0.8891
57.79
0.5563
71.55
w7 FAN (Gatedj
95,33M
0.4250
"80.39
0.58I7
70.12
0.7944
61.94
0.4835
76,89
w/ FAN
95.32M
0.4094
81.54
0.5225
73.98
0.8257
60.93
0.4748
77.63
Improvements
4. 14.166
4.72%
0.43%
4 7.51%
1 5.78%
7.13%
5.43%
4 14.65%
f 8.50%
Results. We report the performance comparison between different sequence models across four senti- ment analysis datasets, as shown in Table 3 The results indicate that Transformer with FAN achieves clear improvements compared to the standard Transformer and other baselines, such as LSTM and Mamba, especially for zero-shot OOD performance on IMDB, Sentiment 140, and Amazon Reviewers datasets. Using FAN achieves the relative improvements up to 14.65% and 8.50% terms of Loss and Accuracy respectively, while reducing parameter numbers by about 14. 16M It indicates the potential of periodicity modeling to enhance both effectiveness and generalization on cross-domain language modeling and sentiment analysis tasks.
7

---

4.3 Further Analysis of FAN
Comparison with Fourier-based Networks. We compare FAN with Fourier-base networks in terms of their periodicity modeling abilities and general-purpon purpose capabilities for language modeling. Some previous works have explored the application of fourier-based Networks specific tasks Oreshkin et al., 2020, Tancik et al. 2020 Sitzmann et al. 2020 Han et al., 2022], but these studies primarily involved shallow/small scale models (i.e., fewer than 1M parameters). Assessing their general modeling capabilities requires evaluating their effectiveness in deeper/larger architectures, we categorize these Fourier-based networks into three main types and systematically evaluate them within the 12-layer Transformer. Specifically, we compare with: 1) Fourier Neural Network (FNN) [Silvescu 1999] using the cosine or sine function or their linear combinations as the activation function, such SIREN (Sitzmann et al. 2020). 2) Fourier Series Neural Network (FSNN) is defined as Eq. which shares the parameters and computation of sine and cosine part. 3) Fourier Transform Neural Network (FTNN) is type of neural network that employs Fourier Transform to process the intermediate output in the neural network, such as FNO [Li et al. 2021]
Figure 5: Comparison FAN with Fourier- based Table 4: Comparison FAN with Fourier-based Networks on complex periodicity modeling (y- = Networks on language modeling tasks, where sin(sae) (ooo- (x mod 8) 1) and language modeling. each of them replaces the MIP layer in the stan- dard transformer and ID means in-domain.
Petoeiiiyy Mode by
Langauga Hedaling
Loss +
Model Num of Params
Train
ID Test
OOD Test
MLP
109.48M
0.2574
0.4297
0.5649
FNN 109.48M
0.6933
0.7103
0.7135
02-
FSNN 95.32M
0.6931
0.7210
0.7249
FINN
300.56M
0.2449
0.4547
0.8128
FAN
95.32M
0.2434
0,4094
0,5225
Eooch
As shown in Figure 5 only FAN achieves excellent performance both tasks, indicating the
superiority of our specially designed architecture of FAN. In contrast, FNN and FSNN cannot fit language modeling tasks, which aligns with previous work [Uteuliyeva al. 2020 Liu al. 2020 and our findings derived from Eq. -3 Moreover, FTNN performs poorly on complex periodic modeling tasks, akin to MLP. This may be attributed to the fact that FTNN does not incorporate the Fourier principle into the network but applies Fourier Transform as an intermediate processing step. which disadvantages FTNN in capturing periodicity. From Table 4 FAN also achieves fewer parameters and better performance than FTNN in language modeling tasks.
Table 5: Comparison of actual runtime between FAN and MLP under different input/output dimen- sions (e.g., 8192x8192 indicates both input and output dimensions are 8192).
1024x 1024
2048 2048
4096x4096
8192x8192
MLP
0.064 ms
0.114 ms
0.212 ms
0.938 ms
FAN
0.128 ms
0.133 ms
0.211 ms
0.704 ms
Runtime FAN. We analyze the actual running time of FAN layer compared to MLP Layer with different input and output dimensions, shown in Table 5 The experimental results show that MLPs exhibit smaller runtimes when the input and output sizes are small, due to Py PyTorhh's 's optimization of MLP. However, the input and output sizes continue to increase, matrix computations become the main contributor to runtime. At this point, FAN's fewer parameters and reduced FLOPSs begin to show significant advantages. Note that FAN can be further optimized from the underlying implementation.
The impact of hyperparameter dp. In our experiments, we fix dp 1 dd intuitively for FAN, where d, denotes the dimension of hidden layers. As shown in Figure of Appendix, we investigate the impact of varying dp empirically on task performance by changing itself. The results indicate that performance initially improves as dg increases, but then decreases beyond a certain point. This trend may be attributed to the number of potential periodic features specific to each task. Furthermore, there remains room for further improvements with the better setup of dp.
8

---

5 Related Work
In this section, we outline the two most relevant directions and associated papers of this work.
Learning Periodicity with Neural Networks. Periodic functions are one of the most basic func- tions of importance human society and natural science [Newton] 1687 Oshorn and Sensier [002] Kwasnicki 2008 De Groot and Franses 2012 Zhang et 2017]. However, commonly used neural networks, such MLPs and transformers, struggle with modeling periodicity. This limitation is attributed to the lack of inherent "periodicity" in their inductive biases, Some previous works (Sil
veseu, 1999 Liu 2013 Parascandolo al.. 2016 [teuliyeva et al., 2020] proposed merely using standard periodic functions themselves ves or their linear combinations as activation functions, which only work well on some shallow and simple models. On this basis. work (Liu et al. 2020] introduced the Snake function, i.e., x + sin"(ar, the activation function. However, it can fit periodic functions to a certain extent, but effect limited especially for OOD scenarios, as demonstrated in Appendix D Therefore, although some previous studies have attempted to integrate periodic information into neural networks, their actual performance and range of applications remain heavily constrained.
Fourier-based Neural Network. Previous studies have explored Fourier-based networks, but these networks generally perform well on specific tasks, while their performance on more general tasks
tends to be poorer (Zuo and Cail 2005 Tan, 2006 [teuliyeva et al.] 2020 Diang al.) [002]] Chen
al. 2022]. Fourier Neural Networks employ the cosine (Silvescu] 1999 Ngom and Marin 202 or
sine function Paaraccanoloo et al. 2016 Sitzmann et al.. 2020 or their combination Liu 2013 as
the activation function. Some work employs Fourier Transform to process the intermediate output of network et. 2021 Lee-Thorp et al., [2022], but they did not address the challenges of periodicity modeling. Some researches focus leveraging the network to simulate the formula of Fourier Series [Rafajlowicz and Pawlak| 1997, Halawa 2008 Lee et al.] 2021]. which generally possesses a similar principle Eq. (3). However, this leads the same problem in Eq. (5) i.e., they are hard to serve as building blocks for deep neural networks, which limits these approaches' capabilities. More detailed discussion can be found in Appendix G
In this paper, we design FAN to address these challenges, which performs exceptionally well on periodicity modeling and maintains broad applicability on real-world tasks.
Discussion
In this section, we have broad discussion on expressive power, extrapolation capability, and applica- tion scope of FAN as follows: 1 FAN theoretically possesses the equal expressive power MLP since it also adheres to Universal Approximation Theorem, which guarantees its capacity for func- tional approximation (refer to Appendix E for the detailed explanation). Moreover, FAN introduces an important enhancement by incorporating periodicity, a feature absent in MLPs. By leveraging this special design, FAN not only retains the capabilities of MLP but also enhances its ability to capture periodic characteristics in data. @ We observe that existing networks often exhibit divergent predictions in OOD scenarios, as shown in Figures 3 4 and 6 for periodicity modeling tasks. In contrast, FAN demonstrates strong OOD extrapolation ability in both periodicity modeling and some real-world tasks. This extrapolation ability indicates that the network is longer restricted the paradigms present in training dataset, but instead exhibits a kind of "transboundary thinking". This could be an important avenue for improving generalization and learning efficiency. @ Beyond tasks that explicitly require periodicity modeling, FAN also has utility in broader range of applications, which has been evidenced by our extensive experiments on real-world tasks, such as symbolic formula representation, time series forecasting, language modeling, and image recognition, where FAN achieve competitive or superior performance than Transformers and other baselines. In fact, many machine learning tasks may harbor hidden forms of periodicity, even without explicitly includ- ing periodicity, such as mathematical operations and logic reasoning. If the neural network lacks the ability to model periodicity, it could impair the learning efficiency (Dong et al.] 2025b). From deeper perspective, periodicity is not just a data feature but reflects form of structural knowledge one that allows for the transfer and reuse of abstract rules and principles across different contexts.

---

7 Conclusion and Future Work
In this paper, we have proposed Fourier Analysis Network (FAN), novel network that addresses periodicity modeling in existing networks while maintaining the general-purpost modeling capability, Experimental results demonstrate that FAN successfully fit both basic and complex periodic functions, whereas other general-purpos purpose networks failed. Moreover, using FAN exhibit clear improvements real-world tasks, such as symbolic formula representation, time series forecasting, and language modeling, outperforming neural networks such as MLP, KAN, LSTM, Mamba, and Transformer. These promising results, especially the stronger performance and the fewer parameters and FLOOPs compared to MI.P, suggest its potential to become key component of foundational models. Some works have demonstrated the superiority of using FAN in diverse tasks, including gravitational wave analysis ZZhoo et al., 22024, EEG-based emotion recognition (Wang et al. 22225, and large language modeling [oog et al. 2025b), etc. In future work, we aim to further broaden the applicability of FAN.
8 Acknowledgement
This research supported by the National Key R&D Program under Grant No. 2023YFB4503801, the National Natural Science Foundation of China under Grant No. 62192733, 62192730, 62192731, the Major Program (JD) of Hubei Province (No. (N..2023BAA024) Moreover, we would like to thank Lecheng Wang and Xuanming Zhang for their participation in discussions related this work.
References
Frank Rosenblatt. The perceptron: probabilistic model for information storage and organization in the brain. Psychological review, 65(6))386, 1958.
Simon Haykin. Neural networks: a comprehensive foundation. Prentice Hall PTR, 1998.
Kurt Hornik, Maxwell B. Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359-366, 1989.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998- 6008,0088800 2017.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog. Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, Abs/2307.09288 2023.
OpenAI. GPT-4 technical report. CORR, abs/2303.087744 20233
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2312.00752, 2023.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. KAN: wwwooooooo-aaaananooooooooo arnold networks. CoRR, abs/2404.197566 2024.
Adrian Silvescu. Fourier neural networks. In IJCNN, pages 488-491 IEEE, 1999.
Shuang Liu. Fourier neural network for machine learning. In ICMLC, pages 285-290. IEEE, 2013.
10

---

Ziyin Liu, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how to fix In NewrIPS, 2020.
Malika Uteuliyeva, Abylay Zhumekenov, Rustem Takhanov, Zhenisbek Assylbekov, Alejandro J. Castro, and Olzhas Kabdolov. Fourier neural networks: A comparative study. Intell. Data Anal., 24(5):117--1220 2020.
Elias M Stein and Guido Weiss. Introduction Fourier analysis Euclidean spaces, volume 1.
Princeton university press, 1971.
Javier Duoandikoetxea. Fourier analysis, volume 29. American Mathematical Society, 2024.
Georgi P Tolstov. Fourier series. Courier Corporation, 2012.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with cuto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419-22430,, 2021.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In SIGIR, pages 95-104. ACM, 2018.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time-series forecasting. AAAI, pages 11106-11115.. AAAI Press, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631-1642. ACL, 2013.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, pages 142-150. The Association for Computer Linguistics, 2011.
Tapan Sahni, Chinmay Chandak, Naveen Reddy Chedeti, and Manish Singh. Efficient twitter sentiment classification using subjective distant supervision. In COMSNETS, pages 548-553,
IEEE, 2017.
Greg Linden, Brent Smith, and Jeremy York. Amazon.com com recommendations: Item-to-item collabo- rative filtering, IEEE Internet Comput., 7(1):76-80, 2003.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS; neural basis expansion analysis for interpretable time series forecasting. In ICLR. OpenReview.net, 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurlPS, 2020.
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NeurlPS, 2020.
Bing Han, Cheng Wang. and Kaushik Roy. Oscillatory fourier neural network: A compact and efficient architecture for sequential processing. In AAAI, pages 6838-68888 6846. AAAI Press, 2022.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadeneshelli Burigede Liu, Kaushik Bhat- tacharya, Andrew M. Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In ICLR. OpenReview.net, 2021.
Isaac Newton. Philosophiae naturalis principia mathematica. William Dawson & Sons Ltd., London, 1687.
Denise R. Osborn and Marianne Sensier. The prediction of business cycle phases: Financial variables and international linkages. National Institute Economic Review, 182(1))96-105, 2002. doi: 11100/79999900022001120000000 URL
htthttps: hattps://dol org/10..1/.. /1000/11777000220002222220011110
11

---

Witold Kwasnicki. Kitchin, juglar and kuznetz business cycles revisited. Economic Sciences, 2008.
Wroclaw: Institute of
Bert De Groot and Philip Hans Franses. Common socio-economic cycle periods. Technological Forecasting and Social Change, 79(1):59-68, 2012.
Liheng Zhang, Charu Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi- frequency trading patterns. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, page 2141-2149, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 978145034887 doi: 10.114573097983330988111
URL wtttt::p///oo// org/10./. 1145/3099966..... 3098117
Giambattista Parascandolo, Heikki Huttunen, and Tuomas Virtanen. Taming the waves: sine as activation function in deep neural networks. 2016.
Wei Zuo and Lilong Cai. Tracking control of nonlinear systems using fourier neural network. In Proceedings, 2005 IEEE/ASME International Conference on Advanced Intelligent Mechatronics., pages 670-675. IEEE, 2005.
HS Tan. Fourier neural networks and generalized single hidden layer networks in aircraft engine fault diagnostics. 2006.
Song Jiang, Tahin Syed, Xuan Zhu, Joshua Levy, Boris Aronchik, and Yizhou Sun. Bridging self- attention and time series decomposition for periodic forecasting. CIKM, pages 3202-3211. ACM, 2022.
Hanlong Chen, Luzhe Huang, Tairan Liu, and Aydogan Ozcan. Fourier imager network (FIN); A deep neural network for hologram reconstruction with superior external generalization. Light: Science & Applications, 2022.
Marieme Ngom and Oana Marin. Fourier neural networks as function approximators and differential equation solvers. Stat. Anal. Data Min., 14(6))647-66) 2021.
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontañón. Fnet: Mixing tokens with fourier transforms. In NAACL-HLT, pages 4296-4313, Association for Computational Linguistics, 2022.
Rafajlowicz and M Pawlak. On function recovery by neural networks based on orthogonal expansions. Nonlinear Analysis: Theory, Methods & Applications, 30(3):1343-1354 1997.
Krzysztof Halawa. Fast and robust way learning the fourier series neural networks on the basis of multidimensional discrete fourier transform. In ICAISC, volume 5097 of Lecture Notes in Computer Science, pages 62-70. Springer, 2008.
Jiyoung Lee, Wonjae Kim, Daehoon Gwak, and Edward Choi. Conditional generation of periodic signals with fourier-based decoder. CORR, Abs/2110.12223 2021.
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In ACI. (Findings), pages 1209-1 Association for Computational Linguistics, 2024.
Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei Ma, Jue Chen, Binhua Li. Zhi Jin, Fei Huang. Yongbin Li, and Ge Li. RL-PLUS: countering capability boundary collapse of Iims in reinforcement learning with hybrid-polige optimization. CoRR, abs/2508 000002200..0222.. 2025a.
Yihong Dong. Ge Li, Xue Jiang, Yongding Tao, Kechi Zhang, Hao Zhu, Huanyu Liu, Jiazheng Ding, Jia Li, Jinliang Deng, and Hong Mei. Fanformer: Improving large language models through effective periodicity modeling. CoRR, abs/2502.21309. 2025b.
Tianyu Zhao, Yue Zhou, Ruijun Shi, Peng Xu, Zhoujian Cao, and Zhixiang Ren. Compact binary coalescence gravitational wave signals counting and separation using unmixformer, 2024. URL.
https://arxiv.. iereiiii.vrggg/aee org/abs//2ee abt///2...222....... 18259
12

---

Jinfeng Wang, Yanhao Huang, Sifan Song, Boqian Wang, Jionglong Su, and Jiaman Ding. A novel fourier adjacency transformer for advanced eeg emotion recognition, 2025. URL https
7/arxiv. httrrv.orr//bbs/5000 13465
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yamn.ecuu httpy/awweeccn...o////dli/ee wwwp//xo/b//iiistt 2, 2010.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarial training of neural networks. J. Mach. Learn. Res., 17:59:1-59:35, 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017. URL http::/aaxiv.orr//aaaat
1708. 07744
Michael Weiss and Paolo Tonella. Simple techniques work surprisingly well for neural network test prioritization and active learning. In Proceedings of the 31th ACM SIGSOFT International Symposium Software Testing and Analysis, 2022.
Tian Zhou. Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 27268 27286.. PMLR, 2022.
Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient for fast stylization. CoRR, abs/ het///6770.022.. 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 973: 1780, 1997.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278-222 1998.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
166.045666 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster). OpenReview.net, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep
bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
/ttt:://axxi... http://axxivvoraaa////// wrwgg///g/////ooooeoeer/o/o 04805
Li Wan, Matthew D. Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus, Regularization of neural networks using dropconnect. In ICML (3), volume 28 of JMLR Workshop and Conference Proceedings, pages 1058-1066. JMLR.org, 2013.
Peter Belcak and Roger Wattenhofer. Periodic extrapolative generalisation neural networks. In SSCI, pages 1066-1073. IEEE, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately present the paper's contributions and scope, with claims that are fully supported by the results and discussion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We demonstrate the superiority of using FAN in some mainstream tasks, and we aim to further broaden the applicability of FAN our future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: he theoretical analysis of this paper is as follows: 1) In Section 3, we theo- retically analyze that existing FSNNS and FNNs have problems in deep expansion while FAN addresses it, and conduct experiments to prove it in Section 4.3. 2) In Table 1, the theoretical analysis shows that FAN has fewer parameters and FLOPs than MLP, and we conduct experiments to prove it in Tables 2, 3, and 5. 3) In Appendix E, we theoretically analyze that FAN complies with Universal Approximation Theorem.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the experimental details in Appendix C, and make reproducible source code available at attttps: htttpp:://anonymouss .oopee scwence/rr/AN--A.....
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
15

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide source code with comprehensive instructions and data acquisition methods at (https: hatp:////nnnmmmuuuuum 4open science//r/ANN--DBB/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
cciips.ccpubliic/uuideee gwwpe//ddee/eee CCodeSubeeiieeeee ssi onyolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide the experimental details in Appendix and open source code wwwp://oouuuooooo..oooo 4open. wtw.eeeccccinccee//A-----ee
(https://anonymouus
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We follow previous work to report the average performance of 5 runs.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides the information computer resources in Appendix C.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp///berrrpppcc//abab/////ee
Answer: [Yes]
Justification: The research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics as outlined at http:////errippccccpppbbbe
Ethicstuidellieel
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
17

---

Justification: Our work is foundational research without direct societal impacts, as it focuses on fundamental algorithmic improvements rather than specific applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeent of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper focuses only model design and algorithms without releasing any high-risk models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit all existing assets used in our research. We cite the original papers for all datasets and code packages utilized in our experiments, including specific versions and URLs where applicable. All datasets are used in accordance with their respective licenses.
Guidelines:
18

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, petiasetssppperssse. paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide comprehensive documentation for our new code assets, including detailed implementation instructions, usage guides, and experimental configurations in anonymized repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Foundation Model Architecture
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy htttps///eaennpseec/////oeereees for what should or should not be described.
20

---

A MLP
The MLP layer (ax is defined as:
((x) = o(Bm + Wm.),
(12)
where Bm E Rdd and W E Rd xdm are learnable parameters with the hyperparameter dm indicating the first dimension W,, denotes the activation function, and MLP be defined as the stacking of the MLP layer (xr):
MLP(x) 1r 11 ..... 4
(13)
where
( m Wh=,, Bh. + WT
ifl L, =
21(x)
(14)
Daraa of
MLP
KAN
Transformer
FAN
sinCe Si((2)
SEAEDOCCCOOOC
MLP
KAN
sin(t) cus (20) cost (3n
Cemain Thintro Cata
MLP
KAN
Transformer
Figure The performance FAN in periodicity modeling compared MLP, KAN, and Transformer (Part II), where the green line represents the test data within the domain of training data, while the blue line represents the test data outside the domain of training data.
B Additional Experiments
B.1 Additional Experiments on Periodicity Modeling Tasks.
More experimental results periodicity modeling tasks are shown in Figure 6
21

---

Comparisons of FAN with the baselines, including MLP, KAN, and Transformer, acTOSS varying numbers of parameters on symbolic formula representation tasks.
Figure 7:
Table 6: Performance different sequence models on time series forecasting tasks, where Input Length 96, the bold values indicate the lowest value each row, and Improve means the relative improvements of using FAN and FAN (Gated) based on standard Transformer.
LSTM
Mamba
Transformee (12.12 M)
Transformer with FAN (11.06 M)
Dataset
Output Length
(12.51 M)
(12.69 M)
Gated
Default
MSE 1
MAE
MSE 4
MAE
MSE
MAE
MSE 1
MAE
MSB
MAE
1.069
0.742
0.552
0.519
0.413
0.438
0.292
0.380
0.313
0.431
Weather
0.700
0.540
0.535
0,550
336
0,992
0.727
0.841
0.667
0.751
0.626
0.637
0.602
0.719
0,581
720
1.391
0.892
1.171
0.803
0.967
0.715
0.845
0.706
0.732
0.670
96
0.938
0.794
0.908
0.748
0.777
0.681
0.685
0.644
0.657
0.623
192
1.241
0.899
1.328
0.925
1.099
0,800
0.998
0.757
0.968
0.741
Exchange
336
1 1.645
1.048
1.512
0,992
1.614
1 1,029
1,511
0,961
1.266
0,905
720
1, 1.949
1.170
2,350
1,271
2.163
1,204
1.658
1.104
1.145
0.659
0.359
0,666
0.377
0.656
0.357
0.647
0,355
0.643
0.347
Traffic
192
0.668
0.360
0.671
0,381
0.672
0.363
0.649
0,353
0.657
0.354
336
0.644
0.342
0.665
0.374
0.673
0.360
0.666
0.358
0.656
0.353
720
0.654
0.351
0.662
0.364
0,701
0,380
0.682
0.369
0.673
0.363
96
0.999
0.738
0.860
0.697
1.139
0.853
0.842
0.736
0.873
0.707
192
1.059
0.759
0.849
0.700
1.373
0.932
0.885
0.748
0.914
0.741
ETTh
1.147
1.005
0.745
1.261
0,924
0,980
0.999
0.793
720
1.206
0.847
0,994
0.758
1.056
0.819
1.002
0.798
1.031
0.818
Average (Imprive)
0.845
0.637
0.839
0.631
1.083
0.726
1,002
0.668
0.994
0.689
15.0%
7.6%
15.6%
8.44%
B.2 Additional Experiments on Image Recognition Tasks.
Image Recognition is key computer vision task where image content is identified and categorized. Our evaluation contains four public benchmarks of image recognition: MNIST [LeCun et 2010 MNIST-M Ganin et al., 2016]. Fashion-MNIST [Xiao et al.) 2017), and
Weiss)
and Tonella 2022]. where MNIST-M and Fashion-MNIST-C are the variants for robustness.
Table 7: Results on image recognition tasks, where OOD Accuracy means the performance on other paired datasets and the Bold values indicate the highest values under the same metric.
Accuracy 1
OOD Aecuracy t
Datasct
CNN
w/ FAN
CNN
w/ FAN
MNIST
99.63
99.67
28.85
30.3
MNIST-M
94.52
994.23
82.85
"83.55
Fashion-Mie -MAIiT
94.15
94.47
49.82
51.88
Pasfion-MMiST 88.67
"88.82"
91.45
"9..5
Results. We apply FAN to image recognition tasks on four classic benchmarks, as shown in Table 7 The results show that using FAN outperforms the standard CNN in most cases. We believe that there are also some latent periodic features in image recognition tasks, and FAN's ability to model these periodic features can help CNN achieve competitive or superior performance, especially in OOD scenarios.
22

---

B.3 Evaluation on LLMs with FAN
Table reports the zero-shot results on the LM Eval Harness benchmark. The results show that using FAN outperforms standard Transformer architecture across various tasks with the same training tokens of 200B.
Table 8: Comparison of our approach with well-trained Transformer language models on LM Eval Harness benchmark. Both of them are trained on 200B tokens and using FAN achieves better accuracy.
Models
challenge
easy
boolg
bella-swag
open bookqa
piqa
sclq
wino-grande
avg.
Transformer-IB
29.7
63.3
59.6
52.5
34,6
71.4
85.8
55.9
56.6
Ours
32.1
63.5
60.1
53.8
34.7
72.5
89.9
56.1
57.9
B.4 FAN for Solving SciML Problems
We conduct experiments on the SciML problem that includes the Fourier function class following the work [Li et al. 2021]. The Burgers' equation, a non-linear partial differential equation, is frequently used scientific computing model shock waves and traffic flow, among other phenomena. The detailed error rate on Burgers' equation is listed in the Table 9 We can find that replacing the can achieve clear
MLP Layer with FAN Layer in Fourier Neural Operator (FNO) Li 2021 improvements on each setting of resolution of this task.
Table 9: The error rate on Burgers' equation. The values in the table represent the Average Relative Error for Burgers' equation with lower values indicating better performance.
Model FNO
256
512
1024
2048
4096
8192
5.93%
6.14%
6.03%
6.75%
7.36%
9.93%
FNO with FAN
5.26 %
5.17%
5.18 %
6.73 %
6.35 %
7.06 %
B.5
Comparison with Frequency-basee Models in Time Series Forecasting Tasks
To compare with frequency-based models in Time Series Forecasting tasks such as FEDformer /Zhou ctt al. 2022], we replace MLP with FAN in frequency-barr models. We present the experimental results in Table 10, where the results of FEDformer are cited from its paper directly. From the results, we can find that FEDformer with FAN can outperform FEDformer in almost all cases.
Table 10: Results of comparison with frequency-basee models in time series forecasting tasks.
Dataset
FEDformer
with FAN
Len
MSE
MAE
MSE
MAE
96
0.587
0.366
0.577
0.357
192
0.604
0.373
0.601
0.366
Traffic
336
0.621
0.383
0.620
0.378
720
0.626
0.382
0.619
0.370
96
0.148
0.278
0.138
0.267
192
0.271
0.380
0.261
0.371
Exchange
336
0.460
0.500
0.461
0.503
720
1.195
0.841
1.159
0.827
96
0.193
0.308
0.184
0.298
192
0.201
0.315
0.199
0.313
Electricity
336
0.214
0.329
0.212
0.325
720
0.246
0.355
0.239
0.347
23

---

B.6 Comparison with Directly Learning the Coefficients
We compare FAN with a baseline of directly learning the coefficients, which inputs sin(x) and cas(x) and then uses the MLP Layer instead of the FAN Layer to model the Fourier coefficients. In this setting, frequencies are fixed and only the coefficients are learned, which may limit the model's ability to capture patterns not aligned with these frequencies. Taking simple f(x) = x mod 5 as an example, this setting may not even converge at all, because the frequency of x mod 5 is inconsistent with sin (x) and cr) The experimental results their loss are shown in Table 11
Table 11: Comparison of FAN and directly learning the coefficients on fitting f(a)) mod 5.
Epoch
50
100
150
200
Directly learning the coefficients
2.10
2.09
2.09
2.08
FAN
0.28
0.23
0.18
0.17
B.7
The influence of hyperparameters dp
We evaluate the influence of hyperparameters dp as shown in Figure
LI72 Jea0
8001
634
25
1.44
1
0.00
0170
Figure 8: The influence of hyper-parametest dp on language modeling tasks. We use the red dashed line to represent the performance of the standard Transformer.
B.8 The effectiveness of the FAN Layer for deep neural networks
We evaluate the effect of varying the number of FAN layers from 3 to 20 periodicity modeling tasks, employing residual connections to mitigate overfitting. The experimental results show that both the best training loss and test loss still decrease slowly as the number of layers increases.
Furthermore, Language Modeling tasks, replaced 24 MLP Layers of Transformer with 24 FAN Layers, i.c. Transformer with FAN, and it also achieved clear improvements on each task, especially for OOD zero-shot evaluation scenarios. These findings indicate that FAN Layer effective for deep neural networks.
B.9 Experiments on Time Series Forecasting with Instance Normalization
We conduct experiments on time series forecasting tasks with instance normalization Ulyanov et al. 2016), and the results are shown in Table 12 We find that applying instance normalization before the architecture can effectively improve the performance.

---

Trininn Loss
Test Loss
144
1.4
1.2
1000 (10g T040
(Log S.a0
1.2
0.6
0.8
12
Layer Num
Lager Roams
Figure 9: Performance of Deeper FAN on fitting = (1 mod 3) 1.
Table 12: Results time series forecasting tasks with instance normalization, where Input Length 96, the bold values indicate the lowest value on each row, and the improve means the relative improvements of using FAN and FAN (Gated) based Transformer.
Transformer
Transformer with FAN (11.06 M)
Output Length
(12.12 12 M)
Dataset
Gated
Default
MSE 4
MAE
MSE
MAE
MSE
MAE $
96
0.1772
0.2301
0.1864
0.2352
0.1756
0.2247
192
0.2438
0.2844
0.2445
0.2834
0.2327
0.2760
Weather
336
0.3077
0.3267
0.3156
0.3320
0.3118
0.3291
720
0.4253
0.3982
0.3909
0.3782
0,4113
0.3906
96
0.1433
0.2653
0.1157
0.2452
0.1436
0.2666
192
0,2563
0.3552
0.2539
0.3611
0.2651
0.3757
Exchange
336
0.5273
0.5218
0.4329
0.4891
0.5092
0.5326
720
1.7401
0.9273
1,5783
0.9303
1.0599
0.7657
0.6160
0.3449
0.6030
0.3334
0.6109
0.3319
192
0.6329
0.3479
0.6239
0.3404
0.6258
0.3370
Traffic
336
0.6369
0.3485
0.6416
0.3487
0.6200
0.3380
720
0.6555
0.3577
0.6645
0.3574
0.6412
0.3525
96
0.3881
0.4097
0.4082
0.4292
0.3833
0.4149
192
0,5766
0.4999
0.4695
0,4514
0.5039
0,4640
ETTh
336
0.5782
0.5100
0.5556
0.5012
0.5417
0.4940
720
0.5841
0.5230
0.5070
0.4943
0.5272
0,4951
Average (Improve)
0.499
0.406
0.472
0.399
0.531
0.416
$ 6.1%%
4 2.2%
11.0%
44.1%
B.10 Layer-wise Spectral Analysis
We conduct experiments on layer-wise spectral analysis below. We perform Fast Fourier Transform (FFT) on each layer's outputs and calculate four key metries to quantify the spectral characteristies:
Spectral Centroid: Measures the "center of mass" of the spectrum, indicating whether the layer's 's features are concentrated in low or high-frequency regions.
2. Spectral Sparsity (L.1/12 Norm): Quantifies how concentrated the spectral energy within few frequency bins. A higher value implies more structured and less noisy signal.
25

---

3. Spectral Entropy: Measures the uniformity and predictability of the spectrum. A lower entropy indicates more ordered and well-defined spectral structure.
4. Dominant Energy Ratio (Top-5): The proportion of total spectral energy contained within the top most dominant frequency components, indicating how focused the representation is key periodic features.
The results reveal a highly effective multi-stage learning process, which more sophisticated than simple monotonic evolution of frequencies. We observe clear three-stage "DDconstruction- Auploration-Reeoonttuciiiooo mechanism:
1. Initial Approximation (Layer 1): The first layer rapidly forms initial, highly-focused approximation of the signal, as shown by its very high Dominant Energy Ratio (96.1 %)..
2. Feature Deconstruction and Exploration (Layers 2-8): To model the function's 's complex, non-sinusoidal components (especially the (mod 3) term, which requires wide range Fourier series terms), the intermediate layers must first "deconstruct" the signal. This is evidenced by sharp increase in Spectral Entropy and decrease in the Dominant Energy Ratio. The network actively disperses energy across broader spectrum to explore and capture these challenging features, showcasing the flexibility afforded by its depth.
3. Integration and Reconstruction (Layers 9-11): In the final layers, the model's task shifts from exploration to integration. "reconsttructs" final, efficient representation from the features learned in the middle layers. This is marked by dramatic decrease in both Spectral Entropy and Spectral Centroid, alongside a sharp increase in the Dominant Energy Ratio final value of 93.8%. The network converges "clean", wwwfrreuuncy, and highly structured representation that is optimal for the final linear layer to map to the target output.
Table 13: Layer-wise spectral analysis of FAN layer outputs.
Layer
Spectral Centroid
Spectral Sparsity
Spectral Entropy
Dominant Energy Ratio (Top-5)
FAN Layer
4.1213
3.4767
1.2264
0.9612
FAN Layer
2,8760
5.0003
3,2549
0.7602
FAN Layer
2,8804
5.0626
3.1556
0.7807
FAN
Layer
2,8810
4.7149
2.6616
0.8426
FAN Layer
3.0820
4,5832
2.2248
0.8753
FAN Layer
3.0815
5.2388
2.5560
0.8378
FAN Layer? 7
2,6955
5.8367
3.0115
0.7806
FAN Layer
2,9132
5,5387
2.7301
0.8086
FAN Layer
2,7376
4.1371
1.6760
0.8986
FAN Layer
2.1266
3.1509
1.0673
0.9356
FAN Layer 11
1.7721
2.9775
0.9270
0.9375
B.11
Ablation Study
We conduct ablation studies on just cosine function, having FAN layers only in part of the network. and freezing W. The results show that FAN demonstrates a clear ad advantage over the variants in Periodicity Modeling and Language Modeling tasks.
Table 14: Results for ablation studies on the Periodicity Modeling task.
Periodicity Modeling
Epoch=0
Epoch=100
Epoch=1000
training loss
test loss
training loss
test loss
training loss
test loss
FAN cos
39.85
63.42
2.67
10.18
1.80
5.26
FAN MONN_repace_fii
39,54
46.15
2.95
6.81
1.37
45.44
FAN replace Jast pprrt
42.82
55.46
21.96
27.86
22.79
30.51
freezing W,, for FAN
40.52
60.20
15.57
89.09
1.13
156.25
FAN
39.62
61.02
2.75
7.43
1.05
4.15
26

---

Table 15: Results for ablation studies on the Language Modeling task.
Language Modeling
Train Loss
In-domain Test Loss OOD Test Loss
FAN cos
0.2419
0.4802
0.7727
FAN replace bataaceffirs_lll first_1/3_par
0.2693
0,4313
0.6700
FAN lastaccellst_//3_____ -pprtt
0.2417
0.4660
0.8052
freezing Wp for FAN
0.2376
0,4736
0.6324
FAN
0.2434
0.4094
0.6077
C Experimental Details
Baselines. In our experiments, we mainly compare FAN with the following baselines: 1) MLP
[osenblatt 1958]. 2) Transformer [Vaswani al.) 2017] 3) KAN (Liu et al. [024], 4) LSTM
Hochreiter and Schmidhuber 1997]. 5) Mamba [Gu and Dao 2023), 6) CNN [LeCun et al. 1998).
Details of the baselines are given Appendix Moreover, we also include the following variants of
FAN into our comparisons: I) FAN (Gated); a variant of FAN that adds gates to control the tendency of the layer, with the formula defined as 6g (z) [g-cos ( pa)) llg-lliii (iggsinn(WW sin(W,=))I(L1 (1)() -
WBr),
where is a learnable parameter. II) Transformer with FAN and Transformer with FAN (Gated): we replace each MLP layer in Transformer with the FAN layer computed via Eq. (9) and the layer of
FAN (Gated), respectively. III) CNN with FAN: similarly, we replace each MLP layer in CNN with the FAN layer.
C.1 Implementation Details.
We conduct our experiments single GPU of Tesla A100-PCIe-00G. Unless otherwise specified, we use the following hyperparameters in the experiments. The model architecture consists of 3 to 24 layers, the activation function set GELU [Hendrycks and Gimpel| 2016), and the dimension of the projection matrix W, is set to d, = dd,, where dz. denotes the dimension of the hidden layers. We employ the AdamW optimizer Loshchilov and Hutter. 2019 for the model's training process.
C.2 Setup Periodicity Modeling
In periodicity modeling tasks, FAN, MLP, and KAN each consist of three layers with comparable FLOPs, while the Transformer model comprises twelve layers. For consistency, we set the hidden layer dimension (d,)) to 2048 for FAN, MLP, and Transformer. In the case of KAN, original paper [Liu et al. 2024]. where the spline order (K) and the number of spline intervals (G) are set to and 50, respectively. We apply learning rate of x 10-5 for training all models. We ensured that the data density of each period in tasks was consistent, meaning that each cycle contained a fixed quantity of 10,000 training data points.
follow its
C.3 Setup Symbolic Formula Representation
In symbolic formula representation tasks, we used the create create_dataser function from the official KAN repository to generate the datasets. Each dataset contains 3000 training samples and 1000 test samples, with all input variables randomly sampled from the range [-1, 1J. We followed the training settings from the original KAN paper, training all methods using LBFGS and Adam for 1800 steps, and select- ing the best persforrromnng result from the two optimization approaches. For KAN, increased the num- ber of grid points to scale up the parameter size, covering G {3, 5, 10, 20, 50, 100, 200, 500, 1000]- For other methods, scaled up the parameter size by increasing the number of layers and the dimensions of hidden layers.
C.4 Setup of Time Series Forecasting
In time series forecasting task, we implement our model based on the codebase by (Wu et al., 2021] Fach model comprises 2 encoder layers and decoder layer. We fix the hidden size for both the Transformer and our model at 512, with the feedforward dimension set 2048 (four times the hidden size). The parameter sizes detailed in the main text correspond to the Exchange dataset; variations in
27

---

the number of variables across different datasets influence the linear layers in the model. We adjust the hidden sizes of the other models to align with the Transformer parameters for fairness.
C.5 Setup Language Modeling
In language modeling task, we employ the BERT tokenizer [evlinn al. 2018 and an embedding layer with a dimensionality of 768, except for Mamba, which adheres to its default settings as specified the original paper [Gu and Dao 2023]. The architecture features 24, and 12 layers with hidden sizes of 1800, 768, and 768 for LSTM, Mamba, and Transformers, respectively. To mitigate training stagnation in deeper LSTM models, we reduce the number of layers while increasing the hidden size to balance the parameters. Importantly, Mamba's layer count is twice that of similarly sized Transformer, as each layer consists of two Mamba blocks (Multihead attention block + ML.P block).
C.6 Setup of Image Recognition
In image recognition tasks, we employ the CNN the baseline model, which consists of four Convolutional Layers and two MLP Layers (It achieves 0.37% error rate on MNIST without augmentation, outperforming the SOTA CNN's 0.63% [Wan et al. 2013). We replace MIP with FAN in CNN, i... CNN with FAN, as the counterpart, ensuring that they have similar parameters. For each task, use stochastic gradient descent with momentum (SGDM) as the optimizer, the learning rate set to 0.01, and the training process runs for 100 epochs.
Domain of Test Dala
sin(x)
Training Data Model Predictions
210
Snake
FAN (Ours)
2 1 0.5))
n-3
0.30
0.5
MLP
MLP
KAN
0.25
Transfomer
0.4
Transformer
Snake
Shake
0.20
FAN (Gaced)
FAN (Gaced)
Locs
0.3
FAN
Test
0.15
0.10
0.00
0.0
2000
4000
5000
8000
30000
2000
4000
6000
8000
30000
Epoch
Epach
Figure 10: Comparisons of FAN with MLP (Snake)
Liu et al.
2020 fitting periodic functions.
D Comparison of FAN and Snake Activation Function
We compare FAN with Snake, previous approach used for improving the fitting of periodic functions with neural networks. The results are shown in Figure I0
28

---

E
Compliance with the Universal Approximation Theorem
The Universal Approximation Theorem asserts that feed-forward network with single hidden layer, containing a sufficiently large and finite number of neurons, can approximate any continuous function defined compact subsets of R", provided that the activation function is non-constant, continuous, and nonlinear. In the case of the Fourier Analysis Network (FAN) layer, we define the mapping as:
<(x)= = (os(W
sin(W
[B, + W22)
where || denotes concatenation, and (-)) represents a standard nonlinear activation function, such as ReLU or GELU. The components cos (W -pz) and sin sin) are non-constant, continuous, and nonlinear functions, satisfying the requisite conditions for an activation function in the Universal Approximation Theorem. Therefore, the FAN layer conforms to the Universal Approximation Theorem, enabling it to approximate arbitrary continuous functions compact subsets of R".
This proof demonstrates that the FAN layer, through its periodic components (sine and cosine func- tions) and the nonlinear activation a(), satisfies the key conditions of the Universal Approximation Theorem, ensuring its capability to approximate complex functional mappings.
F More Details of Baselines
In our experiments, we mainly compare FAN with the following baselines. 1) ML.P Rosenblatt 1958]: the most classic model, which is widely used in the backbone of various models. 2) Trans- former [Vaswani et al. 2017]: a prevalent model known for its self-attention mechanism, which achieves outstanding performance specialized for symbolic formula representation, which uses the b-spline functions instead of fixed activation functions. 4) LSTM [Hochreiter and Schmidhuber 1997]: well-known recurrent neural network (RNN) that can capture long-term dependencies on sequential data. 5) Mamba [Gu and Dao 20233: emerged selective state space model (SSM) that achieves competitive performance on some tasks with sequential inputs. 6) CNN [LeCun et al. 1998): convolutional neural network contains the convolutional layers, which are effective in processing image data.
various tasks. 3) KAN [Liu et al. 2024]: emerged model
For Fourier-basee based Networks, we mainly compare FAN with 1) Fourier Neural Network (FNN) [Silvescu] 1999] using the cosine sine function their linear combinations as the activation function, such as SIREN [Sitzmann et al. [2020]. 2) Fourier Series Neural Network (FSNN) is defined as Eq. (3. which shares the parameters and computation of Sin and Cos part. 3) Fourier Transform Neural Network (FTNN) a type of neural network that employs Fourier Transform to process the intermediate output in the neural network, such FNO Li et al. 2021
G More Detailed Discussion with Fourier-based Neural Network
Parascandolo al. 2016 Uteuliyeva et al., 2020 they face challenges in scaling to deeper networks, I.e., the capacity of their deep networks to fit the Fourier coefficients independent of the network depth, as analyzed in Section 3. The depth scalability limits their applicability to more complex, general-purpose tasks such as language modeling- Our core differences are, "we design FAN based on the following principles: 1) the capacity of FAN to represent the Fourier coefficients should be positively correlated to its depth; 2) the output of any hidden layer can be employed to model periodicity using Fourier Series through the subsequent layers." In Section 4.3, conduct experiments to compare approach with FNNs, and FNNs cannot fit language modeling tasks, but our approach works well, We provide the analysis FNNs compared to FAN below. We mainly discuss the work (Silvescu 1999 Liu 2013 Parascandolo et al. 2016), due to the work (Uteuliyeva et al. 2020 is a comparative study without proposing a new method.
For FNNs [Silveseu] 1999 Liu 2013
Belcak and Wattenhofer 2022]. they focus on different purposes from our work. And work [Lee et al. 2021 assumes all input signals have the period of (as stated in page of paper), which we conducted experiments on the same setting in Appendix B.6] and it cannot fit periodicity modeling tasks.
For work [Lee al. 2021
29

---

Table 16: Comparison of parameters and FLOPs for different layers, where disput (Input dimension hyperparameter), d, duuput (Output dimension hyperparameter), dp = FAN layer hyper- parameter (default [do)), d Hidden dimension hyperparameter, de, d, = Layer hyperparametens of cosine/sine branch dimensions, nonlinear activation (a, cos, or sin).
Layer hyperparameter of projections number, 7 FLOPss per
FAN
[Saverr 1999
Parawardiio 2016
Formataa
Wo I stal Myya e1 (l
Maall IL port 1a-1+ My
Ao, My, R di) abile, Au, Mi.
Wy My
hum Parmna
224444 4. (5) 264 s4
suduts dal +-4,
aid, 43+ 4. dia
ddd 6 -LAL +44
FLOPs
Indal 400 D)- 26,6 mod.
26,8, 4 4, 2423, 44 41
24/6, 4,2 +-
G.1 Limitation
First, we only demonstrate the effectiveness of FAN on some mainstream real-world tasks (including symbolic formula representation, time series forecasting, language modeling, image recognition, etc.), and we aim to further broaden the applicability of FAN in our future work. Second, although we have explored the generalizability of FAN and confirmed that FAN outperforms the baseline method in some real-world tasks, the boundaries of this model's s generalizability remain unknown. However, we have not yet identified specific scenarios where it performs poorly. We leave this for our future work.
30


Reviews:
[{"review_id": "JPwkhENdR4", "reviewer": "Reviewer_kgu3", "summary": "This paper introduces the Fourier Analysis Networks (FAN), which extends the usual MLP networks to model periodicity in inputs by adding sine and cosine as activation functions to each layer. For motivation, this paper notes that sine and cosine together generate the Fourier Basis in Fourier Series expansion (Section 3).\n\nWith the added sine and cosine activation functions in each layer, this paper argues that periodicity modeling is available throughout the network, and together with MLP activations in each layer this periodicity modeling capacity scales with network depth (Sections 1 and 3).\n\nFor validation, this paper conducts experiments with synthetic data (Section 4.1, Periodicity Modeling) and real-world data (Section 4.2, Symbolic Formula Representation, Time Series Forecasting, Language Modeling), showing that FAN (or FAN Gated) gets lower loss (\\~7%, MSE or MAE) and higher accuracy (\\~5%), using fewer params (\\~15%) and less FLOPs, than competing (non-periodicity-aware) models (such as MLP, KAN, Transformers without FAN, LSTM, Mamba).\n\nEven when comparing with other periodicity-aware, Fourier-based models (FNN, FSNN, FTNN), the newly introduced Fourier Analysis Networks (FAN) still performs better in synthetic data (in Periodicity Modeling, Section 4.3) with lower loss and fewer params.", "questions": "Currently in a FAN Layer $\\phi(x) = [\\cos(W_px) || \\sin(W_px) || \\sigma(B_{\\bar p}+W_{\\bar px})]$ (Figure 2), so cosine and sine do not have the bias components ($B_p$), unlike the MLP part. Would adding the bias component help, for example, in modeling “phase shift” in periodicity? Note that after adding bias components, the network can use sine or cosine alone, without using both. In particular, how would such networks flair in experiments?", "limitations": "### Note on periodic generalization\n\nLines 78-79, Page 3: “The power of Fourier Series lies in its ability to represent a wide variety of functions, including non-periodic functions through periodic extensions, enabling the extraction of frequency components.”\n\nSuch periodic extensions assume that the generalization is periodic (outside of the primary period), which may or may not hold.\n\nThe examples in Figure 1, Figure 3 and Figure 6 (See Section 4.1) are the best possible periodic generalization for Fourier Analytic Networks: outside the training set, the test set is perfect periodic extension. So by “over-fitting” to the training set, we get perfect generalization to the test set, which is what FAN does best, compared with other networks (MLP, KAN, Transformers).\n\nBut to be fair, the experimental results in Section 4.2 show that periodicity are indeed present in real-world data, e.g., in Time Series, or in Language Modeling.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "For strengths, the experiments shows that FAN gets lower loss with fewer params in real-world data such as time series forecasting and language modeling, and the added sine and cosine activation functions to model periodicity are well motivated, as elaborated in Section 3.\n\nFor weaknesses, the periodicity ratio ($d_p$) is yet another hyper-parameter to tune.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your in-depth review and valuable feedback. We will respond to your comment point-by-point below.\n \n> Currently in a FAN Layer (Figure 2), so cosine and sine do not have the bias components (B_p), unlike the MLP part. Would adding the bias component help, for example, in modeling “phase shift” in periodicity? Note that after adding bias components, the network can use sine or cosine alone, without using both. In particular, how would such networks flair in experiments?\n\nThanks for your suggestions. We conduct experiments on the setting of adding bias components and use cosine alone, namely FAN_cos, and have added the experimental results to our revised manuscript.  Overall, FAN outperforms FAN_cos on periodicity modeling and language modeling tasks. Although FAN_cos occasionally achieves lower training loss, its significantly higher test loss indicates poor generalization.\n\n| Periodicity Modeling       | training loss (epoch=0) | test loss (epoch=0) | training loss (epoch=100) | test loss (epoch=100) | training loss (epoch=1000) | test loss (epoch=1000) |\n|-|-|-|-|-|-|-|\n| FAN_cos              | 39.85                    | 63.42               | **2.67**                      | 10.18                | 1.80                       | 5.26                   |\n| FAN                  | **39.62**                    | **61.02**               | 2.75                      | **7.43**                 | **1.05**                       | **4.15**                   |\n||\n\n| Language Modeling | Train Loss | In-domain Test Loss | OOD Test Loss |\n|-|-|-|-|\n| FAN_cos           | **0.2419**     | 0.4802       | 0.7727        |\n| FAN               | 0.2434     | **0.4094**       | **0.6077**        |\n||\n\n> periodicity ratio (d_p) is yet another hyper-parameter to tune.\n   \nWe fix the periodicity ratio to 1/4 and achieve stable and clear improvements across all experiments with this same setting. Although we found in the appendix that further adjusting the periodicity ratio can lead to even greater improvements for specific task, we believe that the default parameters used in this paper can already cover common scenarios.\n\n> Note on periodic generalization ...\n\nWe do believe that some generalization properties can be expressed as periodicity, so we aim to extract the information with periodic generalization using the Fourier Principle, which leads to FAN naturally performing better on tasks with periodicity. Of course, for real-world data, periodic extension is a valid and parsimonious assumption that complies with Occam's razor. However, the extent of its generalization capability is likely highly dependent on the specific real-world task. We have added this discussion to our revised manuscript."}, {"review_id": "06ZXOCudR8", "reviewer": "Reviewer_nmeh", "summary": "The paper proposes a contribution to the well-studied fields of periodicity in neural networks and deep learning with Fourier series.\n\nOn the outset, the authors claim to introduce Fourier Analysis Networks, a purportedly novel neural network designed to effectively address periodicity modeling challenges. They claim that FANs overcomes the limitations of existing Fourier-based networks by enabling scaling to deeper and large-scale models, while simultaneously maintaining general-purpose modeling capabilities. The authors assert that FAN demonstrates superior performance in periodicity modeling tasks, especially in out-of-domain (OOD) scenarios, outperforming existing neural architectures.\n\nHowever, as is detailed below in this review, the work still misses important references to previous work in the field, namely existing methods, benchmarks, metrics, and dataset introduced for this purpose, and fails to perform many comparisons with these related works. Moreover, some experimental set-ups and resulting figures and findings already appear in previous work. \n\nIn sum, I would consider the work in principle passable as a contribution of some novelty if properly embedded in the context of previous work. The singular focus on FANs as an architecture with no equal as currently alluded to in the abstract, Sections 1, 4.2, 4.3, and 5 needs to be mitigated by the appropriate acknowledgement of the existing work in the field.", "questions": "See weaknesses.", "limitations": "The authors have not done any explicit discussion of the limitations of their work. The discussion of a potential negative societal impact has not been given, but would also not be entirely necessary, given the nature of the work.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1. The work happily cites work as old as Newton 1687, yet fails to properly compare -- theoretically, experimentally, or both -- to contemporary works many of which have proposed their own methods for the same purpose:\n\n[1] Adrian Silvescu. Fourier neural networks. In IJCNN’99. International Joint Conference on Neural Networks. Proceedings (Cat. No.\n99CH36339), volume 1, pages 488–491. IEEE, 1999.\n\n[2] Shuang Liu. Fourier neural network for machine learning. In 2013 International Conference on Machine Learning and Cybernetics, volume 1, pages 285–290. IEEE, 2013\n\n[3] Giambattista Parascandolo, Heikki Huttunen, and Tuomas Virtanen. Taming the waves: sine as activation function in deep neural networks. 2016.\n\n[4] Abylay Zhumekenov, Malika Uteuliyeva, Olzhas Kabdolov, Rustem Takhanov, Zhenisbek Assylbekov, and Alejandro J Castro. Fourier neural networks: A comparative study. arXiv preprint arXiv:1902.03011, 2019.\n\n[5] Jiyoung Lee, Wonjae Kim, Daehoon Gwak, and Edward Choi. Conditional generation of periodic signals with fourier-based decoder. arXiv preprint arXiv:2110.12365, 2021.\n\n[6] Peter Belcak and Roger Wattenhofer. Periodic extrapolative generalisation in neural networks. In 2022 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1066-1073. IEEE, 2022.\n\n2. Closely related variants of the FAN proposed in Section 3 have been also proposed in each of [1]-[5], yet comparisons in Table 1 are made only against the traditional MLP.\n\n3. Figure 3 closely follows the exposition Figures 1&2 of [6].\n\n4. On lines 125-130, the point of comparison should not have been MLP but each of the networks proposed in [1]-[5] across the past 25 years of activity in this research field.\n\n5. Experimentation of Section 4.1, Appendix C, and Figure 4 mirrors that of [5] and [6] and does not acknowledge the existence of essentially equivalent synthetic benchmark in [6], which is by the virtue of the close similarity prior art. Note similar omission under point 3.\n\n6. Experimentation of Section 4.1 glosses over RNNs, arguably the most potent architecture for the task at hand. In later experiments, LSTM is mentioned with other RNNs disregarded.\n\n8. Table 4 acknowledges the existence of [1] and FTNNs but ignores [2]-[5] . Note similar omissions in points 2 & 4.\n\n9. The inadequacy of snake activations described on lines 255-258 and in Appendix D is presented as a new finding but has already been pointed out and experimentally detailed in [6]. Note similar omissions in points 3 & 5.\n\n10. The authors do not answer truthfully in the Reviewer Checklist under point 2. No explicit discussion of the limitations has been done in the paper, as their response to the checklist question indicates.\n\nI strongly suggest authors revisit and expand the related work section, make further acknowledgement to prior art in the abstract, Section 1, Section 3, Section 4, and introduce additional baselines where possible.", "was_revised": true, "final_justification": "I have reviewed the author response and found it deflective. However, given the enthusiasm of other reviewers and the authors themselves, I have decided to raise the score.\n\nIt should, however, be pointed out to the authors that I did not find the claims made in the paper nor their responses to my and other reviews candid.", "rebuttal": "Thank you for your thorough review of the related literature and detailed feedback. We will respond to your comment point-by-point below (we merged some similar comments).\n\n> Q1: The work happily cites work ... yet fails to properly compare -- theoretically, experimentally, or both -- to contemporary works many of which have proposed their own methods for the same purpose.\n\nThanks for your comment. We had cited works [1-5] and compared with them in our original submission, and we are grateful for your pointing out reference [6], which we have added the discussion of [6] in our revised manuscript. Our work is largely different from this work [1-6], specifically:\n\n**For work [1-4], they face challenges in scaling to deeper networks**, i.e., the capacity of their deep networks to fit the Fourier coefficients is independent of the network depth, as analyzed in Section 3. The depth scalability limits their applicability to more complex, general-purpose tasks such as language modeling. Our core differences are, \"we design FAN based on the following principles: 1) the capacity of FAN to represent the Fourier coefficients should be positively correlated to its depth; 2) the output of any hidden layer can be employed to model periodicity using Fourier Series through the subsequent layers.\" We introduced and categorized them as Fourier Neural Networks (FNNs) [1-4] in Related Work Section. In Section 4.3, we conduct experiments to compare our approach with FNNs (we will detailed discuss the difference between FAN and FNNs in Q2), and FNNs cannot fit language modeling tasks but our approach works well.     \n\n**For work [5,6], they focus on different purposes from our work** (we will detailed discuss in Q3).  And work [5] assumes all input signals have the period of 1 (as stated in page 3 of work [5]), which we conducted experiments on the same setting in Appendix B.6, and it cannot fit our periodicity modeling tasks. Work [6] is a comparative study without proposing new method, and RNN performs well in the setting of work [6], but also fails in our periodicity modeling tasks (detailed results are shown in Q6).\n\nSorry for the misunderstanding. To enhance the clarity, we have enhanced the connection between them and added this discussion to our revised manuscript.\n\n> Q2&Q4: Closely related variants of the FAN proposed in Section 3... On lines 125-130, the point of comparison should not have been MLP...\n\nIn our paper, we compare our approach with not only MLP but also previous Fourier-based networks in experiments of Section 4.3. The reason why we formally analyze the differences with MLP in Section 3 and lines 125-130 is that our goal is to explore the likelihood of FAN to offer broad applicability like MLP. \n\nWe provide the analysis of work [1-4] compared to our approach in below table and have added it to the appendix of our revised version. We mainly discuss work [1-3], due to the work [4] is a comparative study without proposing new method. The work [2, 3] can be regraded as the simplified version of work [1], where work [2,3] are fully equivalent to work [1] when hyperparameter m=1 (They can be converted to each other through trigonometric identities, and the differences in their parameters can be absorbed by the learnable W and B). The work [5] cannot fit our periodicity modeling tasks, so we didn't compared with it.\n\n| Metric  | FAN Layer    | Layer of [1]                               | Layer of [2]    | Layer of [3]                     | \n|-|-|-|-|-|\n| **Formula**    | $[\\cos(W_p x) \\Vert \\sin(W_p x) \\Vert \\sigma(B_{\\bar{p}} + W_{\\bar{p}} x)]$ | $W_f \\Pi_m \\cos(W_{a_m} x+B_{a_m})+B_f$   | $W_{f_c} \\cos(W_{a_c} x+B_{a_c}) + B_{f_c} + W_{f_s} \\sin(W_{a_s} x+B_{a_s}) + B_{f_s}$ | $W_f \\sin(W_a x+B_a)+B_f$        |\n| **Num Params** | $(1-\\frac{d_p}{d_o}) (d_i d_o + d_o)$                                    | $m(d_i d_h + d_h) + d_o d_h + d_o$        | $d_i (d_c + d_s) + (d_c + d_s) + d_o (d_c + d_s) + 2d_o$                                | $d_i d_h + d_h + d_o d_h + d_o$  |\n| **FLOPs**      | $(1-\\frac{d_p}{d_o}) \\times 2d_i d_o + \\gamma d_o$                       | $2m d_h d_i + d_h(m-1) + 2d_h d_o + \\gamma m d_h$ | $2d_i (d_c + d_s) + 2d_o (d_c + d_s) + \\gamma (d_c + d_s)$                              | $2d_h(d_i + d_o) + \\gamma d_h$   |\n\n### Notation Key:\n- $d_i = d_{\\text{input}}$ (Input dimension hyperparameter)\n- $d_o = d_{\\text{output}}$ (Output dimension hyperparameter)\n- $d_p$ = FAN layer hyperparameter (default $\\frac{1}{4}d_o$)\n- $d_h$ = Hidden dimension hyperparameter (for [1], [3])\n- $d_c, d_s$ = Layer [2] hyperparameters (cosine/sine branch dimensions)\n- $m$ = Layer [1] hyperparameter (number of projections)\n- $\\gamma$ = FLOPs per nonlinear activation ($\\sigma$, $\\cos$, or $\\sin$)\n\n> Q3 & Q5: Figure 3 closely follows the exposition Figures 1&2 of [6]. Experimentation of Section 4.1, Appendix C, and Figure 4 mirrors...\n\nOur work and work [6] focus on different problems, with only the display form looking like similar. Specifically, the model's input and output in Figure 3 are both scalars, i.e., function input and function output, and we show the results of entire test dataset. The input of the network in Figures 1 & 2 of [6] is the sequence of previous step, and the output is the value of the next step, and they show the median model prediction and the shaded regions after 30 runs of test sample.  This distinction explains their use of RNN. While an RNN excels in the settings of [6], it performs poorly on our periodicity modeling task (See results in Q6).\n\nMoreover, experiments in Section 4.1, Appendix D (Appendix C in Q5 is maybe a typo. Appendix C includes the experimental setup of real-world tasks, which is not related to work [5,6]), and Figure 4 are also different. First, Figure 4 analyzes the entire model training process from another perspective, proving that its inability to fit periodicity is not due to underfitting. There is no similar experiment in work [5,6]. Second, Section 4.1 and Appendix D includes the experiments of periodicity modeling task, which are different from the problems studied in work [5,6]. \n\n> Q6: Experimentation of Section 4.1 glosses over RNNs...\n\nWe compared with LSTM, considering it is generally considered to be a powerful and effective variant in the RNN family. To address your concerns, we have supplemented the results of the standard RNN and another variant of RNN, GRU, below. The results show that our approach outperforms both RNN and GRU on various tasks, including periodicity modeling (the  experiment in Section 4.1), time series torecasting, and language modeling.\n\n| Periodicity Modeling       | training loss (epoch=0) | test loss (epoch=0) | training loss (epoch=100) | test loss (epoch=100) | training loss (epoch=1000) | test loss (epoch=1000) |\n|-|-|-|-|-|-|-|\n| RNN          | 44.44                    | 49.12                | 32.60                      | 37.52                  | 24.71                       | 190.58                  |\n| GRU          | 44.69                    | 39.76                | 35.95                      | 37.76                  | 20.58                       | 40.00                   |\n| Ours          | **39.62**                   | **61.02**                | **2.75**                       | **7.43**                  | **1.05**                        | **4.15**                    |\n||\n\n| Time Series Forecasting | MSE | MAE | \n|-|-|-|\n| RNN          | 1.369                   | 0.829             | \n| GRU          | 0.986                   | 0.694                | \n| Ours          | **0.839**                   | **0.631**                | \n||\n\n| Language Modeling | Train Loss | In-domain Test Loss | OOD Test Loss |\n|-|-|-|-|\n| RNN   | 0.5036     | 0.5264       | 0.6829        |\n| GRU   | 0.4811     | 0.5049       | 0.6392        |\n| Ours   | **0.2434**     | **0.4094**       | **0.6077**  |\n||\n \n> Q7: Table 4 acknowledges the existence of [1] and FTNNs but ignores [2]-[5]...\n\nWe would like to clarify that we did not ignore work [2-5] in Table 4, where we compared the performance of our approach FAN, FNN, FSNN, FTNN, and MLP on language modeling tasks. Please note that FNN and FSNN are more capable generalizations of [1-4] and [5], respectively. As mentioned in Q2, work [2-4] can be regarded as a simplified version of FNN [1]. And work [5] can be regarded as a simplified version of FSNN, where it assumes all input signals have the period of 1. Therefore, in our comparison with FNN [1] and FSNN, we are, in fact, considering all the works [1-5].\n\n> Q8: The inadequacy of snake activations described on lines 255-258 ...\n\nThe comparison with snake in the appendix is not proposed as a new finding, but only illustrates its shortcomings. We have modified the statement and added the discussion of work [6] in our revised manuscript.\n\n> Q9: The authors do not answer truthfully in the Reviewer Checklist under point 2..\n\nThanks for your feedback. Due to space limitations in the main text, we only discussed the limitations of our work in the Justification of Checklist. We have added a section on limitations to our revised manuscript: \n\nFirst, we only demonstrate the effectiveness of FAN on some mainstream real-world tasks (including symbolic formula representation, time series forecasting, language modeling, image recognition, etc.), and we aim to further broaden the applicability of FAN in our future work. Second, although we have explored the generalizability of FAN and confirmed that FAN outperforms the baseline method in some real-world tasks, the boundaries of this model's generalizability remain unknown. However, we have not yet identified specific scenarios where it performs poorly. We leave this for our future work."}, {"review_id": "ra5tV5AZ4A", "reviewer": "Reviewer_GT2m", "summary": "The authors propose Fourier Analysis Networks (FAN), a replacement for MLP blocks that concatenates $cos(W x)$, $sin (W x)$ and a conventional non-linear projection inside every hidden layer. This design is driven by two principles: (1) depth-scaled capacity to learn Fourier coefficients and (2) periodicity accessibility at any hidden layer. The idea is that this should enable FAN to capture periodic patterns effectively while still remaining applicable to a broad range of tasks. They obtain consistent gains on experiments from a variety of domains, including symbolic formula representation, time-series forecasting and language modeling.", "questions": "Would it be possible to do a layer-wise spectral analysis to illustrate how coefficients/frequencies evolve with depth? \n\nDo FAN’s cosine/sine blocks hamper tasks that are intrinsically non-periodic? A negative result would be informative, and in my opinion, it would not reduce the value of the contribution. \n\nSection on prior work says: \"Previous studies have explored Fourier-based networks, but these networks generally perform well on specific tasks, while their performance on more general tasks tends to be poorer\" and lists a number of citations. It would be informative to expand that sentence and section further (perhaps in the appendix) and detail more precisely how the proposed approach differs from the prior work. \n\nI suggest doing some ablation experiments to better understand what are the key contributors in FAN. For instance, freezing W_p or having just sine or cosine function or having FAN layers only in part of the network.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The approach seems to be novel in the space of Fourier Series NNs, specifically in how it integrates sine and cosine functions with other activation functions.\n\nThe variety and scope of experiments are extensive and show impressive generalization across domains. \n\nThe code is provided and in general, the idea is relatively straightforward so it should be reproducible. \n\nAdditional analyses of layer activations and ablations could help to better understand why the approach works better than existing Fourier Series NNs.", "was_revised": true, "final_justification": "The authors have addressed my comments reasonably well. There are a couple more of small suggestions remaining. Overall, I think the work is interesting and relevant.", "rebuttal": "Thank you for your thoughtful comment and constructive feedback. We will respond to your comment point-by-point below.\n\n> Additional analyses of layer activations and ablations could help to better understand why the approach works better than existing Fourier Series NNs. Would it be possible to do a layer-wise spectral analysis to illustrate how coefficients/frequencies evolve with depth?\n\nThanks for your suggestions. We conduct experiments on layer-wise spectral analysis and have added the experimental results to our revised manuscript.  We perform a Fast Fourier Transform (FFT) on each layer's outputs and calculate four key metrics to quantify the spectral characteristics: \n1)  Spectral Centroid: Measures the \"center of mass\" of the spectrum, indicating whether the layer's features are concentrated in low or high-frequency regions. \n2)  Spectral Sparsity (L1/L2 Norm): Quantifies how concentrated the spectral energy is within a few frequency bins. A higher value implies a more structured and less noisy signal. \n3)  Spectral Entropy: Measures the uniformity and predictability of the spectrum. A lower entropy indicates a more ordered and well-defined spectral structure. \n4)  Dominant Energy Ratio (Top-5): The proportion of total spectral energy contained within the top 5 most dominant frequency components, indicating how focused the representation is on key periodic features.\n\nThe results reveal a highly effective multi-stage learning process, which is more sophisticated than a simple monotonic evolution of frequencies. We observe a clear three-stage \"Deconstruction-Exploration-Reconstruction\" mechanism:\n\n1. Initial Approximation (Layer 1): The first layer rapidly forms an initial, highly-focused approximation of the signal, as shown by its very high Dominant Energy Ratio (96.1%).\n\n2. Feature Deconstruction and Exploration (Layers 2-8): To model the function's complex, non-sinusoidal components (especially the x mod 3 term, which requires a wide range of Fourier series terms), the intermediate layers must first \"deconstruct\" the signal. This is evidenced by a sharp increase in Spectral Entropy and a decrease in the Dominant Energy Ratio. The network actively disperses energy across a broader spectrum to explore and capture these challenging features, showcasing the flexibility afforded by its depth.\n\n3. Integration and Reconstruction (Layers 9-11): In the final layers, the model's task shifts from exploration to integration. It \"reconstructs\" a final, efficient representation from the features learned in the middle layers. This is marked by a dramatic decrease in both Spectral Entropy and Spectral Centroid, alongside a sharp increase in the Dominant Energy Ratio to a final value of 93.8%. The network converges to a \"clean\", low-frequency, and highly structured representation that is optimal for the final linear layer to map to the target output.\n\n| Layer        | Spectral Centroid | Spectral Sparsity | Spectral Entropy | Dominant Energy Ratio (Top-5) |\n|--------------|-------------------|-------------------|------------------|-------------------------------|\n| FAN Layer 1  | 4.1213            | 3.4767            | 1.2264           | 0.9612                        |\n| FAN Layer 2  | 2.8760            | 5.0003            | 3.2549           | 0.7602                        |\n| FAN Layer 3  | 2.8804            | 5.0626            | 3.1556           | 0.7807                        |\n| FAN Layer 4  | 2.8810            | 4.7149            | 2.6616           | 0.8426                        |\n| FAN Layer 5  | 3.0820            | 4.5832            | 2.2248           | 0.8753                        |\n| FAN Layer 6  | 3.0815            | 5.2388            | 2.5560           | 0.8378                        |\n| FAN Layer 7  | 2.6955            | 5.8367            | 3.0115           | 0.7806                        |\n| FAN Layer 8  | 2.9132            | 5.5387            | 2.7301           | 0.8086                        |\n| FAN Layer 9  | 2.7376            | 4.1371            | 1.6760           | 0.8986                        |\n| FAN Layer 10 | 2.1266            | 3.1509            | 1.0673           | 0.9356                        |\n| FAN Layer 11 | 1.7721            | 2.9775            | 0.9270           | 0.9375                        |\n||\n\n\n\n> Do FAN’s cosine/sine blocks hamper tasks that are intrinsically non-periodic? A negative result would be informative, and in my opinion, it would not reduce the value of the contribution.\n\nFAN's cosine/sine blocks do not hamper non-periodic tasks. Specifically, non-periodic tasks can be categorized into three types: 1) Completely non-periodic tasks: Tasks with no periodic patterns at all. 2) Implicitly periodic tasks: Tasks that are mathematically non-periodic but can be viewed as periodic from a different perspective. For example, the operation y = a + b can be seen as the result of periodic operations of single-digit addition and carrying. 3) Partially periodic tasks: Tasks where only some components are periodic, such as a rising spiral. For implicitly and partially periodic tasks, which are common in the real world, FAN can achieve good results. For completely non-periodic tasks, FAN degenerates to a standard MLP, and we have not yet identified specific scenarios where it performs negatively. We leave this for our future work.\n\n> Section on prior work says: \"Previous studies have explored Fourier-based networks, but these networks generally perform well on specific tasks, while their performance on more general tasks tends to be poorer\" and lists a number of citations. It would be informative to expand that sentence and section further (perhaps in the appendix) and detail more precisely how the proposed approach differs from the prior work.\n\nThanks for your suggestions! We have expanded on this sentence as follows: Zuo and Cai (2005) proposed a control scheme based on Fourier Neural Networks, specifically for solving the tracking control problem for a class of unknown nonlinear systems. Tan (2006) studied the application of Fourier Neural Networks in the problem of aircraft engine fault classification. Jiang et al. (2022) utilized Fourier series to improve the accuracy of time series forecasting tasks. Chen et al. (2022) proposed a Fourier Imager Network that can perform end-to-end phase recovery and image reconstruction for raw holograms of novel samples. Uteuliyeva et al. investigated the poor performance of Fourier Neural Networks on general tasks such as language modeling. These works are often task-specific, and it has been proven difficult to handle more general tasks. We have added the detailed introduction of each citations to the appendix of our revised manuscript. \n\nHow the proposed approach differs from the prior work: Previous Fourier-based networks face challenges in scaling to deeper networks, i.e., the capacity of their deep networks to fit the Fourier coefficients is independent of the network depth, as analyzed in Section 3. The depth scalability limits their applicability to more complex, general-purpose tasks such as language modeling. Our core differences are, \"we design FAN based on the following principles: 1) the capacity of FAN to represent the Fourier coefficients should be positively correlated to its depth; 2) the output of any hidden layer can be employed to model periodicity using Fourier Series through the subsequent layers.\" In Section 4.3, we categorized these Fourier-based networks into three main types and compared with them, the experimental results show that only our approach achieves superior performance on both periodicity modeling task and language modeling task.\n\n\n> I suggest doing some ablation experiments to better understand what are the key contributors in FAN. For instance, freezing W_p or having just sine or cosine function or having FAN layers only in part of the network.\n\nThanks for your suggestions.  We conduct experiments on just cosine function and having FAN layers only in part of the network,  and have added the experimental results to our revised manuscript. The results show that our current approach demonstrates a clear advantage over the variants in Periodicity Modeling and Language Modeling tasks.\n\n| Periodicity Modeling       | training loss (epoch=0) | test loss (epoch=0) | training loss (epoch=100) | test loss (epoch=100) | training loss (epoch=1000) | test loss (epoch=1000) |\n|-|-|-|-|-|-|-|\n| FAN_cos                           | 39.85                   | 63.42               | **2.67**                      | 10.18                | 1.80                       | 5.26                   |\n| FAN_replace_first_1/3_part   | 39.54                   | 46.15               | 2.95                      | 6.81                 | 1.37                       | 45.44                  |\n| FAN_replace_last_1/3_part | 42.82                   | 55.46               | 21.96                     | 27.86                | 22.79                      | 30.51                  |\n| FAN                  | **39.62**                    | **61.02**               | 2.75                      | **7.43**                 | **1.05**                       | **4.15**                   |\n||\n\n| Language Modeling | Train Loss | In-domain Test Loss | OOD Test Loss |\n|-|-|-|-|\n| FAN_cos           | 0.2419     | 0.4802       | 0.7727        |\n| FAN_replace_first_1/3_part    | 0.2693     | 0.4313              | 0.6700        |\n| FAN_replace_last_1/3_part | **0.2417**     | 0.4660              | 0.8052        |\n| FAN               | 0.2434     | **0.4094**       | **0.6077**        |\n||"}, {"review_id": "EDyykHoCmA", "reviewer": "Reviewer_rvp8", "summary": "The paper addresses the limitation of traditional neural networks that can’t properly model periodic functions. The paper introduces Fourier Analysis Networks (FAN), a novel network architecture, based on Fourier analysis, that can replace MLP blocks so models can better represent periodic structure with fewer parameters and FLOPs. The paper shows the effectiveness of FAN, compared to other baseline architectures like MLP, KAN and Transformer on a variety of periodic modelling tasks, time series forecasting and sentiment analysis tasks.", "questions": "I believe FAN represents a valuable contribution for modeling periodic functions, but I struggle to see how it qualifies as a general-purpose architecture given the experimental evidence. Could the authors clarify their vision for FAN's practical applications? Specifically, which real-world tasks would benefit from the Fourier-based inductive bias, and how would you recommend identifying when FAN layers would be preferable to standard MLPs? While I understand the authors' argument that many machine learning tasks may benefit from FAN due to hidden periodicity, there's still a gap between this theoretical vision and the experimental validation.\n\n\nMinor point:\nTo improve clarity, I’d suggest moving the description of the FAN(Gated) variant to the main paper.", "limitations": "I'd recommend the authors include a more thorough discussion of the limitations in their real-world task selection, specifically addressing why certain tasks were chosen and which types of applications might not benefit from FAN.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The writing is clear and the motivation is well explained with experiments showing that standard networks extrapolate poorly in periodic data.\n- The proposed architecture embeds efficiently periodicity while reducing the parameter count.\n- The experimental results on periodic tasks are convincing and the method clearly improves OOD generalization.\n\n## Weaknesses\nMy main concern is that the paper claims FAN to be a general-purpose model, while lacking extensive experiment analysis to support this claim. The paper compares against vanilla architectures rather than specialized state-of-the-art models. If FAN layers can serve as replacements for MLPs in existing architectures, would current specialized SOTA actually benefit from this substitution? Sentiment analysis seems like an odd choice to showcase FAN's language modeling benefits, there is no clear periodic structure that Fourier analysis would capture. Combined with the lack of comparison against SOTA models this experiment fails to demonstrate any meaningful advantage for real-world language tasks.", "was_revised": true, "final_justification": "The authors addressed my concerns about the lack of experimental evidence to support the claim of FAN being a general-purpose model. They provided a better explanation for their motivation and pointed to additional experiments in the Appendix. Although I still believe that this claim of general-purpose capabilities is a bit bold and requires further experimentation, I agree that it can be left for future work, especially since the authors promised to rephrase their statements and add discussion of limitations. I'm increasing my score accordingly.", "rebuttal": "Thank you for your insightful comments. We will respond to your comment point-by-point below.\n\n> I believe FAN represents a valuable contribution for modeling periodic functions, but I struggle to see how it qualifies as a general-purpose architecture given the experimental evidence.\n\nThanks for recognizing the value of our work. Compared with existing Fourier-based networks, which are typically designed for specific tasks, we have explored the capability of FAN to be applied to more types of tasks. Based on the current experimental results, we observed that FAN has the potential to be applied to a range of real-world tasks. However, whether FAN equips the general-purpose capabilities remains a question to be answered. We will adjust the statements like \"maintains general-purpose modeling capability\" to more precise descriptions, such as \"maintains capability to be applied to more types of tasks\" in our revised manuscript.\n\n> Could the authors clarify their vision for FAN's practical applications? Specifically, which real-world tasks would benefit from the Fourier-based inductive bias, and how would you recommend identifying when FAN layers would be preferable to standard MLPs? &  I'd recommend the authors include a more thorough discussion of the limitations in their real-world task selection, specifically addressing why certain tasks were chosen and which types of applications might not benefit from FAN.\n\nThat's a good question. Periodicity is a fundamental and ubiquitous characteristic in both the natural sciences and human activities. However, existing general-purpose neural networks, such as MLPs and Transformers, exhibit inherent limitations in effectively capturing and modeling these periodic features, while existing Fourier-based networks face challenges in scaling to deeper networks and are typically designed for specific tasks. To this end, we propose FAN to effectively address periodicity modeling challenges while maintaining capability to be applied to more types of tasks. \n\nVarious forms of real-world data contain either explicit or implicit periodic patterns. We believe that these real-world tasks will benefit from FAN. When the task is completely aperiodic, FAN degenerates to MLP. Although we have explored the generalizability of FAN and confirmed that FAN outperforms the baseline method in some mainstream real-world tasks (including symbolic formula representation, time series forecasting, language modeling, image recognition, etc.), the boundaries of this model's generalizability remain unknown. However, we have not yet identified specific scenarios where it performs poorly. We leave this for our future work. We have added this discussion to our revised manuscript.\n\n>  The paper compares against vanilla architectures rather than specialized state-of-the-art models. If FAN layers can serve as replacements for MLPs in existing architectures, would current specialized SOTA actually benefit from this substitution?\n\nYes, we hope it can replace MLP in some aspects, and we attempted to demonstrate this in our paper (due to space and experimental limitations, we only explored its advantages on existing tasks). As shown in Appendix B.3 and B.4, we compared SOTA methods with using FAN in Large Language Modeling scenario and Solving SciML Problems scenario, where it outperforms specialized SOTA methods. \n\n> Sentiment analysis seems like an odd choice to showcase FAN's language modeling benefits.\n\nThanks for your comment. Our intention with the experiments was twofold. First, our primary goal was to validate FAN's effectiveness in the fundamental task of language modeling, which we demonstrated through its ability to achieve a lower entropy loss. This part of the experiment directly addresses its language modeling capabilities. Second, we wanted to conduct an exploratory investigation into whether FAN has the potential to be applied to higher-level, downstream NLP tasks, such as sentiment analysis (one of the most widespread real-world NLP applications, used for analyzing user reviews, social media sentiment, and brand monitoring). If you think this result is odd, we are happy to remove the sentiment analysis results, with only maintaining the results of language modeling, in the camera-ready version.\n\n> To improve clarity, I’d suggest moving the description of the FAN(Gated) variant to the main paper.\n\nThank you for your suggestion! We have moved the description of the FAN(Gated) variant to the main paper in our revised version."}]

Author rebuttal:

Meta-review: This paper develops a novel neural network that effectively addresses the challenges of periodicity modeling. All reviews are positive, specifically: A, BA, A, BA. Reviewers kgu3 and GT2m commend the work as "a technically solid paper with high impact"; Reviewer rvp8 recognizes it for its "well-explained motivation, valuable contribution, and convincing results." Most concerns have been adequately resolved. Therefore, I recommend the acceptance of this submission. Additionally, I expect that the authors will incorporate the suggested modifications from the rebuttal phase into the final version.
Decision: Accept (poster)