FAN: Fourier Analysis Networks

## Abstract

Abstract Despite the remarkable success achieved by neural networks, particularly those represented by MLP and Transformer, we reveal that they exhibit potential flaws in the modeling and reasoning of periodicity, i.e., they tend to memorize the periodic data rather than genuinely understanding the underlying principles of periodicity.
However, periodicity is a crucial trait in various forms of reasoning and generalization, underpinning predictability across natural and engineered systems through recurring patterns in observations.
In this paper, we propose \modelname , a novel network architecture based on Fourier Analysis, which empowers the ability to efficiently model and reason about periodic phenomena. By introducing Fourier Series, the periodicity is naturally integrated into the structure and computational processes of the neural network, thus achieving a more accurate expression and prediction of periodic patterns.
As a promising substitute to multi-layer perceptron (MLP), \modelname can seamlessly replace MLP in various models with fewer parameters and FLOPs.
Through extensive experiments, we demonstrate the effectiveness of \modelname in modeling and reasoning about periodic functions, and the superiority and generalizability of \modelname across a range of real-world tasks, including symbolic formula representation, time series forecasting, and language modeling. † † footnotetext: This work was supported by a cooperation project between Peking University and ByteDance Company. During this time, Yihong was also an intern at ByteDance. ‡ ‡ footnotetext: The code is available at https://github.com/YihongDong/FAN

## 1 Introduction

The flourishing of modern machine learning and artificial intelligence is inextricably linked to the revolutionary advancements in the foundational architecture of neural networks. For instance, multi-layer perceptron (MLP) (Rosenblatt, ; Haykin, ) plays a pivotal role in laying the groundwork for current deep learning models, with its expressive power guaranteed by the universal approximation theorem (Hornik et al., ).
Recent claims about the impressive performance of large models on various tasks are typically supported by Transformer architecture (Vaswani et al., ; Touvron et al., ; OpenAI, ).
In this context, the community’s enthusiasm for research on neural networks has never diminished.
Some emerged neural networks demonstrate notable capabilities in specific fields (Gu & Dao, ; Liu et al., ), sparking widespread discussion within the community.

Figure: Figure 1: The performance of different neural networks within and outside the domain of their training data for the sine function, where $x$ is a scalar variable.
Refer to caption: x1.png

Beneath the surface of apparent prosperity, we uncover a critical issue that remains in existing neural networks: they struggle to model the periodicity from data. We showcase this issue through an empirical study as illustrated in Figure 1. The results indicate that existing neural networks, including MLP (Rosenblatt, ), KAN (Liu et al., ), and Transformer (Vaswani et al., ), face difficulties in fitting periodic functions, even on a simple sine function.
Although they demonstrate proficiency in interpolation within the domain of training data, they tend to falter when faced with extrapolation challenges of test data, especially in the out-of-domain (OOD) scenarios. Therefore, their generalization capacity is primarily dictated by the scale and diversity of the training data, rather than by the learned principles of periodicity to perform reasoning. We argue that periodicity is an essential characteristic in various forms of reasoning and generalization, as it provides a basis for predictability in many natural and engineered systems by leveraging recurring patterns in observations.

In this paper, we investigate a key research problem: How to enable neural networks to model periodicity? One core reason existing neural networks fail to model periodicity is that they heavily rely on data-driven optimization without explicit mechanisms to understand the underlying principles in the data. To this end, we propose a Fourier Analysis Network (\modelname), a novel neural network framework based on Fourier Analysis. By leveraging the power of Fourier Series, we explicitly encode periodic patterns within the neural network, offering a way to model the general principles from the data. \modelname holds great potential as a substitute to traditional MLP, which not only exhibits exceptional capabilities in periodicity modeling but also demonstrates competitive or superior effects on general tasks.

To verify the effectiveness of \modelname, we conduct extensive experiments from two main aspects: periodicity modeling and application of real-world tasks. 1) For periodicity modeling, \modelname achieves significant improvements in fitting both basic and complex periodic functions, compared to existing neural networks (including MLP, KAN, and Transformer), particularly in OOD scenarios. 2) \modelname demonstrates superior performance in real-world tasks, including symbolic formula representation, time series forecasting, and language modeling. The experimental results indicate that \modelname outperform baselines (including MLP, KAN, and Transformer) for symbolic formula representation task, and Transformer with \modelname surpasses the competing models (including Transformer, LSTM (Hochreiter & Schmidhuber, ), and Mamba (Gu & Dao, )), for time series forecasting and language modeling tasks.
As a promising substitute to MLP, \modelname improves the model’s generalization performance meanwhile reducing the number of parameters and floating point of operations (FLOPs) employed.
We believe \modelname is promising to be an important part of the fundamental model backbone.

## 2 Preliminary Knowledge

Fourier Analysis (Stein & Weiss, ; Duoandikoetxea, ) is a mathematical framework that decomposes functions into their constituent frequencies, revealing the underlying periodic structures within complex functions. At the heart of this analysis lies Fourier Series (Tolstov, ), which expresses a periodic function as an infinite sum of sine and cosine terms. Mathematically, for a function $f(x)$, its Fourier Series expansion can be represented as:

$$ $f(x)=a_{0}+\sum_{n=1}^{\infty}\left(a_{n}\cos\left(\frac{2\pi nx}{T}\right)+b_ {n}\sin\left(\frac{2\pi nx}{T}\right)\right),$ (1) $$

where $T$ is the period of the function, and the coefficients $a_{n}$ and $b_{n}$ are determined by integrating the function over one period:

$$ $a_{n}=\frac{1}{T}\int_{0}^{T}f(x)\cos\left(\frac{2\pi nx}{T}\right)\,dx,\quad b _{n}=\frac{1}{T}\int_{0}^{T}f(x)\sin\left(\frac{2\pi nx}{T}\right)\,dx.$ (2) $$

The power of Fourier Series lies in its ability to represent a wide variety of functions, including non-periodic functions through periodic extensions, enabling the extraction of frequency components.
Building on this mathematical foundation, \modelname  aims to embed the periodic characteristics directly into network architecture, enhancing generalization capabilities and performance on various tasks, particularly in scenarios requiring the identification of patterns and regularities.

## 3 Fourier Analysis Network ( \modelname )

In this section, we first construct a simple neural network modeled by the formula of Fourier Series, and then on this basis, we design \modelname and provide its details. Finally, we discuss the difference between the \modelname layer and the MLP layer.

Consider a task involving input-output pairs $\{x_{i},y_{i}\}$, with the objective of identifying a function $f(x):\mathbb{R}^{d_{x}}\rightarrow\mathbb{R}^{d_{y}}$ that approximates the relationship such that $y_{i}\approx f(x_{i})$ for all $x_{i}$, where $d_{x}$ and $d_{y}$ denote the dimensions of $x$ and $y$, respectively. To build a simple neural network $f_{\text{S}}(x)$ that represents Fourier Series expansion of the function, specifically $\mathcal{F}\{f(x)\}$, as described in Eq. (1), we can express $f_{\text{S}}(x)$ as follows:

$$ $\displaystyle f_{\text{S}}(x)$ $\displaystyle\triangleq a_{0}+\sum_{n=1}^{N}\left(a_{n}\cos\left(\frac{2\pi nx }{T}\right)+b_{n}\sin\left(\frac{2\pi nx}{T}\right)\right),$ (3) $\displaystyle\mathop{=}\limits^{(\text{I})}a_{0}+\sum_{n=1}^{N}\left(w^{c}_{n} \cos\left(w^{\text{in}}_{n}x\right)+w^{s}_{n}\sin\left(w^{\text{in}}_{n}x \right)\right),$ $\displaystyle\mathop{=}\limits^{(\text{II})}B+[w^{c}_{1},w^{c}_{2},\cdots,w^{c }_{n}]\cos([w^{\text{in}}_{1}||w^{\text{in}}_{2}||\cdots||w^{\text{in}}_{n}]x)$ $\displaystyle\quad+[w^{s}_{1},w^{s}_{2},\cdots,w^{s}_{n}]\sin([w^{\text{in}}_{ 1}||w^{\text{in}}_{2}||\cdots||w^{\text{in}}_{n}]x)$ $\displaystyle=B+W_{c}\cos(W_{\text{in}}x)+W_{s}\sin(W_{\text{in}}x),$ $\displaystyle\mathop{=}\limits^{(\text{III})}B+W_{\text{out}}[\cos(W_{\text{in }}x)||\sin(W_{\text{in}}x)],$ $$

where $B\in\mathbb{R}^{d_{y}},W_{\text{in}}\in\mathbb{R}^{N\times d_{x}}$, and $W_{\text{out}}\in\mathbb{R}^{d_{y}\times 2N}$ are learnable parameters, $(\text{I})$ follows that the computation of $a_{n}$ and $b_{n}$ computed via Eq. (2) is definite integral, $(\text{II})$ and $(\text{III})$ follows the equivalence of the matrix operations, $[\cdot||\cdot]$ and $[\cdot,\cdot]$ denotes the concatenation along the first and second dimension, respectively.

To fully leverage the advantages of deep learning, we can stack the aforementioned network $f_{\text{S}}(x)$ to form a deep network $f_{\text{D}}(x)$, where the $i$-th layer, denoted as $l_{i}(x)$, retains the same structural design as $f_{\text{S}}(x)$. Therefore, $f_{\text{D}}(x)$ can be formulated as:

$$ $f_{\text{D}}(x)=l_{L}\circ l_{L-1}\circ\cdots\circ l_{1}\circ x,$ (4) $$

where $l_{1}\circ x$ denotes the application of the left function $l_{1}$ to the right input $x$, that is $l_{1}(x)$. However, we discover that the direct stacking of $f_{\text{S}}(x)$ results in the primary parameters of the model $f_{\text{D}}(x)$ focusing on learning the angular frequency ($\omega_{n}=\frac{2\pi n}{T}$), thereby neglecting the learning of the Fourier coefficients ($a_{n}$ and $b_{n}$), as follows:

$$ $\displaystyle f_{\text{D}}(x)$ $\displaystyle=l_{L}(l_{L-1}\circ l_{L-2}\circ\cdots\circ l_{1}\circ x)$ $\displaystyle=B^{L}+W^{L}_{\text{out}}[\cos(W^{L}_{\text{in}}(l_{1:L-1}\circ x )||\sin(W^{L}_{\text{in}}(l_{1:L-1}\circ x))]$ (5) $$

where $l_{1:L-1}\circ x$ is defined as $l_{L-1}\circ l_{L-2}\circ\cdots\circ l_{1}\circ x$, $W^{L}_{\text{in}}(l_{1:L-1}\circ x)$ is used to approximate the angular frequencies, and $W^{L}_{\text{out}}$ is used to approximate the Fourier coefficients. Therefore, the capacity of $f_{\text{D}}(x)$ to fit the Fourier coefficients is independent of the depth of $f_{\text{D}}(x)$, which is an undesirable outcome.

To this end, we design \modelname based on the following principles: 1) the capacity of \modelname to represent the Fourier coefficients should be positively related to its depth; 2) the output of any hidden layer can be employed to model periodicity using Fourier Series through the subsequent layers. The first one enhances the expressive power of \modelname for periodicity modeling by leveraging its depth, while the second one ensures that the features of \modelname’s intermediate layers are available to perform periodicity modeling.

Suppose we decouple $f_{\text{S}}(x)$ as follows:

$$ $f_{\text{S}}(x)=f_{out}\circ f_{in}\circ x,$ (6) $$

where

$$ $\displaystyle f_{in}(x)=[\cos(W_{\text{in}}x)||\sin(W_{\text{in}}x)],$ (7) $\displaystyle f_{out}(x)=B+W_{\text{out}}x.$ (8) $$

To satisfy both principles, the inputs of the intermediate layers in \modelname necessitate to employ $f_{in}$ and $f_{out}$ simultaneously, rather than applying them sequentially.

Finally, \modelname is designed on this basis, with the \modelname layer $\phi(x)$ defined as below:

$$ $\phi(x)\triangleq[\cos(W_{p}x)||\sin(W_{p}x)||\sigma(B_{\bar{p}}+W_{\bar{p}}x)],$ (9) $$

where $W_{p}\in\mathbb{R}^{d_{x}\times d_{p}},W_{\bar{p}}\in\mathbb{R}^{d_{x}\times d
_{\bar{p}}}$, and $B_{\bar{p}}\in\mathbb{R}^{d_{\bar{p}}}$ are learnable parameters (with the hyperparameters $d_{p}$ and $d_{\bar{p}}$ indicating the first dimension of $W_{p}$ and $W_{\bar{p}}$, respectively), the layer output $\phi(x)\in\mathbb{R}^{2d_{p}+d_{\bar{p}}}$, and $\sigma$ denotes the activation function, which can further enhance its expressive power for periodicity modeling.

Figure: Figure 2: Illustrations of MLP layer $\Phi(x)$ vs. \modelname layer $\phi(x)$.
Refer to caption: x2.png

The entire \modelname is defined as the stacking of the \modelname layer $\phi(x)$:

$$ $\text{FAN}(x)=\phi_{L}\circ\phi_{L-1}\circ\cdots\circ\phi_{1}\circ x,$ (10) $$

where

$$ $\phi_{l}(x)=\left\{\begin{array}[]{ll}[\cos(W^{l}_{p}x)||\sin(W^{l}_{p}x)|| \sigma(B^{l}_{\bar{p}}+W^{l}_{\bar{p}}x)],,&\text{if }l<L,\\ B^{L}+W^{L}x,&\text{if }l=L,\end{array}\right.$ (11) $$

The illustrations of the MLP layer $\Phi(x)$ vs. the \modelname layer $\phi(x)$ are shown in Figure 2.
Note that the \modelname layer $\phi(x)$ computed via Eq. (9) can seamlessly replace the MLP layer $\Phi(x)$ computed via Eq. (12) in various models with fewer parameters and FLOPs. The number of parameters and FLOPs of the \modelname layer compared to the MLP layer are presented in Table 1.

**Table 1: Comparison of MLP layer and \modelname layer, where $d_{\text{p}}$ is a hyperparameter of \modelname layer and defaults to $\frac{1}{4}d_{\text{output}}$ in this paper, $d_{\text{input}}$ and $d_{\text{output}}$ denote the input and output dimensions of the neural network layer, respectively. In our evaluation, the FLOPs for any arithmetic operations are considered as 1, and for Boolean operations as 0.**
|  | MLP Layer | \modelname layer |
| --- | --- | --- |
| Formula | $\Phi(x)=\sigma(B_{m}+W_{m}x)$ | $\phi(x)=[\cos(W_{p}x)||\sin(W_{p}x)||\sigma(B_{\bar{p}}+W_{\bar{p}}x)]$ |
| \hdashlineNum of Params | $(d_{\text{input}}\times d_{\text{output}})+d_{\text{output}}$ | $(1-\frac{d_{p}}{d_{\text{output}}})\times((d_{\text{input}}\times d_{\text{<br>output}})+d_{\text{output}})$ |
| \hdashlineFLOPs | $2\times(d_{\text{input}}\times d_{\text{output}})$<br>$+\text{FLOPs}_{\text{non-linear}}\times d_{\text{output}}$ | $(1-\frac{d_{p}}{d_{\text{output}}})\times 2\times(d_{\text{input}}\times d_{<br>\text{output}})$<br>$+\text{FLOPs}_{\text{non-linear}}\times d_{\text{output}}$ |

## 4 Experiments

In this section, we first introduce the baselines and implementation details of our experiments. Next, we verify the effectiveness of \modelname in modeling and reasoning about periodic functions (Section 4.1). Finally, we demonstrate the superiority and generalizability of \modelname across a range of real-world tasks, including symbolic formula representation (Section 4.2), time series forecasting (Section 4.3), and language modeling (Section 4.4).

#### Baselines.

In our experiments, we mainly compare \modelname with the following baselines. 1) MLP (Rosenblatt, ): the most classic model, which is widely used in the backbone of various models. 2) Transformer (Vaswani et al., ): a prevalent model known for its self-attention mechanism, which achieves outstanding performance on various tasks. 3) KAN (Liu et al., ): an emerged model specialized for symbolic formula representation, which uses the b-spline functions instead of fixed activation functions. 4) LSTM (Hochreiter & Schmidhuber, ): a well-known recurrent neural network (RNN) that can capture long-term dependencies on sequential data. 5) Mamba (Gu & Dao, ): an emerged selective state space model (SSM) that achieves competitive performance on some tasks with sequential inputs. Moreover, we also include the following variants of \modelname into our comparisons: I) FAN (Gated): a variant of \modelname that adds gates to control the tendency of the layer, with the formula defined as $\phi_{g}(x)=[g\cdot\cos(W_{p}x)||g\cdot\sin(W_{p}x)||(1-g)\cdot\sigma(B_{\bar{
p}}+W_{\bar{p}}x)]$, where $g$ is a learnable parameter. II) Transformer with FAN and Transformer with FAN (Gated): we replace each MLP layer in Transformer with the \modelname layer computed via Eq. (9) and the layer of \modelname (Gated), respectively.

#### Implementation Details.

We conduct our experiments on a single GPU of Tesla A100-PCIe-40G.
Unless otherwise specified, we use the following hyperparameters in the experiments. The model architecture consists of 3-12 layers, the activation function $\sigma$ is set to GELU (Hendrycks & Gimpel, ), and the dimension of the projection matrix $W_{p}$ is set to $d_{p}=\frac{1}{4}d_{h}$, where $d_{h}$ denotes the dimension of the hidden layers. We employ the AdamW optimizer (Loshchilov & Hutter, ) for the model’s training process. More experimental details and comprehensive setups of each task can be found in Appendix C.

Figure: Figure 3: The performance of \modelname in periodicity modeling compared to MLP, KAN, and Transformer, where the green line represents the test data within the domain of the training data, while the blue line represents the test data outside the domain of the training data.
Refer to caption: x3.png

### 4.1 Periodicity Modeling

#### Setup.

In periodic modeling tasks, we select periodic functions with practical significance and compare the models’ performance in learning the underlying principles of periodicity. Specifically, we generate data from periodic functions over a large domain, using a portion of this domain as training data and the entire domain as test data, i.e., a part of test data would be out of the domain of training data. In this task, we compare \modelname  and its variant, \modelname (Gated), with MLP, KAN, and Transformer. The input of each task is a scalar.

#### Results.

Figure 3 illustrates the performance of \modelname and other baselines in periodicity modeling.
The results indicate that existing neural networks, including MLP, KAN, and Transformers, exhibit notable deficiencies in their ability to model periodicity. Although they attempt to fit these periodic functions, their ability limits their performance in modeling a large domain of periodicity. In contrast, \modelname  significantly outperforms the baselines in all these tasks of periodicity modeling. Moreover, \modelname  performs exceptionally well on test data both within and outside the domain of the training data, indicating that it is genuinely modeling periodicity rather than merely memorizing the training data.

Figure: Figure 4: Comparison of training and test losses for different models on the tasks of learning complex periodic functions.
Refer to caption: x6.png

We also analyze the training process of different models on the tasks of learning complex periodic functions, as illustrated in Figure 4, which leads to the following findings. 1) \modelname far exceeds the other baselines in both convergence speed and final effects. 2) In comparison to \modelname, \modelname (Gated) often achieves faster convergence, but the final performance remains comparable. 3) Although the baselines show stabilization or gradual reductions in training loss as the number of epochs increases, their modelingy may have diverged considerably from the distribution of the test data, resulting in a sharp increase in test loss. This phenomenon further demonstrates the shortcomings of these models in capturing periodicity.

### 4.2 Symbolic Formula Representation

#### Setup.

Symbolic formula representation is a common task in both mathematics and physics. We follow the experiments conducted in KAN’s paper (Liu et al., ), adhering to the same tasks, data, hyperparameters, and baselines. In addition to the original baselines, we also include Transformer for comparison in this task.

#### Results.

Figure 5 demonstrates the performance of different models applied to four common functions in mathematics and physics. From Figure 5, we can observe that while KAN remains competitive with \modelname  when the number of parameters is small, its performance declines significantly as the number of parameters increases. In contrast, as the number of parameters grows, \modelname  consistently outperforms the other baselines, including MLP, KAN, and Transformer, in fitting these functions, despite many of these functions being only partially periodic or entirely non-periodic. These results indicate that although \modelname  enhances its ability to model periodicity, it does not compromise its capacity to fit non-periodic functions.

Figure: Figure 5: Comparisons of \modelname with the baselines, including MLP, KAN, and Transformer, across varying numbers of parameters on symbolic formula representation tasks.
Refer to caption: x7.png

**Table 2: Performance of different sequence models on time series forecasting tasks, where Input Length = 96, the bold values indicate the lowest value on each row, and the improve means the relative improvements of using \modelname and \modelname (Gated) based on Transformer.**
| Dataset | Output<br>Length | LSTM<br>(12.51 M) | Mamba<br>(12.69 M) | Transformer<br>(12.12 M) | Transformer with \modelname (11.06 M) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gated | Default |  |  |  |  |  |  |  |  |  |  |
| MSE $\downarrow$ | MAE $\downarrow$ | MSE $\downarrow$ | MAE $\downarrow$ | MSE $\downarrow$ | MAE $\downarrow$ | MSE $\downarrow$ | MAE $\downarrow$ | MSE $\downarrow$ | MAE $\downarrow$ |  |  |
| Weather | 96 | 1.069 | 0.742 | 0.552 | 0.519 | 0.413 | 0.438 | 0.292 | 0.380 | 0.313 | 0.431 |
| 192 | 1.090 | 0.778 | 0.700 | 0.595 | 0.582 | 0.540 | 0.535 | 0.550 | 0.472 | 0.525 |  |
| 336 | 0.992 | 0.727 | 0.841 | 0.667 | 0.751 | 0.626 | 0.637 | 0.602 | 0.719 | 0.581 |  |
| 720 | 1.391 | 0.892 | 1.171 | 0.803 | 0.967 | 0.715 | 0.845 | 0.706 | 0.732 | 0.670 |  |
| Exchange | 96 | 0.938 | 0.794 | 0.908 | 0.748 | 0.777 | 0.681 | 0.685 | 0.644 | 0.657 | 0.623 |
| 192 | 1.241 | 0.899 | 1.328 | 0.925 | 1.099 | 0.800 | 0.998 | 0.757 | 0.968 | 0.741 |  |
| 336 | 1.645 | 1.048 | 1.512 | 0.992 | 1.614 | 1.029 | 1.511 | 0.961 | 1.266 | 0.905 |  |
| 720 | 1.949 | 1.170 | 2.350 | 1.271 | 2.163 | 1.204 | 1.658 | 1.104 | 2.063 | 1.205 |  |
| Traffic | 96 | 0.659 | 0.359 | 0.666 | 0.377 | 0.656 | 0.357 | 0.647 | 0.355 | 0.643 | 0.347 |
| 192 | 0.668 | 0.360 | 0.671 | 0.381 | 0.672 | 0.363 | 0.649 | 0.353 | 0.657 | 0.354 |  |
| 336 | 0.644 | 0.342 | 0.665 | 0.374 | 0.673 | 0.360 | 0.665 | 0.358 | 0.656 | 0.353 |  |
| 720 | 0.654 | 0.351 | 0.662 | 0.364 | 0.701 | 0.380 | 0.682 | 0.369 | 0.673 | 0.363 |  |
| ETTh | 96 | 0.999 | 0.738 | 0.860 | 0.697 | 1.139 | 0.853 | 0.842 | 0.736 | 0.873 | 0.707 |
| 192 | 1.059 | 0.759 | 0.849 | 0.700 | 1.373 | 0.932 | 0.885 | 0.748 | 0.914 | 0.741 |  |
| 336 | 1.147 | 0.820 | 1.005 | 0.745 | 1.261 | 0.924 | 0.980 | 0.770 | 0.999 | 0.793 |  |
| 720 | 1.206 | 0.847 | 0.994 | 0.758 | 1.056 | 0.819 | 1.002 | 0.798 | 1.031 | 0.818 |  |
| Average<br>(Improve) | – | 1.083 | 0.726 | 1.002 | 0.668 | 0.994 | 0.689 | 0.845<br>$\downarrow$ 15.0% | 0.637<br>$\downarrow$ 7.6% | 0.852<br>$\downarrow$ 14.3% | 0.635<br>$\downarrow$ 7.9% |

### 4.3 Time Series Forecasting

#### Setup.

Time series forecasting plays a critical role in various real-world applications. In our experiments, we employ four public datasets of this task to assess the model performance on time series forecasting, including Weather (Wu et al., ), Exchange (Lai et al., ), Traffic (Wu et al., ), and ETTh (Zhou et al., ) datasets. For each dataset, we input 96 previous time steps and forecast the subsequent time steps of {96, 192, 336, 720}. In this task, we choose the sequence models as baselines, including LSTM, Mamba, Transformer, Transformer with \modelname , and Transformer with \modelname (Gated).

#### Results.

As presented in Table 2, we compare the performance of Transformer with \modelname and other sequence models for time series forecasting tasks on four public datasets. In most cases, Transformer with \modelname and its gated version achieves the best performance on these tasks, compared to LSTM, Mamba, and the standard Transformer. The improvements of Transformer with \modelname and \modelname (Gated) over the standard Transformer are notable, with the average relative improvements ranging from 14.3% to 15.0% for MSE and from 7.6% to 7.9% for MAE. These results suggest that incorporating explicit periodic pattern encoding within neural networks improves time series forecasting performance in real-world applications.

### 4.4 Language Modeling

#### Setup.

Language modeling is a fundamental task in natural language processing. In this experiment, we conduct language modeling using the SST-2 (Socher et al., ) dataset and evaluate the model’s performance on its test set, as well as on the related datasets such as IMDB (Maas et al., ), Sentiment140 (Sahni et al., ), and Amazon Reviews (Linden et al., ). These four classic datasets all belong to the field of sentiment analysis. In this task, the comparisons are between Transformer with \modelname and \modelname (Gated), along with other sequence models, including LSTM, Mamba, and Transformer.

**Table 3: Performance of different sequence models on language modeling tasks, where the models are trained on the training set of SST-2 and evaluated on the other datasets, the bold value indicates the best performance on each column, the bold italic indicates the best performance other than Transformer with \modelname and \modelname (Gated), and the improvements represent our relative improvements of using \modelname based on Transformer.**
| Model | Num of<br>Params | SST-2 (test) | IMDB | Sentiment140 | Amazon Reviews |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Loss $\downarrow$ | Acc $\uparrow$ | Loss $\downarrow$ | Acc $\uparrow$ | Loss $\downarrow$ | Acc $\uparrow$ | Loss $\downarrow$ | Acc $\uparrow$ |  |  |
| LSTM | 120.14M | 0.4760 | 0.8060 | 0.6449 | 0.6438 | 0.8026 | 0.5979 | 0.5791 | 0.7152 |
| Mamba | 129.73M | 0.4335 | 0.7959 | 0.6863 | 0.6203 | 0.7871 | 0.5874 | 0.6163 | 0.6719 |
| Transformer | 109.48M | 0.4297 | 0.8119 | 0.5649 | 0.6994 | 0.8891 | 0.5779 | 0.5563 | 0.7155 |
| \hdashlinew/ \modelname (Gated) | 95.33M | 0.4250 | 0.8039 | 0.5817 | 0.7012 | 0.7941 | 0.6194 | 0.4835 | 0.7689 |
| w/ \modelname | 95.32M | 0.4094 | 0.8154 | 0.5225 | 0.7398 | 0.8257 | 0.6093 | 0.4748 | 0.7763 |
| Improvements | $\downarrow$ 14.16M | $\downarrow$ 4.72% | $\uparrow$ 0.43% | $\downarrow$ 7.51% | $\uparrow$ 5.78% | $\downarrow$ 7.13% | $\uparrow$ 5.43% | $\downarrow$ 14.65% | $\uparrow$ 8.50% |

#### Results.

We report the performance comparison between different sequence models across four sentiment analysis datasets, as shown in Table 3. We can find that our proposed Transformer with \modelname demonstrates significantly superior performance compared to the standard Transformer and other baselines, such as LSTM and Mamba, especially for zero-shot cross-domain performance on IMDB, Sentiment140, and Amazon Reviewers datasets. Transformer with \modelname achieves the relative improvements up to 14.65% and 8.50% in terms of Loss and Accuracy respectively, while reducing the number of parameters by about 14.16M. The result indicates the potential of periodicity modeling to enhance both effectiveness and generalization on cross-domain language modeling and sentiment analysis tasks.

## 5 Related Work

In this section, we outline the two most relevant directions and associated papers of this work.

#### Learning Periodicity with Neural Networks.

Periodic functions are one of the most basic functions of importance to human society and natural science (Newton, ; Osborn & Sensier, ; Kwasnicki, ; De Groot & Franses, ; Zhang et al., ).
However, commonly used neural networks, such as MLPs and transformers, struggle to extrapolate periodic functions beyond the scope of the training data. This limitation arises from the lack of inherent “periodicity” in their inductive biases. Some previous works (Silvescu, ; Liu, ; Parascandolo et al., ; Uteuliyeva et al., ) proposed merely using standard periodic functions themselves or their linear
combinations as activation functions, which only work well on some shallow and simple models. On this basis, work (Liu et al., ) introduced the Snake function, i.e., $x+\sin^{2}(x)$, as the activation function. However, we observed that it can fit periodic functions to a certain extent, but its effect is limited, as demonstrated in Appendix D. Therefore, although some previous studies have attempted to integrate the periodic information into neural networks, their actual performance and range of applications remain heavily constrained.

#### Fourier-based Neural Network.

Previous studies have explored Fourier-based neural networks to enhance the computational tasks (Zuo & Cai, ; Tan, ; Zuo & Cai, ; Zuo et al., ; Li et al., ; Chen et al., ). Fourier Neural Networks (Silvescu, ; Ngom & Marin, ) are shallow feedforward networks that employ cosine activation functions to map inputs to their Fourier decompositions. Work (Lee et al., ) directly utilized the Fourier Series constructed by a shallow neural network for generating periodic signals. In addition, work (Jiang et al., ) introduces Fourier Series at the end of models to embed periodic components within the network. These approaches generally possess a similar principle as Eq. (3), using a neural network to simulate the formula of Fourier Series. However, this leads to the same problem as in Eq. (5), i.e., they are hard to serve as building blocks for deep neural networks, which limits these approaches’ capabilities.

In this paper, we design \modelname to address these challenges, which performs exceptionally well on periodicity modeling and a range of real-world tasks.

## 6 Discussion

In this section, we mainly discuss the expressive power and application scope of \modelname as follows.

First, \modelname theoretically possesses the same expressive power as MLP as it also adheres to the universal approximation theorem, which ensures its capacity for functional approximation. Moreover, \modelname introduces an important enhancement by explicitly incorporating periodicity, a feature absent in traditional MLPs. Through this design, \modelname not only retains the capabilities of MLP but also enhances its ability to capture periodic characteristics in data. Therefore, \modelname can be seen as a strong alternative to MLP.

Second, beyond tasks that explicitly require periodicity modeling, \modelname also has utility in a broader range of applications. This has been evidenced by our experiments on real-world tasks, such as symbolic formula representation, time series forecasting, and language modeling, where \modelname outperforms MLP and other baselines.
In fact, many machine learning tasks may harbor hidden forms of periodicity, even without explicit requirements to include periodicity, such as mathematical operations and logic reasoning. If the neural network lacks the ability to model periodic components, it could impair its learning efficiency. From a deeper perspective, periodicity is not just a data feature but reflects a form of structural knowledge—one that allows for the transfer and reuse of abstract rules and principles across different contexts.

## 7 Conclusion

In this paper, we have proposed Fourier Analysis Network (\modelname), a novel neural network architecture for tackling the problem of periodicity modeling, which utilizes Fourier Series to facilitate capturing the underlying principles within data and reasoning.
Experimental results demonstrate that \modelname can successfully fit a variety of both basic and complex periodic functions, whereas other approaches failed. Moreover,
\modelname and its combination with Transformer also exhibit superior performance in multiple real-world tasks, including symbolic formula representation, time series forecasting, and language modeling tasks, outperforming existing neural networks such as MLP, KAN, Transformer, LSTM, and Mamba.
These promising results, especially the stronger performance and the fewer parameters and FLOPs compared to MLP, suggest its potential to become a key component of foundational models. In the future, we aim to further increase the scale of \modelname and expand its scope of application, reinforcing its role as a versatile and powerful building block in the machine learning landscape.

## 8 Acknowledgement

We would like to thank Lecheng Wang and Xuanming Zhang for their participation in discussions related to this work.

## Appendix A MLP

The MLP layer $\Phi(x)$ is defined as:

$$ $\Phi(x)=\sigma(B_{m}+W_{m}x),$ (12) $$

where $B_{m}\in\mathbb{R}^{d_{m}}$ and $W_{\bar{p}}\in\mathbb{R}^{d_{x}\times d_{m}}$ are learnable parameters with the hyperparameter $d_{m}$ indicating the first dimension of $W_{m}$, $\sigma$ denotes the activation function, and MLP can be defined as the stacking of the MLP layer $\Phi(x)$:

$$ $\text{MLP}(x)=\Phi_{L}\circ\Phi_{L-1}\circ\cdots\circ\Phi_{1}\circ x,$ (13) $$

where

$$ $\Phi_{l}(x)=\left\{\begin{array}[]{ll}\sigma(B_{m}^{l}+W^{l}_{m}x),&\text{if } l<L,\\ B^{L}+W^{L}x,&\text{if }l=L,\end{array}\right.$ (14) $$

## Appendix B Additional Experiments

**Table 4: Experiments on image recognition tasks, where Accuracy* means the best Accuracy, Accuracy means the Accuracy at the last epoch, and OOD Accuracy means the Accuracy on other paired datasets. The bold values indicate the highest value between CNN and CNN w/ FAN under the same setting, and the improve means the relative improvements of using FAN based on CNN. Fashion-MNIST-C represents the Fashion-MNIST-corrupted dataset.**
| Dataset | Accuracy* | OOD Accuracy* | Accuracy | OOD Accuracy |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CNN | w/ FAN | CNN | w/ FAN | CNN | w/ FAN | CNN | w/ FAN |  |
| MNIST | 99.63 | 99.67 | 28.85 | 30.3 | 99.55 | 99.64 | 22.12 | 21.64 |
| \hdashlineMNIST-M | 94.52 | 94.23 | 82.85 | 83.55 | 94.29 | 94.22 | 80.07 | 81.44 |
| Fashion-MNIST | 94.15 | 94.47 | 49.82 | 51.88 | 94.05 | 94.21 | 48.08 | 50.3 |
| \hdashlineFashion-MNIST-C | 88.61 | 88.82 | 91.45 | 91.59 | 88.6 | 88.59 | 91.41 | 91.47 |

**Table 5: Additional Experiments on time series forecasting tasks with Instance normalization (Instance_Normalization), where Input Length = 96, the bold values indicate the lowest value on each row, and the improve means the relative improvements of using \modelname and \modelname (Gated) based on Transformer. We find that applying instance normalization before the architecture can effectively improve the performance, therefore we report the results of this setup here.**
| Dataset | Output<br>Length | Transformer<br>(12.12 M) | Transformer with \modelname (11.06 M) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Gated | Default |  |  |  |  |  |  |
| MSE $\downarrow$ | MAE $\downarrow$ | MSE $\downarrow$ | MAE $\downarrow$ | MSE $\downarrow$ | MAE $\downarrow$ |  |  |
| Weather | 96 | 0.1772 | 0.2301 | 0.1864 | 0.2352 | 0.1756 | 0.2247 |
| 192 | 0.2438 | 0.2844 | 0.2445 | 0.2834 | 0.2327 | 0.2760 |  |
| 336 | 0.3077 | 0.3267 | 0.3156 | 0.3320 | 0.3118 | 0.3291 |  |
| 720 | 0.4253 | 0.3982 | 0.3909 | 0.3782 | 0.4113 | 0.3906 |  |
| Exchange | 96 | 0.1433 | 0.2653 | 0.1157 | 0.2452 | 0.1436 | 0.2666 |
| 192 | 0.2563 | 0.3552 | 0.2539 | 0.3611 | 0.2651 | 0.3757 |  |
| 336 | 0.5273 | 0.5218 | 0.4329 | 0.4891 | 0.5092 | 0.5326 |  |
| 720 | 1.7401 | 0.9273 | 1.5783 | 0.9303 | 1.0599 | 0.7657 |  |
| Traffic | 96 | 0.6160 | 0.3449 | 0.6030 | 0.3334 | 0.6109 | 0.3319 |
| 192 | 0.6329 | 0.3479 | 0.6239 | 0.3404 | 0.6258 | 0.3370 |  |
| 336 | 0.6369 | 0.3485 | 0.6416 | 0.3487 | 0.6200 | 0.3380 |  |
| 720 | 0.6555 | 0.3577 | 0.6645 | 0.3574 | 0.6412 | 0.3525 |  |
| ETTh1 | 96 | 0.5339 | 0.4910 | 0.5503 | 0.5216 | 0.5378 | 0.4983 |
| 192 | 0.5633 | 0.5209 | 0.5906 | 0.5346 | 0.5968 | 0.5265 |  |
| 336 | 0.7576 | 0.5813 | 0.6640 | 0.5636 | 0.7525 | 0.5933 |  |
| 720 | 0.7411 | 0.6177 | 0.7411 | 0.6066 | 0.7328 | 0.6142 |  |
| ETTh2 | 96 | 0.3881 | 0.4097 | 0.4082 | 0.4292 | 0.3833 | 0.4149 |
| 192 | 0.5766 | 0.4999 | 0.4695 | 0.4514 | 0.5039 | 0.4640 |  |
| 336 | 0.5782 | 0.5100 | 0.5556 | 0.5012 | 0.5417 | 0.4940 |  |
| 720 | 0.5841 | 0.5230 | 0.5070 | 0.4943 | 0.5272 | 0.4951 |  |
| Average<br>(Improve) | – | 0.554 | 0.444 | 0.526<br>$\downarrow$ 5.1% | 0.436<br>$\downarrow$ 1.9% | 0.509<br>$\downarrow$ 8.2% | 0.430<br>$\downarrow$ 3.2% |

Figure: Figure 6: Additional Experiments on Periodicity Modeling Tasks.
Refer to caption: x8.png

## Appendix C Experimental Details

### C.1 Setup of Periodicity Modeling

In periodicity modeling tasks, \modelname, MLP, and KAN each consist of three layers with comparable FLOPs, while the Transformer model comprises twelve layers. For consistency, we set the hidden layer dimension ($d_{h}$) to 2048 for \modelname, MLP, and Transformer. In the case of KAN, we follow its original paper (Liu et al., ), where the spline order ($K$) and the number of spline intervals ($G$) are set to 3 and 50, respectively. We apply a learning rate of $1\times 10^{-5}$ for training all models. We ensured that the data density of each period in tasks was consistent, meaning that each cycle contained a fixed quantity of 10,000 training data points.

### C.2 Setup of Symbolic Formula Representation

In symbolic formula representation tasks, we used the create_dataset function from the official KAN repository to generate the datasets. Each dataset contains 3000 training samples and 1000 test samples, with all input variables randomly sampled from the range [-1, 1]. We followed the training settings from the original KAN paper, training all methods using LBFGS for 1800 steps. For KAN, we increased the number of grid points to scale up the parameter size, covering $G=\{3,5,10,20,50,100,200,500,1000\}$. For other methods, we scaled up the parameter size by increasing the number of layers and the dimensions of hidden layers.

### C.3 Setup of Time Series Forecasting

In time series forecasting task, we implement our model based on the codebase by (Wu et al., ). Each model comprises 2 encoder layers and 1 decoder layer. We fix the hidden size for both the Transformer and our model at 512, with the feedforward dimension set to 2048 (four times the hidden size). The parameter sizes detailed in the main text correspond to the Exchange dataset; variations in the number of variables across different datasets influence the linear layers in the model. We adjust the hidden sizes of the other models to align with the Transformer parameters for fairness.

### C.4 Setup of Language Modeling

In language modeling task, we employ the BERT tokenizer (Devlin et al., ) and an embedding layer with a dimensionality of 768, except for Mamba, which adheres to its default settings as specified in the original paper (Gu & Dao, ). The architecture features 4, 24, and 12 layers with hidden sizes of 1800, 768, and 768 for LSTM, Mamba, and Transformers, respectively. To mitigate training stagnation in deeper LSTM models, we reduce the number of layers while increasing the hidden size to balance the parameters. Importantly, Mamba’s layer count is twice that of a similarly sized Transformer, as each layer consists of two Mamba blocks (Multihead attention block + MLP block).

### C.5 Setup of Image Recognition

In image recognition tasks, we employ a simple CNN generated by ChatGPT as the baseline model, which consists of four Convolutional Layers and two MLP Layers. We replace MLP with FAN in CNN, i.e. CNN with FAN, as the counterpart, ensuring that they have similar parameters. For each task, we use stochastic gradient descent with momentum (SGDM) as the optimizer, the learning rate is set to 0.01, and the training process runs for 100 epochs.

## Appendix D Comparison of \modelname and Snake Activation Function

Figure: Figure 7: Comparisons of \modelname with MLP (Snake) (Liu et al., ) in fitting periodic functions.
Refer to caption: x10.png

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 1
presentation: 2
questions: 1. Lack of detailed explanation about parameter efficiency:
The abstract claims that FAN can substitute MLP with fewer parameters. However, the theoretical justification and experimental evidence supporting this claim are not sufficiently detailed in the main text. It would be beneficial to include a comprehensive analysis comparing the parameter count and FLOPs of FAN versus MLP in various tasks.

2. Doubtful results in symbolic formula representation:
In Figure 5, the performance of KAN (Liu et al., 2024) was not replicated accurately. As KAN has shown strong results in symbolic formula representation in previous works, the omission of this accuracy raises concerns. I suggest the authors provide a more in-depth discussion of why FAN outperforms KAN in their experiments, or at least explain the differences in experimental settings that may have led to this discrepancy.

3. Practical Use Cases:
Can the practical utility of Fourier Approximation Networks (FAN) be demonstrated with real-world applications beyond synthetic and controlled experiments? How does FAN improve performance on industry-relevant tasks?

4. Clarification of Contributions:
What aspects of FAN are novel compared to existing Fourier-based neural networks? Can the authors clarify potential overlaps with prior work and better situate FAN within the context of ongoing research?
soundness: 2
strengths: The paper proposes a novel approach based on Fourier Analysis, which is a well-established mathematical tool for modeling periodic functions. The integration of Fourier Series into the network design could potentially address the limitations of traditional architectures like MLP and Transformer in periodicity modeling.

The writing is generally clear and the figures, such as Figure 1 and others, are well-presented. They help in visualizing the performance of FAN relative to other models.

The experiments cover a variety of tasks, including symbolic formula representation, time series forecasting, and language modeling, which demonstrates the general applicability of the proposed architecture.
summary: The paper addresses the critical challenge of periodicity modeling in neural networks, proposing the Fourier Analysis Network (FAN) to overcome inherent limitations of MLPs and Transformers in handling periodic data. The attempt to leverage Fourier Series for improving periodicity modeling is a novel approach, and integrating this into neural network design could potentially enhance tasks like time series forecasting and symbolic formula representation.
weaknesses: Insufficient Theoretical Justification:
The paper lacks a rigorous theoretical foundation that convincingly demonstrates the superiority of FAN over existing methods. While the integration of Fourier Series is discussed, the connection to the universal approximation theorem and how FAN mitigates spectral bias needs clearer elaboration.

Inadequate Experimental Validation:
The experiments provided are not sufficient to substantiate the claims. The datasets used for evaluation, especially in time series forecasting and symbolic formula representation, do not comprehensively cover real-world complexities. Additionally, key experimental details such as hyperparameter tuning, model initialization, and evaluation metrics are missing or inadequately described.

Limited Comparisons with Baselines:
The paper claims FAN outperforms MLPs, KANs, and Transformers, yet the comparisons lack depth. There is insufficient analysis of why FAN performs better, and the results for non-periodic tasks suggest that FAN’s benefits might be overstated.  It lacks comparisons with other neural networks that also leverage Fourier analysis. The absence of this comparison makes it difficult to fully evaluate the effectiveness of FAN, and this omission raises questions about the extent of its claimed advantages.

Unclear Practical Utility:
While the theoretical motivation is apparent, the practical implications are less clear. The paper does not provide compelling use cases or applications where FAN’s periodicity modeling significantly outperforms traditional methods in practical scenarios.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The paper lacks theoretical justification for why FAN enhances generalization capability; 
2. They don't explain how FAN avoids overfitting, such as through some model selection methods.
soundness: 3
strengths: 1. This paper cleverly integrates Fourier analysis into neural network architecture, including both periodic and aperiodic parts, which can seamlessly replace MLP layer.
2. FAN uses fewer parameters and FLOPs than traditional MLPS and is more computationally efficient.
3. This model is more interpretable than the traditional black box model, such as MLP.
summary: This paper proposes a new network architecture FAN based on Fourier analysis, which improves the ability of efficient modeling and reasoning of periodic phenomena. By introducing Fourier series, periodicity is naturally integrated into the structure and calculation process of neural network, and more accurate expression and prediction of periodicity pattern are realized. FAN can seamlessly replace MLP in a variety of models while using fewer parameters and FLOPs. Through numerous experiments, the paper demonstrates the effectiveness of FAN in modeling and reasoning periodic functions, as well as its superiority and generalization ability for a range of real world tasks including symbolic formula representation, time series prediction, and language modeling.
weaknesses: 1. FLOPs analysis is given in this paper, but the comparison of actual training time is lacking
2. Will the result of a deeper FAN yield a better fit? Will overfitting results be produced? It is recommended to test the performance changes of layers 3-20 at different depths
3. This article compares performance with traditional MLP, KAN and transformer. Whether to compare and discuss with deep learning frameworks that actually deal with time series data, such as LSTM, RNN

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1.	Clarify the distinction between your approach and directly learning the coefficients.
	•	Suggestion: Conduct an ablation study to quantify the improvement in results when incorporating the linear component.
	2.	Explain the effect of the parameter ￼ on model performance.
	•	Recommendation: Provide details on how ￼ influences the results.
soundness: 1
strengths: The paper is well-written, and I was able to quickly understand both the core concepts and the practical implementation. The numerous graphs effectively illustrate the methodology and results, enhancing the clarity and impact of the work.
summary: This paper introduces FAN (Fourier Analysis Network), a new neural network architecture designed to better model periodic patterns. Unlike traditional neural networks, which often struggle to generalize periodic data and tend to memorize it instead, FAN uses Fourier Series to directly incorporate periodicity into the model’s structure. This approach allows FAN to more effectively learn and generalize periodic functions, especially in cases where the data goes beyond the training range. The authors show that FAN can replace multi-layer perceptrons (MLPs) in many tasks, achieving similar or better results with fewer parameters and computational needs. Experiments demonstrate FAN’s strengths across tasks like time series forecasting, symbolic formula representation, and language modeling. FAN outperforms models such as MLPs, KAN, and Transformers, highlighting its effectiveness in both periodic and non-periodic contexts. Additionally, FAN achieves competitive or superior results in real-world applications, showing potential as a foundational model component.
weaknesses: 1.	When discussing the advantages of predicting Fourier coefficients, consider citing the theorem that emphasizes the significance of Fourier coefficient singularity. Additionally, referencing the foundational property that smooth functions can often be approximated well with a finite number of Fourier coefficients would strengthen the argument that predicting the first ￼ coefficients is typically sufficient.
	2.	The syntax and structure are somewhat confusing. Specifically, the distinction between ￼ and ￼ is unclear upon initial reading and could benefit from a more explicit explanation.
	3.	In demonstrating results on known functions, it is worth noting that functions with a finite number of Fourier coefficients (finite in the frequency domain but infinite in the time domain) will predictably perform better when analyzed in the frequency domain. Consequently, comparing this to models that do not use the frequency domain is not particularly informative. Instead, consider comparing with frequency-based models such as FRETS, FEDformer, and other similar approaches. Furthermore, adding examples with different types of functions would provide a more comprehensive evaluation, such as:
	•	Periodic functions that include both low and high frequencies (e.g., ￼).
	•	Non-periodic functions.
	4.	When discussing other real-world datasets, the choice of datasets could be expanded. The omission of non-periodic datasets, such as exchange rates and electricity usage, is particularly noticeable, and including them would enhance the comprehensiveness of your analysis.
	5.	In the results section, comparing your model solely with non-frequency models may be less meaningful, as the use of FFT in time series prediction is not a novel approach, and several previous models have integrated FFT with MLPs. A more relevant comparison would be to other FFT-based models to effectively demonstrate the advantage of your approach, particularly in terms of the integration between FFT coefficients and linear modeling.
	6.	If you are presenting the FLOP count, it would be beneficial to include a runtime graph. My intuition suggests that the MLP model may have a higher runtime, and a runtime graph would provide a clearer perspective on computational efficiency.
	7.	Including a link to the code within the abstract would be valuable for readers interested in exploring the implementation.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: + Why hasn't the issue with periodic functions been more widely studied? Are there prior works along these lines?

+ Why do the FAN Transformers work on the language task, given how different this is from the area where Fourier approaches are normally applied?
soundness: 3
strengths: + I think the paper is clearly written and makes a good-faith effort to benchmark, matching the design of benchmarks used in prior works like KAN. I appreciate that the authors benchmark on a wide variety of time series and language tasks. I particularly like the benchmarks on long-horizon forecasting datasets, like ETTh, which has a dominant periodic component that could potentially benefit from a model with an inductive bias for periodicity

+ The authors have included some interesting ablations and generalizations, including a Transformer architecture with FAN replacing the MLP layers, allowing them to directly process language data.

+ I very much appreciate that controlled for model size, by ensuring that all models shown in Table 2 are roughly the same size.
summary: + This paper introduces Fourier Analysis Networks (FAN), a class of learning models with an inductive bias for periodic data.

+ The authors show that FAN match the performance of KAN and MLP, two similar architectures, on a variety of symbolic formula regression tasks. 

+ The authors introduce a variety of baseline models, and they show that their method is competitive on general problems like time series forecasting and language modelling.
weaknesses: **Scope**

I read this paper as arguing that FAN represents a good general-purpose model for any task. The authors perform experiments showing that it exceeds the performance of KAN on general fitting tasks like fitting non-periodic symbolic functions or language modelling.
But, if this is really the case, why would this be true? What about the Fourier basis compared to the b-spline in KAN or sigmoids in MLP would allow it to perform better? I feel like this paper is missing either a theoretical justification, or a set of ablations or other experiments that probe the trained FAN models, that can help establish intuition for why this would work.

If the authors’ argument is instead that FAN has an inductive bias for periodic data, and so it should be best-in-class for a particular type of problem (like forecasting time series with mixed periodicity), then the experiments should be narrower and focus on establishing precedence on this problem. In this case, the model would need to be compared against simpler methods, including a simple Fourier transform forecast model (this is included in the darts library).

Either of the approaches above would make sense to me, but right now the paper seems to be arguing the need for a model with strong inductive biases for time series, and then doing experiments that seem to argue that this model is general-purpose,


**Claims**

+ I don’t completely buy that models like the Transformer can’t model purely periodic functions. Did the authors fairly tune the baselines here? Is this a known problem in the literature? This seems like it would be a widely-known and reported problem, which would preclude any prior progress on datasets like ETTh, and so I am skeptical of this claim. The papers the authors cite in support of this argument in the related work section ((Silvescu, 1999; Liu, 2013; Parascandolo et al., 2016; Uteuliyeva et al., 2020) only show periodic modifications to architectures; none make broader claims about periodic functions being inaccessible to learning models.

+ Why would this work on language data? The authors seem to be arguing that periodic functions are a more expressive general function class than the b-splines of KAN or linear -> activation structure of MLPs, since language itself has no periodic component.


**Novelty**

+ This is my primary justification for my score. I don’t necessarily think that modifying the architecture of KANs and MLP with a different transcendental function is conceptually novel, and the current text doesn’t provide any further mechanistic analysis of trained models 

+ For a higher score I would either need a motivating application, perhaps a SciML problem where the Fourier function class is needed, or I would need some theoretical or empirical insight into why FAN works better than other methods as a general approximator.

Meta-review: The paper presents a Fourier Analysis Network (FAN) to mitigate the inherent limitation of MLP and transformers of being unable to model periodicity in data. The method is novel and is intuitive considering periodic nature of fourier analysis. 

The paper suffers from the following weaknesses:
1. Practical usability of the method need to be emphasized through evaluations on relevant data sets [Reviewer 7Gdo]
2. Authors make some claims that requires strong justification such as the following:
       - FANs mitigating spectral bias - they refute their claim and then try to make sweeping justifications; 
       - missing comparison with KAN approaches 
       - Evaluation on sufficient non-periodic data sets
      - Inconsistent baselines across different data sets
      - theoretical justification for why FAN enhances generalization capability.

Decision: Reject