Title: FAN: Fourier Analysis Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Foundation model architecture
Abstract: Despite the remarkable successes of general-purpose neural networks, such as MLPs and Transformers, we find that they exhibit notable shortcomings in modeling and reasoning about periodic phenomena, achieving only marginal performance within the training domain and failing to generalize effectively to out-of-domain (OOD) scenarios. Periodicity is ubiquitous throughout nature and science. Therefore, neural networks should be equipped with the essential ability to model and handle periodicity. In this work, we propose FAN, a novel neural network that effectively addresses periodicity modeling challenges while offering broad applicability similar to MLP with fewer parameters and FLOPs. Periodicity is naturally integrated into FAN's structure and computational processes by introducing the Fourier Principle. Unlike existing Fourier-based networks, which possess particular periodicity modeling abilities but face challenges in scaling to deeper networks and are typically designed for specific tasks, our approach overcomes this challenge to enable scaling to large-scale models and maintains the capability to be applied to more types of tasks. Through extensive experiments, we demonstrate the superiority of FAN in periodicity modeling tasks and the effectiveness and generalizability of FAN across a range of real-world tasks. Moreover, we reveal that compared to existing Fourier-based networks, FAN accommodates both periodicity modeling and general-purpose modeling well.

FAN: Fourier Analysis Networks

Abstract

Despite the remarkable successes of general-purpose neural networks, such as MLPs and Transformers, we find that they exhibit notable shortcomings in modeling and reasoning about periodic phenomena, achieving only marginal performance within the training domain and failing to generalize effectively to out-of-domain (OOD) scenarios. Periodicity is ubiquitous throughout nature and science. Therefore, neural networks should be equipped with the essential ability to model and handle periodicity. In this work, we propose FAN, a novel neural network that effectively addresses periodicity modeling challenges while offering broad applicability similar to MLP with fewer parameters and FLOPs. Periodicity is naturally integrated into FAN's structure and computational processes by introducing the Fourier Principle. Unlike existing Fourier-based networks, which possess particular periodicity modeling abilities but face challenges in scaling to deeper networks and are typically designed for specific tasks, our approach overcomes this challenge to enable scaling to large-scale models and maintains the capability to be applied to more types of tasks. Through extensive experiments, we demonstrate the superiority of FAN in periodicity modeling tasks and the effectiveness and generalizability of FAN across a range of real-world tasks. Moreover, we reveal that compared to existing Fourier-based networks, FAN accommodates both periodicity modeling and general-purpose modeling well.

1 Introduction

The flourishing of modern machine learning and artificial intelligence is inextricably linked to the revolutionary advancements in the foundational architecture of general-purpose neural networks. For instance, multi-layer perceptron (MLP) [Rosenblatt, 1958, Haykin, 1998] plays a pivotal role in laying the groundwork for current deep learning models, with its expressive power guaranteed by the universal approximation theorem [Hornik et al., 1989]. Recent claims about the impressive performance of large models on various tasks are typically supported by Transformer architecture [Vaswani et al., 2017, Touvron et al., 2023, OpenAI, 2023]. In this context, the community's enthusiasm for research on neural networks has never diminished. Some emerged neural networks demonstrate notable capabilities in specific fields [Gu and Dao, 2023, Liu et al., 2024], sparking widespread discussion within the community.

Beneath the surface of apparent prosperity, we uncover a critical issue that remains in existing general-purpose neural networks: they struggle to model the periodicity from data, especially in OOD

KAN

Transformer

MLP

FAN (Ours)

𝑦 = 	sin(𝑥)

scenarios . We showcase this issue through an empirical study as illustrated in Figure 1. The results indicate that existing neural networks, including MLP [Rosenblatt, 1958], KAN [Liu et al., 2024], and Transformer [Vaswani et al., 2017], face difficulties in fitting periodic functions, even on a simple sine function. Although they demonstrate some proficiency in interpolation within the domain of training data, they tend to falter when faced with extrapolation challenges of test data. This signifies that their generalization capacity is primarily dictated by the scale and diversity of the training data, rather than by the learned principles of periodicity to perform reasoning.

Periodicity is an essential characteristic in various forms of reasoning and generalization, as it provides a basis for predictability in many natural and engineered systems by leveraging recurring patterns in observations. Besides periodic phenomena, non-periodic phenomena can also be contextualized or explained within some larger or more macro-periodic framework. Although some Fourier-based networks exhibit particular periodic modeling abilities, they are primarily tailored for specific tasks [Silvescu, 1999, Liu, 2013] and do not work well as the networks deepen [Liu et al., 2020], which limits their applicability to the general task such as language modeling [Uteuliyeva et al., 2020]. However, our goal is to exploit periodicity to benefit a broader range of tasks including language modeling. To achieve this, we aim to develop a neural network that accommodates modeling and reasoning capabilities for periodicity while maintaining the capability to be applied to more types of tasks.

In this paper, we propose Fourier Analysis Network (FAN), a novel neural network built upon the principle of Fourier Analysis. By leveraging the power of Fourier Series, we enable the neural network to model periodic patterns and extrapolate beyond them, offering the network a way to model the general principles from the data. FAN follows two core principles, the first ensures that its periodic modeling capacity scales with network depth, while the second guarantees periodic modeling is available throughout the network. These principles allow it to scale to deeper networks, a capability where existing Fourier neural networks fall short. As a result, FAN exhibits exceptional capabilities in periodicity modeling, while maintaining broad applicability to the general task, which holds great potential as a substitute for MLP, with fewer parameters and FLOPs.

To verify the effectiveness of FAN, we conduct extensive experiments from three main aspects: 1) For periodicity modeling, FAN achieves significant improvements in fitting both basic and complex periodic functions, compared to existing neural networks (including MLP, KAN, and Transformer), particularly in OOD scenarios. 2) FAN shows superior performance in various real-world tasks, such as symbolic formula representation, time series forecasting, and language modeling. Using FAN outperforms the representative models in various tasks, including MLP, KAN, LSTM, Mamba, and Transformer. 3) Compared to existing Fourier-based networks, FAN accommodates both periodicity modeling and general-purpose modeling well. The advantageous characteristics and promising results indicate that FAN has the potential to become a basic component for building fundamental large models.

2 Preliminary Knowledge

Fourier Analysis [Stein and Weiss, 1971, Duoandikoetxea, 2024] is a mathematical framework that decomposes functions into their constituent frequencies, revealing the underlying periodic structures within complex functions. At the heart of this analysis lies Fourier Series [Tolstov, 2012], which expresses a periodic function as an infinite sum of sine and cosine terms. Mathematically, for a

function f ( x ) , its Fourier Series expansion can be represented as:

where T is the period of the function, and the coefficients a n and b n are determined by integrating the function over one period:

The power of Fourier Series lies in its ability to represent a wide variety of functions, including non-periodic functions through periodic extensions, enabling the extraction of frequency components. Building on this math foundation, FAN aims to embed the periodic characteristics into network architecture, enhancing generalization capabilities and performance on various tasks, particularly in scenarios requiring the identification of patterns and regularities.

(a) MLP Layer

(b) FAN Layer

Activation Function

Activation Function

Cosine

Sine

𝑾

𝑾

Learnable Weights

𝜙 𝑥

=	[	cos(𝑊

!

𝑥)	||	sin(𝑊

!

𝑥)	||	𝜎(𝐵

!̅

+𝑊

!̅

𝑥)]

Φ 𝑥

= 	𝜎(𝐵

#

+𝑊

#

𝑥)

3 Fourier Analysis Network (FAN)

In this section, we first construct a naive neural network modeled by the formula of Fourier Series. Then, by modifying and improving it, we design FAN adhering to two core principles. Finally, we discuss the difference between the FAN layer and MLP layer.

Consider a task involving input-output pairs { x i , y i } , with the objective of identifying a function f ( x ) : R d x → R d y that approximates the relationship such that y i ≈ f ( x i ) for all x i , where d x and d y denote the dimensions of x and y , respectively. We first construct a shallow neural network f S ( x ) that represents Fourier Series expansion of the function, specifically F{ f ( x ) } , as described in Eq. (1), we can express f S ( x ) as follows:

where B ∈ R d y , W in ∈ R N × d x , and W out ∈ R d y × 2 N are learnable parameters, ( I ) follows that the computation of a n and b n computed via Eq. (2) is definite integral, ( II ) and ( III ) follows the equivalence of the matrix operations, [ ·||· ] and [ · , · ] denotes the concatenation along the first and second dimension, respectively.

To fully leverage the advantages of deep learning, we can stack the aforementioned network f S ( x ) to form a deep network f D ( x ) , where the i -th layer, denoted as l i ( x ) , retains the same structural design as f S ( x ) . Therefore, f D ( x ) can be formulated as:

where l 1 ◦ x denotes the application of the left function l 1 to the right input x , that is l 1 ( x ) . However, we discover that the direct stacking of f S ( x ) results in the primary parameters of the network f D ( x ) focusing on learning the angular frequency ( ω n = 2 πn T ), thereby neglecting the learning of the Fourier coefficients ( a n and b n ), as follows:

where l 1: L ◦ x is defined as l L -1 ◦ l L -2 ◦· · · ◦ l 1 ◦ x , W L in ( l 1: L ◦ x ) is used to approximate the angular frequencies, and W L out is used to approximate the Fourier coefficients. We can find that the capacity of f D ( x ) to fit the Fourier coefficients is independent of the depth of f D ( x ) , which is an undesirable outcome. It will limit the network's representation ability, hindering to address the complex tasks.

To this end, we design FAN based on the following principles: 1) the capacity of FAN to represent the Fourier coefficients should be positively correlated to its depth; 2) the output of any hidden layer can be employed to model periodicity using Fourier Series through the subsequent layers. The first one enhances the expressive power of FAN for periodicity modeling by leveraging its depth, while the second one ensures that the features of FAN's intermediate layers are available to perform periodicity modeling.

Suppose we decouple f S ( x ) as follows:

To satisfy both principles, the inputs of the intermediate layers in FAN necessitate to employ f in and f out simultaneously, rather than applying them sequentially.

Finally, FAN is designed on this basis, with the FAN layer ϕ ( x ) defined as below:

where W p ∈ R d x × d p , W ¯ p ∈ R d x × d ¯ p , and B ¯ p ∈ R d ¯ p are learnable parameters (with the hyperparameters d p and d ¯ p indicating the first dimension of W p and W ¯ p , respectively), the layer output ϕ ( x ) ∈ R 2 d p + d ¯ p , and σ denotes the activation function. Under this definition, the MLP layer can be where

KAN

Transformer

MLP

FAN

𝑦 = 𝑥	𝑚𝑜𝑑	5

Domain of Training Data

DomalnoflestData

Model Predictions

1.0

150

100

50

100

150

200

200

150

100

05-

oct

150

200

002-

159

oot-

150

200

200

150

100

199

150

200

KAN

Transformer

MLP

FAN

𝑦 =

𝑒

!"# $

1 + cos

%

(2𝑥)

Model Predictions

regarded as a special form of Eq. (9), when W p are learned to be zero metrics, which provides a way for FAN to maintain general-purpose modeling abilities as MLP.

The entire FAN is defined as the stacking of the FAN layer ϕ ( x ) as follows:

where

The difference between FAN and MLP. The illustrations of FAN layer ϕ ( x ) vs. MLP layer Φ( x ) are shown in Figure 2. Note that the FAN layer ϕ ( x ) computed via Eq. (9) can seamlessly replace the MLP layer Φ( x ) computed via Eq. (12) in various models with fewer parameters and FLOPs, achieved by sharing the parameters and computation of Sin and Cos parts. The number of parameters and FLOPs of the FAN layer compared to the MLP layer are presented in Table 1. The reduction ratio of parameters and FLOPs is about d p d output , which is set to 1 4 by default in this paper.

4 Experiments

In this section, we first verify the superiority of FAN in periodicity modeling tasks (Section 4.1). Second, we demonstrate the effectiveness and generalizability of FAN across a range of real-world tasks (Section 4.2). Finally, we conduct further analysis of FAN (Section 4.3), including comparisons with Fourier-based networks, running time, hyperparameter impact, and more. See Appendix B for more experiments and the experimental details can be found in Appendix C.

4.1 Periodicity Modeling

Setup. In periodic modeling tasks, we select periodic functions with practical significance and compare the models' performance in learning the underlying principles of periodicity. Specifically, we generate data from periodic functions over a large domain, using a portion of this domain as training data and the entire domain as test data, i.e., a part of test data would be out of the domain of

training data. We compare FAN and its variant FAN (Gated) § , with MLP, KAN, and Transformer. The input of this task is scalar.

Results. Figure 3, as well as Figure 6 in Appendix, show the performance of FAN and other baselines in periodicity modeling. The results indicate that existing neural networks, including MLP, KAN, and Transformers, exhibit notable deficiencies in their ability to model periodicity. Although they attempt to fit these periodic functions, their ability limits their performance in modeling a large domain of periodicity, including the test data within and outside the domain of the training data. In contrast, FAN significantly outperforms baselines in all these tasks of periodicity modeling. Moreover, FAN performs exceptionally well on the test data both within and outside the domain, indicating that our specialized design of FAN can effectively model and understand periodicity rather than merely memorize the training data.

We also compare the training process of different models on the tasks of learning complex periodic functions, as shown in Figure 4, which leads to the following findings. 1) FAN far exceeds the other baselines in both convergence speed and final effects. 2) FAN (Gated) often achieves faster convergence than FAN, but the final performance remains comparable. 3) Although the baselines show stabilization or gradual reductions in training loss as the number of epochs increases, their modeling may have diverged considerably from the distribution of the test data, resulting in a sharp increase in test loss. This phenomenon further demonstrates the shortcomings of these models in capturing periodicity.

4.2 Application of Real-world Task

1) Symbolic Formula Representation is a common task in both mathematics and physics. We follow the experiments conducted in KAN's paper [Liu et al., 2024], adhering to the same tasks, data, hyperparameters, and baselines. In addition to the original baselines, we also include Transformer for comparison in this task.

Results. Figure 7 in Appendix shows the performance of different models applied to common functions in mathematics and physics. We can observe that while KAN remains competitive with FAN when the number of parameters is small, its performance declines clearly as the number of parameters increases, which exhibits a U-shaped trend [Liu et al., 2024]. In contrast, as the number of parameters becomes large, FAN consistently outperforms the other baselines, including MLP, KAN, and Transformer, in fitting these functions, despite many of these functions being only partially periodic or even implicitly periodic. This may be attributed to FAN's ability to capture and model both

periodic and non-periodic features and the advantages of fewer parameters. These results indicate that although FAN enhances its ability to model periodicity, it does not compromise its capacity to fit non-periodic functions.

2) Time Series Forecasting plays a critical role in various real-world applications. We employ four public datasets of this task to assess the model performance on time series forecasting, including Weather [Wu et al., 2021], Exchange [Lai et al., 2018], Traffic [Wu et al., 2021], and ETTh [Zhou et al., 2021] datasets. For each dataset, we input 96 previous time steps and forecast the subsequent time steps of { 96, 192, 336, 720 } . In this task, we choose the sequence models as baselines, including LSTM, Mamba, and Transformer.

Results. As shown in Table 2 (See Table 6 in Appendix for complete results), we compare the performance of Transformer with FAN and other baselines for time series forecasting tasks. The results indicate that Transformer with FAN outperforms other representative sequence models in these tasks. The improvements of Transformer with FAN and FAN (Gated) over the standard Transformer are notable, with the average relative improve-

Table 2: Average performance on different public datasets and output lengths in time series forecasting tasks, where Input Length = 96 and the bold value indicates the best performance.

ments ranging from 15.0% to 15.6% for MSE and from 7.6% to 8.4% for MAE. It suggests that incorporating explicit periodic pattern encoding within neural networks improves time series forecasting performance in real-world applications.

3) Language Modeling is a fundamental task in natural language processing. We conduct language modeling using the SST-2 [Socher et al., 2013] dataset and evaluate the model's performance on its test set, as well as on the related datasets such as IMDB [Maas et al., 2011], Sentiment140 [Sahni et al., 2017], and Amazon Reviews [Linden et al., 2003]. These four classic datasets all belong to the field of sentiment analysis. The comparisons are between Transformer with FAN and FAN (Gated), along with the classic sequence models, including LSTM, Mamba, and Transformer.

Results. We report the performance comparison between different sequence models across four sentiment analysis datasets, as shown in Table 3. The results indicate that Transformer with FAN achieves clear improvements compared to the standard Transformer and other baselines, such as LSTM and Mamba, especially for zero-shot OOD performance on IMDB, Sentiment140, and Amazon Reviewers datasets. Using FAN achieves the relative improvements up to 14.65% and 8.50% in terms of Loss and Accuracy respectively, while reducing parameter numbers by about 14.16M. It indicates the potential of periodicity modeling to enhance both effectiveness and generalization on cross-domain language modeling and sentiment analysis tasks.

4.3 Further Analysis of FAN

Comparison with Fourier-based Networks. We compare FAN with Fourier-based networks in terms of their periodicity modeling abilities and general-purpose capabilities for language modeling. Some previous works have explored the application of Fourier-based Networks in specific tasks [Oreshkin et al., 2020, Tancik et al., 2020, Sitzmann et al., 2020, Han et al., 2022], but these studies primarily involved shallow/small-scale models (i.e., fewer than 1M parameters). Assessing their general modeling capabilities requires evaluating their effectiveness in deeper/larger architectures, we categorize these Fourier-based networks into three main types and systematically evaluate them within the 12-layer Transformer. Specifically, we compare with: 1) Fourier Neural Network (FNN) [Silvescu, 1999] using the cosine or sine function or their linear combinations as the activation function, such as SIREN [Sitzmann et al., 2020]. 2) Fourier Series Neural Network (FSNN) is defined as Eq. (3), which shares the parameters and computation of sine and cosine part. 3) Fourier Transform Neural Network (FTNN) is a type of neural network that employs Fourier Transform to process the intermediate output in the neural network, such as FNO [Li et al., 2021].

Table 4: Comparison FAN with Fourier-based Networks on language modeling tasks, where each of them replaces the MLP layer in the standard transformer and ID means in-domain.

As shown in Figure 5, only FAN achieves excellent performance on both tasks, indicating the superiority of our specially designed architecture of FAN. In contrast, FNN and FSNN cannot fit language modeling tasks, which aligns with previous work [Uteuliyeva et al., 2020, Liu et al., 2020] and our findings derived from Eq. (3)-(5). Moreover, FTNN performs poorly on complex periodic modeling tasks, akin to MLP. This may be attributed to the fact that FTNN does not incorporate the Fourier principle into the network but applies Fourier Transform as an intermediate processing step, which disadvantages FTNN in capturing periodicity. From Table 4, FAN also achieves fewer parameters and better performance than FTNN in language modeling tasks.

Runtime of FAN. We analyze the actual running time of FAN layer compared to MLP Layer with different input and output dimensions, as shown in Table 5. The experimental results show that MLPs exhibit smaller runtimes when the input and output sizes are small, due to PyTorch's optimization of MLP. However, as the input and output sizes continue to increase, matrix computations become the main contributor to runtime. At this point, FAN's fewer parameters and reduced FLOPs begin to show significant advantages. Note that FAN can be further optimized from the underlying implementation.

The impact of hyperparameter d p. In our experiments, we fix d p = 1 4 d h intuitively for FAN, where d h denotes the dimension of hidden layers. As shown in Figure 8 of Appendix, we investigate the impact of varying d p empirically on task performance by changing itself. The results indicate that performance initially improves as d p increases, but then decreases beyond a certain point. This trend may be attributed to the number of potential periodic features specific to each task. Furthermore, there remains room for further improvements with the better setup of d p.

5 Related Work

In this section, we outline the two most relevant directions and associated papers of this work.

Learning Periodicity with Neural Networks. Periodic functions are one of the most basic functions of importance to human society and natural science [Newton, 1687, Osborn and Sensier, 2002, Kwasnicki, 2008, De Groot and Franses, 2012, Zhang et al., 2017]. However, commonly used neural networks, such as MLPs and transformers, struggle with modeling periodicity. This limitation is attributed to the lack of inherent 'periodicity' in their inductive biases. Some previous works [Silvescu, 1999, Liu, 2013, Parascandolo et al., 2016, Uteuliyeva et al., 2020] proposed merely using standard periodic functions themselves or their linear combinations as activation functions, which only work well on some shallow and simple models. On this basis, work [Liu et al., 2020] introduced the Snake function, i.e., x +sin 2 ( x ) , as the activation function. However, it can fit periodic functions to a certain extent, but its effect is limited especially for OOD scenarios, as demonstrated in Appendix D. Therefore, although some previous studies have attempted to integrate periodic information into neural networks, their actual performance and range of applications remain heavily constrained.

Fourier-based Neural Network. Previous studies have explored Fourier-based networks, but these networks generally perform well on specific tasks, while their performance on more general tasks tends to be poorer [Zuo and Cai, 2005, Tan, 2006, Uteuliyeva et al., 2020, Jiang et al., 2022, Chen et al., 2022]. Fourier Neural Networks employ the cosine [Silvescu, 1999, Ngom and Marin, 2021] or sine function [Parascandolo et al., 2016, Sitzmann et al., 2020] or their combination [Liu, 2013] as the activation function. Some work employs Fourier Transform to process the intermediate output of network [Li et al., 2021, Lee-Thorp et al., 2022], but they did not address the challenges of periodicity modeling. Some researches focus on leveraging the network to simulate the formula of Fourier Series [Rafajłowicz and Pawlak, 1997, Halawa, 2008, Lee et al., 2021], which generally possesses a similar principle as Eq. (3). However, this leads to the same problem as in Eq. (5), i.e., they are hard to serve as building blocks for deep neural networks, which limits these approaches' capabilities. More detailed discussion can be found in Appendix G.

In this paper, we design FAN to address these challenges, which performs exceptionally well on periodicity modeling and maintains broad applicability on real-world tasks.

6 Discussion

In this section, we have a broad discussion on expressive power, extrapolation capability, and application scope of FAN as follows: ❶ FAN theoretically possesses the equal expressive power as MLP since it also adheres to Universal Approximation Theorem, which guarantees its capacity for functional approximation (refer to Appendix E for the detailed explanation). Moreover, FAN introduces an important enhancement by incorporating periodicity, a feature absent in MLPs. By leveraging this special design, FAN not only retains the capabilities of MLP but also enhances its ability to capture periodic characteristics in data. ❷ We observe that existing networks often exhibit divergent predictions in OOD scenarios, as shown in Figures 3, 4, and 6 for periodicity modeling tasks. In contrast, FAN demonstrates strong OOD extrapolation ability in both periodicity modeling and some real-world tasks. This extrapolation ability indicates that the network is no longer restricted to the paradigms present in training dataset, but instead exhibits a kind of 'transboundary thinking'. This could be an important avenue for improving generalization and learning efficiency. ❸ Beyond tasks that explicitly require periodicity modeling, FAN also has utility in a broader range of applications, which has been evidenced by our extensive experiments on real-world tasks, such as symbolic formula representation, time series forecasting, language modeling, and image recognition, where FAN achieve competitive or superior performance than Transformers and other baselines. In fact, many machine learning tasks may harbor hidden forms of periodicity, even without explicitly including periodicity, such as mathematical operations and logic reasoning. If the neural network lacks the ability to model periodicity, it could impair the learning efficiency [Dong et al., 2025b]. From a deeper perspective, periodicity is not just a data feature but reflects a form of structural knowledge one that allows for the transfer and reuse of abstract rules and principles across different contexts.

7 Conclusion and Future Work

In this paper, we have proposed Fourier Analysis Network (FAN), a novel network that addresses periodicity modeling in existing networks while maintaining the general-purpose modeling capability. Experimental results demonstrate that FAN successfully fit both basic and complex periodic functions, whereas other general-purpose networks failed. Moreover, using FAN exhibit clear improvements in real-world tasks, such as symbolic formula representation, time series forecasting, and language modeling, outperforming neural networks such as MLP, KAN, LSTM, Mamba, and Transformer. These promising results, especially the stronger performance and the fewer parameters and FLOPs compared to MLP, suggest its potential to become a key component of foundational models. Some works have demonstrated the superiority of using FAN in diverse tasks, including gravitational wave analysis [Zhao et al., 2024], EEG-based emotion recognition [Wang et al., 2025], and large language modeling [Dong et al., 2025b], etc. In future work, we aim to further broaden the applicability of FAN.

References

Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review , 65(6):386, 1958.

Simon Haykin. Neural networks: a comprehensive foundation . Prentice Hall PTR, 1998.

Kurt Hornik, Maxwell B. Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks , 2(5):359-366, 1989.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , pages 5998-6008, 2017.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aur´ elien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR , abs/2307.09288, 2023.

OpenAI. GPT-4 technical report. CoRR , abs/2303.08774, 2023.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR , abs/2312.00752, 2023.

Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. KAN: kolmogorov-arnold networks. CoRR , abs/2404.19756, 2024.

Adrian Silvescu. Fourier neural networks. In IJCNN , pages 488-491. IEEE, 1999.

Shuang Liu. Fourier neural network for machine learning. In ICMLC , pages 285-290. IEEE, 2013.

Ziyin Liu, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how to fix it. In NeurIPS , 2020.

Malika Uteuliyeva, Abylay Zhumekenov, Rustem Takhanov, Zhenisbek Assylbekov, Alejandro J. Castro, and Olzhas Kabdolov. Fourier neural networks: A comparative study. Intell. Data Anal. , 24(5):1107-1120, 2020.

Elias M Stein and Guido Weiss. Introduction to Fourier analysis on Euclidean spaces , volume 1. Princeton university press, 1971.

Javier Duoandikoetxea. Fourier analysis , volume 29. American Mathematical Society, 2024.

Georgi P Tolstov. Fourier series . Courier Corporation, 2012.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems , 34:22419-22430, 2021.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In SIGIR , pages 95-104. ACM, 2018.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI , pages 11106-11115. AAAI Press, 2021.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP , pages 1631-1642. ACL, 2013.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL , pages 142-150. The Association for Computer Linguistics, 2011.

Tapan Sahni, Chinmay Chandak, Naveen Reddy Chedeti, and Manish Singh. Efficient twitter sentiment classification using subjective distant supervision. In COMSNETS , pages 548-553. IEEE, 2017.

Greg Linden, Brent Smith, and Jeremy York. Amazon.com recommendations: Item-to-item collaborative filtering. IEEE Internet Comput. , 7(1):76-80, 2003.

Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: neural basis expansion analysis for interpretable time series forecasting. In ICLR . OpenReview.net, 2020.

Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS , 2020.

Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NeurIPS , 2020.

Bing Han, Cheng Wang, and Kaushik Roy. Oscillatory fourier neural network: A compact and efficient architecture for sequential processing. In AAAI , pages 6838-6846. AAAI Press, 2022.

Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In ICLR . OpenReview.net, 2021.

Isaac Newton. Philosophiae naturalis principia mathematica . William Dawson & Sons Ltd., London, 1687.

Denise R. Osborn and Marianne Sensier. The prediction of business cycle phases: Financial variables and international linkages. National Institute Economic Review , 182(1):96-105, 2002. doi: 10.1177/002795010218200110. URL https://doi.org/10.1177/002795010218200110 .

Witold Kwasnicki. Kitchin, juglar and kuznetz business cycles revisited. Wroclaw: Institute of Economic Sciences , 2008.

Bert De Groot and Philip Hans Franses. Common socio-economic cycle periods. Technological Forecasting and Social Change , 79(1):59-68, 2012.

Liheng Zhang, Charu Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multifrequency trading patterns. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '17, page 2141-2149, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874. doi: 10.1145/3097983.3098117. URL https://doi.org/10.1145/3097983.3098117 .

Giambattista Parascandolo, Heikki Huttunen, and Tuomas Virtanen. Taming the waves: sine as activation function in deep neural networks. 2016.

Wei Zuo and Lilong Cai. Tracking control of nonlinear systems using fourier neural network. In Proceedings, 2005 IEEE/ASME International Conference on Advanced Intelligent Mechatronics. , pages 670-675. IEEE, 2005.

HS Tan. Fourier neural networks and generalized single hidden layer networks in aircraft engine fault diagnostics. 2006.

Song Jiang, Tahin Syed, Xuan Zhu, Joshua Levy, Boris Aronchik, and Yizhou Sun. Bridging selfattention and time series decomposition for periodic forecasting. In CIKM , pages 3202-3211. ACM, 2022.

Hanlong Chen, Luzhe Huang, Tairan Liu, and Aydogan Ozcan. Fourier imager network (FIN): A deep neural network for hologram reconstruction with superior external generalization. Light: Science & Applications , 2022.

Marieme Ngom and Oana Marin. Fourier neural networks as function approximators and differential equation solvers. Stat. Anal. Data Min. , 14(6):647-661, 2021.

James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Onta˜ n´ on. Fnet: Mixing tokens with fourier transforms. In NAACL-HLT , pages 4296-4313. Association for Computational Linguistics, 2022.

E Rafajłowicz and M Pawlak. On function recovery by neural networks based on orthogonal expansions. Nonlinear Analysis: Theory, Methods & Applications , 30(3):1343-1354, 1997.

Krzysztof Halawa. Fast and robust way of learning the fourier series neural networks on the basis of multidimensional discrete fourier transform. In ICAISC , volume 5097 of Lecture Notes in Computer Science , pages 62-70. Springer, 2008.

Jiyoung Lee, Wonjae Kim, Daehoon Gwak, and Edward Choi. Conditional generation of periodic signals with fourier-based decoder. CoRR , abs/2110.12365, 2021.

Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In ACL (Findings) , pages 12039-12050. Association for Computational Linguistics, 2024.

Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei Ma, Jue Chen, Binhua Li, Zhi Jin, Fei Huang, Yongbin Li, and Ge Li. RL-PLUS: countering capability boundary collapse of llms in reinforcement learning with hybrid-policy optimization. CoRR , abs/2508.00222, 2025a.

Yihong Dong, Ge Li, Xue Jiang, Yongding Tao, Kechi Zhang, Hao Zhu, Huanyu Liu, Jiazheng Ding, Jia Li, Jinliang Deng, and Hong Mei. Fanformer: Improving large language models through effective periodicity modeling. CoRR , abs/2502.21309, 2025b.

Tianyu Zhao, Yue Zhou, Ruijun Shi, Peng Xu, Zhoujian Cao, and Zhixiang Ren. Compact binary coalescence gravitational wave signals counting and separation using unmixformer, 2024. URL https://arxiv.org/abs/2412.18259 .

Jinfeng Wang, Yanhao Huang, Sifan Song, Boqian Wang, Jionglong Su, and Jiaman Ding. A novel fourier adjacency transformer for advanced eeg emotion recognition, 2025. URL https: //arxiv.org/abs/2503.13465 .

Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2, 2010.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarial training of neural networks. J. Mach. Learn. Res. , 17:59:1-59:35, 2016.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR , abs/1708.07747, 2017. URL http://arxiv.org/abs/ 1708.07747 .

Michael Weiss and Paolo Tonella. Simple techniques work surprisingly well for neural network test prioritization and active learning. In Proceedings of the 31th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2022.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML , volume 162 of Proceedings of Machine Learning Research , pages 27268-27286. PMLR, 2022.

Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient for fast stylization. CoRR , abs/1607.08022, 2016.

Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory. Neural Comput. , 9(8):17351780, 1997.

Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE , 86(11):2278-2324, 1998.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster) . OpenReview.net, 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR , abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805 .

Li Wan, Matthew D. Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. Regularization of neural networks using dropconnect. In ICML (3) , volume 28 of JMLR Workshop and Conference Proceedings , pages 1058-1066. JMLR.org, 2013.

Peter Belcak and Roger Wattenhofer. Periodic extrapolative generalisation in neural networks. In SSCI , pages 1066-1073. IEEE, 2022.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately present the paper's contributions and scope, with claims that are fully supported by the results and discussion.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We demonstrate the superiority of using FAN in some mainstream tasks, and we aim to further broaden the applicability of FAN in our future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: he theoretical analysis of this paper is as follows: 1) In Section 3, we theoretically analyze that existing FSNNs and FNNs have problems in deep expansion while FAN addresses it, and conduct experiments to prove it in Section 4.3. 2) In Table 1, the theoretical analysis shows that FAN has fewer parameters and FLOPs than MLP, and we conduct experiments to prove it in Tables 2, 3, and 5. 3) In Appendix E, we theoretically analyze that FAN complies with Universal Approximation Theorem.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the experimental details in Appendix C, and make reproducible source code available at https://anonymous.4open.science/r/FAN-D43C .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide source code with comprehensive instructions and data acquisition methods at ( https://anonymous.4open.science/r/FAN-D43C ).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the experimental details in Appendix C and open source code ( https://anonymous.4open.science/r/FAN-D43C )

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We follow previous work to report the average performance of 5 runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides the information on computer resources in Appendix C.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics as outlined at https://neurips.cc/public/ EthicsGuidelines .

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is foundational research without direct societal impacts, as it focuses on fundamental algorithmic improvements rather than specific applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper focuses only on model design and algorithms without releasing any high-risk models or datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly credit all existing assets used in our research. We cite the original papers for all datasets and code packages utilized in our experiments, including specific versions and URLs where applicable. All datasets are used in accordance with their respective licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide comprehensive documentation for our new code assets, including detailed implementation instructions, usage guides, and experimental configurations in our anonymized repository.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: Foundation Model Architecture

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A MLP

The MLP layer Φ( x ) is defined as:

where

NNNNNN

MLP

Transformer

KAN

FAN

KAN

Transformer

MLP

FAN

𝑦 = sin	 (𝑥	 + 	sin	 (2𝑥))

Domain of Training Data

Model Predictions

𝑦 = sin 𝑡

cos

!

(2𝑡) + cos 𝑡

!

(3𝑡)

Domain of Training Data

Model Predictions

MLP

Transformer

KAN

FAN

B Additional Experiments

B.1 Additional Experiments on Periodicity Modeling Tasks.

More experimental results on periodicity modeling tasks are shown in Figure 6.

where B m ∈ R d m and W ¯ p ∈ R d x × d m are learnable parameters with the hyperparameter d m indicating the first dimension of W m , σ denotes the activation function, and MLP can be defined as the stacking of the MLP layer Φ( x ) :

𝑦 = (1	 +	sin(𝑥)) sin(2𝑥)

Domain of Training Data

Model Predictions

NNNNNNNNN

f(x) = b(20x)

f(x,y) = exp(sin(nx) + y²)

f(x,x,xy, x4) = exp(sin(x² +x[) + sin(x² + x[1)

MLP (depth 4)

FAN (depth).3),

Transformer (depth 3)

Transformer (depth 4)

KAN

MLP (depth 3)

FAN (depth 4)

10°

3SIe

10-1

2SHe

20-1

15ag

10-2

101

19-1

10-2

10-3.

103

102

Number of parameters

103

10

Ot

103

102

Number of parameters

103

10

101

192

Number of parameters

103

10

10

103

102

Number of parameters

103

104

10

B.2 Additional Experiments on Image Recognition Tasks.

Image Recognition is a key computer vision task where image content is identified and categorized. Our evaluation contains four public benchmarks of image recognition: MNIST [LeCun et al., 2010], MNIST-M [Ganin et al., 2016], Fashion-MNIST [Xiao et al., 2017], and Fashion-MNIST-C [Weiss and Tonella, 2022], where MNIST-M and Fashion-MNIST-C are the variants for robustness.

B.3 Evaluation on LLMs with FAN

Table 8 reports the zero-shot results on the LM Eval Harness benchmark. The results show that using FAN outperforms standard Transformer architecture across various tasks with the same training tokens of 200B.

B.4 FAN for Solving SciML Problems

We conduct experiments on the SciML problem that includes the Fourier function class following the work [Li et al., 2021]. The Burgers' equation, a non-linear partial differential equation, is frequently used in scientific computing to model shock waves and traffic flow, among other phenomena. The detailed error rate on Burgers' equation is listed in the Table 9. We can find that replacing the MLP Layer with FAN Layer in Fourier Neural Operator (FNO) [Li et al., 2021] can achieve clear improvements on each setting of resolution s of this task.

B.5 Comparison with Frequency-based Models in Time Series Forecasting Tasks

To compare with frequency-based models in Time Series Forecasting tasks such as FEDformer [Zhou et al., 2022], we replace MLP with FAN in frequency-based models. We present the experimental results in Table 10, where the results of FEDformer are cited from its paper directly. From the results, we can find that FEDformer with FAN can outperform FEDformer in almost all cases.

B.6 Comparison with Directly Learning the Coefficients

We compare FAN with a baseline of directly learning the coefficients, which inputs sin ( x ) and cos ( x ) and then uses the MLP Layer instead of the FAN Layer to model the Fourier coefficients. In this setting, frequencies are fixed and only the coefficients are learned, which may limit the model's ability to capture patterns not aligned with these frequencies. Taking simple f ( x ) = x mod 5 as an example, this setting may not even converge at all, because the frequency of x mod 5 is inconsistent with sin ( x ) and cos ( x ) . The experimental results of their loss are shown in Table 11.

B.7 The influence of hyperparameters d p

We evaluate the influence of hyperparameters d p as shown in Figure 8.

0.05

0.10

0.15

0.20

0.25

0.30

dp

/

doutput

0.42

0.44

0.46

0.48

0.50

0.52

Value

SST-2 (Loss)

Transformer with FAN

Transformer

0.05

0.10

0.15

0.20

0.25

0.30

dp

/

doutput

0.50

0.51

0.52

0.53

0.54

0.55

0.56

0.57

0.58

Value

IMDB (Loss)

Transformer with FAN

Transformer

0.05

0.10

0.15

0.20

0.25

0.30

dp

/

doutput

0.80

0.85

0.90

0.95

Value

Sentiment140 (Loss)

Transformer with FAN

Transformer

0.05

0.10

0.15

0.20

0.25

0.30

dp

/

doutput

0.44

0.45

0.46

0.47

0.48

0.49

0.50

0.51

Value

Amazon Reviews (Loss)

Transformer with FAN

Transformer

0.05

0.10

0.15

0.20

0.25

0.30

dp

/

doutput

0.804

0.806

0.808

0.810

0.812

0.814

Value

Transformer with FAN

Transformer

0.05

0.10

0.15

0.20

0.25

0.30

dp

/

doutput

0.7325

0.7350

0.7375

0.7400

0.7425

0.7450

0.7475

0.7500

Value

IMDB  Test Set (Accuracy)

Transformer with FAN

Transformer

0.05

0.10

0.15

0.20

0.25

0.30

dp

/

doutput

0.610

0.612

0.614

0.616

0.618

0.620

0.622

0.624

Value

Sentiment140 (Accuracy)

Transformer with FAN

Transformer

0.05

0.10

0.15

0.20

0.25

0.30

dp

/

doutput

0.775

0.780

0.785

0.790

Value

Transformer with FAN

Transformer

B.8 The effectiveness of the FAN Layer for deep neural networks

We evaluate the effect of varying the number of FAN layers from 3 to 20 on periodicity modeling tasks, employing residual connections to mitigate overfitting. The experimental results show that both the best training loss and test loss still decrease slowly as the number of layers increases.

Furthermore, on Language Modeling tasks, we replaced 24 MLP Layers of Transformer with 24 FAN Layers, i.e. Transformer with FAN, and it also achieved clear improvements on each task, especially for OOD zero-shot evaluation scenarios. These findings indicate that FAN Layer is effective for deep neural networks.

B.9 Experiments on Time Series Forecasting with Instance Normalization

We conduct experiments on time series forecasting tasks with instance normalization [Ulyanov et al., 2016], and the results are shown in Table 12. We find that applying instance normalization before the architecture can effectively improve the performance.

3

6

12

18

0.6

0.8

1.0

1.2

1.4

Training Loss

FAN

MLP (3 Layer)

3

6

12

18

0.8

1.0

1.2

1.4

1.6

Test Loss

FAN

MLP (3 Layer)

B.10 Layer-wise Spectral Analysis

We conduct experiments on layer-wise spectral analysis below. We perform a Fast Fourier Transform (FFT) on each layer's outputs and calculate four key metrics to quantify the spectral characteristics:

Spectral Centroid: Measures the 'center of mass' of the spectrum, indicating whether the layer's features are concentrated in low or high-frequency regions.

Spectral Sparsity (L1/L2 Norm): Quantifies how concentrated the spectral energy is within a few frequency bins. A higher value implies a more structured and less noisy signal.

Spectral Entropy: Measures the uniformity and predictability of the spectrum. A lower entropy indicates a more ordered and well-defined spectral structure.

Dominant Energy Ratio (Top-5): The proportion of total spectral energy contained within the top 5 most dominant frequency components, indicating how focused the representation is on key periodic features.

The results reveal a highly effective multi-stage learning process, which is more sophisticated than a simple monotonic evolution of frequencies. We observe a clear three-stage 'DeconstructionExploration-Reconstruction' mechanism:

Initial Approximation (Layer 1): The first layer rapidly forms an initial, highly-focused approximation of the signal, as shown by its very high Dominant Energy Ratio (96.1%).

Feature Deconstruction and Exploration (Layers 2-8): To model the function's complex, non-sinusoidal components (especially the x (mod 3) term, which requires a wide range of Fourier series terms), the intermediate layers must first 'deconstruct' the signal. This is evidenced by a sharp increase in Spectral Entropy and a decrease in the Dominant Energy Ratio. The network actively disperses energy across a broader spectrum to explore and capture these challenging features, showcasing the flexibility afforded by its depth.

Integration and Reconstruction (Layers 9-11): In the final layers, the model's task shifts from exploration to integration. It 'reconstructs' a final, efficient representation from the features learned in the middle layers. This is marked by a dramatic decrease in both Spectral Entropy and Spectral Centroid, alongside a sharp increase in the Dominant Energy Ratio to a final value of 93.8%. The network converges to a 'clean', low-frequency, and highly structured representation that is optimal for the final linear layer to map to the target output.

B.11 Ablation Study

We conduct ablation studies on just cosine function, having FAN layers only in part of the network, and freezing W p . The results show that FAN demonstrates a clear advantage over the variants in Periodicity Modeling and Language Modeling tasks.

C Experimental Details

Baselines. In our experiments, we mainly compare FAN with the following baselines: 1) MLP [Rosenblatt, 1958], 2) Transformer [Vaswani et al., 2017], 3) KAN [Liu et al., 2024], 4) LSTM [Hochreiter and Schmidhuber, 1997], 5) Mamba [Gu and Dao, 2023], 6) CNN [LeCun et al., 1998]. Details of the baselines are given in Appendix F. Moreover, we also include the following variants of FAN into our comparisons: I) FAN (Gated) : a variant of FAN that adds gates to control the tendency of the layer, with the formula defined as ϕ g ( x ) = [ g · cos( W p x ) || g · sin( W p x ) || (1 -g ) · σ ( B ¯ p + W ¯ p x )] , where g is a learnable parameter. II) Transformer with FAN and Transformer with FAN (Gated) : we replace each MLP layer in Transformer with the FAN layer computed via Eq. (9) and the layer of FAN (Gated), respectively. III) CNN with FAN : similarly, we replace each MLP layer in CNN with the FAN layer.

C.1 Implementation Details.

We conduct our experiments on a single GPU of Tesla A100-PCIe-40G. Unless otherwise specified, we use the following hyperparameters in the experiments. The model architecture consists of 3 to 24 layers, the activation function σ is set to GELU [Hendrycks and Gimpel, 2016], and the dimension of the projection matrix W p is set to d p = 1 4 d h , where d h denotes the dimension of the hidden layers. We employ the AdamW optimizer [Loshchilov and Hutter, 2019] for the model's training process.

C.2 Setup of Periodicity Modeling

In periodicity modeling tasks, FAN, MLP, and KAN each consist of three layers with comparable FLOPs, while the Transformer model comprises twelve layers. For consistency, we set the hidden layer dimension ( d h ) to 2048 for FAN, MLP, and Transformer. In the case of KAN, we follow its original paper [Liu et al., 2024], where the spline order ( K ) and the number of spline intervals ( G ) are set to 3 and 50, respectively. We apply a learning rate of 1 × 10 -5 for training all models. We ensured that the data density of each period in tasks was consistent, meaning that each cycle contained a fixed quantity of 10,000 training data points.

C.3 Setup of Symbolic Formula Representation

In symbolic formula representation tasks, we used the create dataset function from the official KAN repository to generate the datasets. Each dataset contains 3000 training samples and 1000 test samples, with all input variables randomly sampled from the range [-1, 1]. We followed the training settings from the original KAN paper, training all methods using LBFGS and Adam for 1800 steps, and selecting the best-performing result from the two optimization approaches. For KAN, we increased the number of grid points to scale up the parameter size, covering G = { 3 , 5 , 10 , 20 , 50 , 100 , 200 , 500 , 1000 } . For other methods, we scaled up the parameter size by increasing the number of layers and the dimensions of hidden layers.

C.4 Setup of Time Series Forecasting

In time series forecasting task, we implement our model based on the codebase by [Wu et al., 2021]. Each model comprises 2 encoder layers and 1 decoder layer. We fix the hidden size for both the Transformer and our model at 512, with the feedforward dimension set to 2048 (four times the hidden size). The parameter sizes detailed in the main text correspond to the Exchange dataset; variations in

the number of variables across different datasets influence the linear layers in the model. We adjust the hidden sizes of the other models to align with the Transformer parameters for fairness.

C.5 Setup of Language Modeling

In language modeling task, we employ the BERT tokenizer [Devlin et al., 2018] and an embedding layer with a dimensionality of 768, except for Mamba, which adheres to its default settings as specified in the original paper [Gu and Dao, 2023]. The architecture features 4, 24, and 12 layers with hidden sizes of 1800, 768, and 768 for LSTM, Mamba, and Transformers, respectively. To mitigate training stagnation in deeper LSTM models, we reduce the number of layers while increasing the hidden size to balance the parameters. Importantly, Mamba's layer count is twice that of a similarly sized Transformer, as each layer consists of two Mamba blocks (Multihead attention block + MLP block).

C.6 Setup of Image Recognition

In image recognition tasks, we employ the CNN as the baseline model, which consists of four Convolutional Layers and two MLP Layers (It achieves a 0.37% error rate on MNIST without augmentation, outperforming the SOTA CNN's 0.63% [Wan et al., 2013]). We replace MLP with FAN in CNN, i.e., CNN with FAN, as the counterpart, ensuring that they have similar parameters. For each task, we use stochastic gradient descent with momentum (SGDM) as the optimizer, the learning rate is set to 0.01, and the training process runs for 100 epochs.

Snake

FAN (Ours)

𝑦 = 	sin(𝑥)

D Comparison of FAN and Snake Activation Function

We compare FAN with Snake, a previous approach used for improving the fitting of periodic functions with neural networks. The results are shown in Figure 10.

E Compliance with the Universal Approximation Theorem

The Universal Approximation Theorem asserts that a feed-forward network with a single hidden layer, containing a sufficiently large and finite number of neurons, can approximate any continuous function defined on compact subsets of R n , provided that the activation function is non-constant, continuous, and nonlinear. In the case of the Fourier Analysis Network (FAN) layer, we define the mapping as:

where || denotes concatenation, and σ ( · ) represents a standard nonlinear activation function, such as ReLU or GELU. The components cos( W px ) and sin( W px ) are non-constant, continuous, and nonlinear functions, satisfying the requisite conditions for an activation function in the Universal Approximation Theorem. Therefore, the FAN layer conforms to the Universal Approximation Theorem, enabling it to approximate arbitrary continuous functions on compact subsets of R n .

This proof demonstrates that the FAN layer, through its periodic components (sine and cosine functions) and the nonlinear activation σ ( · ) , satisfies the key conditions of the Universal Approximation Theorem, ensuring its capability to approximate complex functional mappings.

F More Details of Baselines

In our experiments, we mainly compare FAN with the following baselines. 1) MLP [Rosenblatt, 1958]: the most classic model, which is widely used in the backbone of various models. 2) Transformer [Vaswani et al., 2017]: a prevalent model known for its self-attention mechanism, which achieves outstanding performance on various tasks. 3) KAN [Liu et al., 2024]: an emerged model specialized for symbolic formula representation, which uses the b-spline functions instead of fixed activation functions. 4) LSTM [Hochreiter and Schmidhuber, 1997]: a well-known recurrent neural network (RNN) that can capture long-term dependencies on sequential data. 5) Mamba [Gu and Dao, 2023]: an emerged selective state space model (SSM) that achieves competitive performance on some tasks with sequential inputs. 6) CNN [LeCun et al., 1998]: convolutional neural network contains the convolutional layers, which are effective in processing image data.

For Fourier-based Networks, we mainly compare FAN with 1) Fourier Neural Network (FNN) [Silvescu, 1999] using the cosine or sine function or their linear combinations as the activation function, such as SIREN [Sitzmann et al., 2020]. 2) Fourier Series Neural Network (FSNN) is defined as Eq. (3), which shares the parameters and computation of Sin and Cos part. 3) Fourier Transform Neural Network (FTNN) is a type of neural network that employs Fourier Transform to process the intermediate output in the neural network, such as FNO [Li et al., 2021].

G More Detailed Discussion with Fourier-based Neural Network

For FNNs [Silvescu, 1999, Liu, 2013, Parascandolo et al., 2016, Uteuliyeva et al., 2020], they face challenges in scaling to deeper networks, i.e., the capacity of their deep networks to fit the Fourier coefficients is independent of the network depth, as analyzed in Section 3. The depth scalability limits their applicability to more complex, general-purpose tasks such as language modeling. Our core differences are, 'we design FAN based on the following principles: 1) the capacity of FAN to represent the Fourier coefficients should be positively correlated to its depth; 2) the output of any hidden layer can be employed to model periodicity using Fourier Series through the subsequent layers.' In Section 4.3, we conduct experiments to compare our approach with FNNs, and FNNs cannot fit language modeling tasks, but our approach works well. We provide the analysis of FNNs compared to FAN below. We mainly discuss the work [Silvescu, 1999, Liu, 2013, Parascandolo et al., 2016], due to the work [Uteuliyeva et al., 2020] is a comparative study without proposing a new method.

For work [Lee et al., 2021, Belcak and Wattenhofer, 2022], they focus on different purposes from our work. And work [Lee et al., 2021] assumes all input signals have the period of 1 (as stated in page 3 of its paper), which we conducted experiments on the same setting in Appendix B.6, and it cannot fit our periodicity modeling tasks.

G.1 Limitation

First, we only demonstrate the effectiveness of FAN on some mainstream real-world tasks (including symbolic formula representation, time series forecasting, language modeling, image recognition, etc.), and we aim to further broaden the applicability of FAN in our future work. Second, although we have explored the generalizability of FAN and confirmed that FAN outperforms the baseline method in some real-world tasks, the boundaries of this model's generalizability remain unknown. However, we have not yet identified specific scenarios where it performs poorly. We leave this for our future work.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the limitation of traditional neural networks that can’t properly model periodic functions. The paper introduces Fourier Analysis Networks (FAN), a novel network architecture, based on Fourier analysis, that can replace MLP blocks so models can better represent periodic structure with fewer parameters and FLOPs. The paper shows the effectiveness of FAN, compared to other baseline architectures like MLP, KAN and Transformer on a variety of periodic modelling tasks, time series forecasting and sentiment analysis tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The writing is clear and the motivation is well explained with experiments showing that standard networks extrapolate poorly in periodic data.\n- The proposed architecture embeds efficiently periodicity while reducing the parameter count.\n- The experimental results on periodic tasks are convincing and the method clearly improves OOD generalization.\n\n## Weaknesses\nMy main concern is that the paper claims FAN to be a general-purpose model, while lacking extensive experiment analysis to support this claim. The paper compares against vanilla architectures rather than specialized state-of-the-art models. If FAN layers can serve as replacements for MLPs in existing architectures, would current specialized SOTA actually benefit from this substitution? Sentiment analysis seems like an odd choice to showcase FAN's language modeling benefits, there is no clear periodic structure that Fourier analysis would capture. Combined with the lack of comparison against SOTA models this experiment fails to demonstrate any meaningful advantage for real-world language tasks.\n\nQUESTIONS:\nI believe FAN represents a valuable contribution for modeling periodic functions, but I struggle to see how it qualifies as a general-purpose architecture given the experimental evidence. Could the authors clarify their vision for FAN's practical applications? Specifically, which real-world tasks would benefit from the Fourier-based inductive bias, and how would you recommend identifying when FAN layers would be preferable to standard MLPs? While I understand the authors' argument that many machine learning tasks may benefit from FAN due to hidden periodicity, there's still a gap between this theoretical vision and the experimental validation.\n\n\nMinor point:\nTo improve clarity, I’d suggest moving the description of the FAN(Gated) variant to the main paper.\n\nLIMITATIONS:\nI'd recommend the authors include a more thorough discussion of the limitations in their real-world task selection, specifically addressing why certain tasks were chosen and which types of applications might not benefit from FAN.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Fourier Analysis Networks (FAN), a replacement for MLP blocks that concatenates $cos(W x)$, $sin (W x)$ and a conventional non-linear projection inside every hidden layer. This design is driven by two principles: (1) depth-scaled capacity to learn Fourier coefficients and (2) periodicity accessibility at any hidden layer. The idea is that this should enable FAN to capture periodic patterns effectively while still remaining applicable to a broad range of tasks. They obtain consistent gains on experiments from a variety of domains, including symbolic formula representation, time-series forecasting and language modeling.\n\nSTRENGTHS AND WEAKNESSES:\nThe approach seems to be novel in the space of Fourier Series NNs, specifically in how it integrates sine and cosine functions with other activation functions.\n\nThe variety and scope of experiments are extensive and show impressive generalization across domains. \n\nThe code is provided and in general, the idea is relatively straightforward so it should be reproducible. \n\nAdditional analyses of layer activations and ablations could help to better understand why the approach works better than existing Fourier Series NNs.\n\nQUESTIONS:\nWould it be possible to do a layer-wise spectral analysis to illustrate how coefficients/frequencies evolve with depth? \n\nDo FAN’s cosine/sine blocks hamper tasks that are intrinsically non-periodic? A negative result would be informative, and in my opinion, it would not reduce the value of the contribution. \n\nSection on prior work says: \"Previous studies have explored Fourier-based networks, but these networks generally perform well on specific tasks, while their performance on more general tasks tends to be poorer\" and lists a number of citations. It would be informative to expand that sentence and section further (perhaps in the appendix) and detail more precisely how the proposed approach differs from the prior work. \n\nI suggest doing some ablation experiments to better understand what are the key contributors in FAN. For instance, freezing W_p or having just sine or cosine function or having FAN layers only in part of the network.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a contribution to the well-studied fields of periodicity in neural networks and deep learning with Fourier series.\n\nOn the outset, the authors claim to introduce Fourier Analysis Networks, a purportedly novel neural network designed to effectively address periodicity modeling challenges. They claim that FANs overcomes the limitations of existing Fourier-based networks by enabling scaling to deeper and large-scale models, while simultaneously maintaining general-purpose modeling capabilities. The authors assert that FAN demonstrates superior performance in periodicity modeling tasks, especially in out-of-domain (OOD) scenarios, outperforming existing neural architectures.\n\nHowever, as is detailed below in this review, the work still misses important references to previous work in the field, namely existing methods, benchmarks, metrics, and dataset introduced for this purpose, and fails to perform many comparisons with these related works. Moreover, some experimental set-ups and resulting figures and findings already appear in previous work. \n\nIn sum, I would consider the work in principle passable as a contribution of some novelty if properly embedded in the context of previous work. The singular focus on FANs as an architecture with no equal as currently alluded to in the abstract, Sections 1, 4.2, 4.3, and 5 needs to be mitigated by the appropriate acknowledgement of the existing work in the field.\n\nSTRENGTHS AND WEAKNESSES:\n1. The work happily cites work as old as Newton 1687, yet fails to properly compare -- theoretically, experimentally, or both -- to contemporary works many of which have proposed their own methods for the same purpose:\n\n[1] Adrian Silvescu. Fourier neural networks. In IJCNN’99. International Joint Conference on Neural Networks. Proceedings (Cat. No.\n99CH36339), volume 1, pages 488–491. IEEE, 1999.\n\n[2] Shuang Liu. Fourier neural network for machine learning. In 2013 International Conference on Machine Learning and Cybernetics, volume 1, pages 285–290. IEEE, 2013\n\n[3] Giambattista Parascandolo, Heikki Huttunen, and Tuomas Virtanen. Taming the waves: sine as activation function in deep neural networks. 2016.\n\n[4] Abylay Zhumekenov, Malika Uteuliyeva, Olzhas Kabdolov, Rustem Takhanov, Zhenisbek Assylbekov, and Alejandro J Castro. Fourier neural networks: A comparative study. arXiv preprint arXiv:1902.03011, 2019.\n\n[5] Jiyoung Lee, Wonjae Kim, Daehoon Gwak, and Edward Choi. Conditional generation of periodic signals with fourier-based decoder. arXiv preprint arXiv:2110.12365, 2021.\n\n[6] Peter Belcak and Roger Wattenhofer. Periodic extrapolative generalisation in neural networks. In 2022 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1066-1073. IEEE, 2022.\n\n2. Closely related variants of the FAN proposed in Section 3 have been also proposed in each of [1]-[5], yet comparisons in Table 1 are made only against the traditional MLP.\n\n3. Figure 3 closely follows the exposition Figures 1&2 of [6].\n\n4. On lines 125-130, the point of comparison should not have been MLP but each of the networks proposed in [1]-[5] across the past 25 years of activity in this research field.\n\n5. Experimentation of Section 4.1, Appendix C, and Figure 4 mirrors that of [5] and [6] and does not acknowledge the existence of essentially equivalent synthetic benchmark in [6], which is by the virtue of the close similarity prior art. Note similar omission under point 3.\n\n6. Experimentation of Section 4.1 glosses over RNNs, arguably the most potent architecture for the task at hand. In later experiments, LSTM is mentioned with other RNNs disregarded.\n\n8. Table 4 acknowledges the existence of [1] and FTNNs but ignores [2]-[5] . Note similar omissions in points 2 & 4.\n\n9. The inadequacy of snake activations described on lines 255-258 and in Appendix D is presented as a new finding but has already been pointed out and experimentally detailed in [6]. Note similar omissions in points 3 & 5.\n\n10. The authors do not answer truthfully in the Reviewer Checklist under point 2. No explicit discussion of the limitations has been done in the paper, as their response to the checklist question indicates.\n\nI strongly suggest authors revisit and expand the related work section, make further acknowledgement to prior art in the abstract, Section 1, Section 3, Section 4, and introduce additional baselines where possible.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors have not done any explicit discussion of the limitations of their work. The discussion of a potential negative societal impact has not been given, but would also not be entirely necessary, given the nature of the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Fourier Analysis Networks (FAN), which extends the usual MLP networks to model periodicity in inputs by adding sine and cosine as activation functions to each layer. For motivation, this paper notes that sine and cosine together generate the Fourier Basis in Fourier Series expansion (Section 3).\n\nWith the added sine and cosine activation functions in each layer, this paper argues that periodicity modeling is available throughout the network, and together with MLP activations in each layer this periodicity modeling capacity scales with network depth (Sections 1 and 3).\n\nFor validation, this paper conducts experiments with synthetic data (Section 4.1, Periodicity Modeling) and real-world data (Section 4.2, Symbolic Formula Representation, Time Series Forecasting, Language Modeling), showing that FAN (or FAN Gated) gets lower loss (\\~7%, MSE or MAE) and higher accuracy (\\~5%), using fewer params (\\~15%) and less FLOPs, than competing (non-periodicity-aware) models (such as MLP, KAN, Transformers without FAN, LSTM, Mamba).\n\nEven when comparing with other periodicity-aware, Fourier-based models (FNN, FSNN, FTNN), the newly introduced Fourier Analysis Networks (FAN) still performs better in synthetic data (in Periodicity Modeling, Section 4.3) with lower loss and fewer params.\n\nSTRENGTHS AND WEAKNESSES:\nFor strengths, the experiments shows that FAN gets lower loss with fewer params in real-world data such as time series forecasting and language modeling, and the added sine and cosine activation functions to model periodicity are well motivated, as elaborated in Section 3.\n\nFor weaknesses, the periodicity ratio ($d_p$) is yet another hyper-parameter to tune.\n\nQUESTIONS:\nCurrently in a FAN Layer $\\phi(x) = [\\cos(W_px) || \\sin(W_px) || \\sigma(B_{\\bar p}+W_{\\bar px})]$ (Figure 2), so cosine and sine do not have the bias components ($B_p$), unlike the MLP part. Would adding the bias component help, for example, in modeling “phase shift” in periodicity? Note that after adding bias components, the network can use sine or cosine alone, without using both. In particular, how would such networks flair in experiments?\n\nLIMITATIONS:\n### Note on periodic generalization\n\nLines 78-79, Page 3: “The power of Fourier Series lies in its ability to represent a wide variety of functions, including non-periodic functions through periodic extensions, enabling the extraction of frequency components.”\n\nSuch periodic extensions assume that the generalization is periodic (outside of the primary period), which may or may not hold.\n\nThe examples in Figure 1, Figure 3 and Figure 6 (See Section 4.1) are the best possible periodic generalization for Fourier Analytic Networks: outside the training set, the test set is perfect periodic extension. So by “over-fitting” to the training set, we get perfect generalization to the test set, which is what FAN does best, compared with other networks (MLP, KAN, Transformers).\n\nBut to be fair, the experimental results in Section 4.2 show that periodicity are indeed present in real-world data, e.g., in Time Series, or in Language Modeling.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}