Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling
Authors: Yihong Dong
Ge Li
Xue Jiang
Yongding Tao
Kechi Zhang
Lecheng Wang
Hao Zhu
Huanyu Liu
jiazheng ding
Jia Li
Jinliang Deng
Hong Mei
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Periodicity Modeling
Primary area: deep_learning

Abstract: Periodicity, as one of the most important basic characteristics, lays the foundation for facilitating structured knowledge acquisition and systematic cognitive processes within human learning paradigms. However, the potential flaws of periodicity modeling in Transformer affect the learning efficiency and establishment of underlying principles from data for large language models (LLMs) built upon it. In this paper, we demonstrate that integrating effective periodicity modeling can improve the learning efficiency and performance of LLMs. We introduce FANformer, which adapts Fourier Analysis Network (FAN) into attention mechanism to achieve efficient periodicity modeling, by modifying the feature projection process of attention mechanism. Extensive experimental results on language modeling show that FANformer consistently outperforms Transformer when scaling up model size and training tokens, underscoring its superior learning efficiency. Our pretrained FANformer-1B exhibits marked improvements on downstream tasks compared to open-source LLMs with similar model parameters or training tokens. Moreover, we reveal that FANformer exhibits superior ability to learn and apply rules for reasoning compared to Transformer. The results position FANformer as an effective and promising architecture for advancing LLMs.

Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling
Yihong Dong', Ge Li Xue Jiang', Yongding Tao', Kechi Zhang', Hao Zhu', Huanyu Liu',
Jiazheng Ding?, Jia Li d² Jinliang Deng'. and Hong Mei "School of Computer Science, Peking University PaiXcoder
The Hong Kong University of Science and Technology 4 *Avvancdd Institute of Big Data dongyh @ stu stu.pku.edu.en, lige pku.edu.en
Abstract
Periodicity, one of the most important basic characteristics, lays the founda- tion for facilitating structured knowledge acquisition and systematic cognitive processes within human learning paradigms. However, the potential flaws of peri- odicity modeling in Transformer affect the learning efficiency and establishment underlying principles from data for large language models (LLMs) built upon it. In this paper, we demonstrate that integrating effective periodicity modeling can improve the learning efficiency and performance of LLMs. We introduce FANformer, which adapts Fourier Analysis Network (FAN) into attention mecha- nism achieve efficient periodicity modeling, by modifying the feature projection process of attention mechanism. Extensive experimental results on language mod- eling show that FANformer consistently outperforms Transformer when scaling up model size and training tokens, underscoring its superior learning efficiency. Our pretrained FANformer- BB exhibits marked improvements on downstream tasks compared to open-source LLMs with similar model parameters or training tokens. Moreover, we reveal that FANformer exhibits superior ability to learn and apply rules for reasoning compared to Transformer. The results position FANformer an effective and promising architecture for advancing LLMs. Our code available at httttt:///gthhbb www.:///ooooooongg///aoooooo
1 Introduction
In recent years, large language models (LLMs) have achieved remarkable progress across various natural language processing tasks, establishing themselves comnersoon of modern artificial intelligence [Brown et al. 2020 Zhao et al.) 2023 Minace et al. 2024]. The decoder-only Transformer architecture, in particular, has emerged the de facto standard for L.LM development due to its superior performance and scalability [OpenA] 2023 DeepSeek-A1 et al. 2024a Groeneveld et al.) 2024). Besides these advancements, Transformer- heassforme--baaed models also known for their immense demand for data and computational resources during training [Kaplan et al. 2020 Hoffmann et al., al., 2023). In comparison, humans lasks with far Tewer resources. This discrepancy suggests that existing LLM architectures still suffer from low learning efficiency, leaving substantial room for improvement their ability extract and generalize the knowledge from data.
2022 Chowdhery
able to accomplish similar learning
Periodicity, characterized by recurring patterns, is ubiquitous phenomenon in human life and learning processes [Buzsaki] 20066] Lake et al.) 2017). The human brain leverages pattern recognition mechanisms to process information and acquire knowledge efficiently (Zalta et al., (2200 EEalatii et al.) 2023 Zhan et al.) 2018). However, general network architectures represented by Transformers
have potential flaws periodicity modeling, which could hinder their learning efficiency (Dong et al.)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

68
18
Transformer FANtormer
Avarage Aurfomanca
64
PETMACA
22
5o
Trydameert
02
0
1000
2000 Epoch
3000
4000
54
18
Training Tokens (Thi lons)
(a) Performance Transformer and FANformer pariodicity modeling.
(b) Performance of FANformer language modeling.
The performance of FANformer on periodicity modeling and language modeling. (a) shows the training loss of Transformer and FANformer on the fitting mod function and their performance the 4, 8,000th epoch. (b) shows the average performance of 8 core commonsense tasks between FANformer- IB and the open-source LLMs with comparable model parameters and training tokens.
Figure 1:
2024a Liu et 2020]. As shown in Figure (a), even for simple mod function, Transformer
demonstrates suboptimal performance despite being provided with sufficient training data and model capacity | This inefficiency can be exacerbated during the training process of LLMs to affect their performance, considering the periodicity hidden in large amounts of language data. Fourier Analysis Network (FAN) [Dong al., [024aa has shown preliminary success in tasks with explicit or implicit periodic features, but its adaptation to Transformer architectures for large-scale language modeling remains an open challenge.
In this paper, we present FANformer, a novel foundation architecture for LLMs that adapts FAN into the attention mechanism of Transformer to improve learning efficiency and model performance through effective periodicity modeling. It leverages FAN to introduce Fourier principles for capturing and representing periodic patterns, thereby enhancing the Transformer's capability learn and generalize from data. Specifically, we modify the feature projection process of attention mechanism facilitate modeling periodicity. Figure [ (a)
to incorporate frequency-done domain representations demonstrates the significant ad advantages of FANformer over Transformer on periodicity modeling, with faster convergence speed and better results. In Figure (b), we can observe that FANformer- achieves superior performance with higher utilization efficincyy of model parameters and training tokens when benchmarked against comparable Transformer-bast based LLMs.
To comprehensively validate the effectiveness and scalability of FANformer, we conduct extensive experiments on language modeling tasks. The results of scaling both model parameters and training tokens highlight that FANformer consistently surpasses Transformer, requiring only 69.2% of model parameters or 79.7% of training tokens achieve comparable performance. We also implement a complete pretraining pipeline to pretrain a 1.1-billlllle 1.bbbillion parameter FANformer (FANformer- IB) on trillion tokens. Experiments on various downstream tasks demonstrate that FANformer- IB outperforms open-source LLMs of the same size with fewer training tokens, and exceeds LLMs with three times the parameters when using the same training token. Through further analysis, we reveal that FANformer is superior choice compared to other variant architectures and discover three interesting findings: 1) By observing the training process, we discover the notable enhancements in FANformer's 's learning efficiency over Transformer as the model continues to learn from the data. 2) FANformer facilitates the rule-based reasoning paradigm, mitigating the occurrence of "holes" inherent in the case-based learning of Transformer (Hu et al.) 2024]. Under the stress test of logical reasoning [Wang et al.) 2024]. FANformer- 1B demonstrates superior performance compared to OLMo-IB and Dowen2.5-128 I.5B. 3) FANformer's 's representational capacity consistently surpasses that of Transformer across various layer depths, as evidenced by evaluations of the model's Lipschitz constant [Latorre et al. 2020]. These findings underscore the potential of FANformer an effective and scalable architecture for advancing LLMs.
The main contributions of our work can be summarized as three points: ! We first demonstrate that integrating effective periodicity modeling can improve the learning efficiency and performance of
We sample 400K training data from the function of mod 5 and train 110M Transformer for 4K epochs.

---

LLMs. 2 We propose FANformer, a novel LLM architecture, which uses simple yet effective approach adapt FAN into attention mechanism for efficient periodicity modeling. consistently outperforming Transformers in scaling model parameters and training tokens. @ We pretrain and open-source FANformer-IB IB, which surpasses SOTA publicly available L.L.s with similar parameter counts or training token budgets on downstream tasks.
2
Motivation
In this section, we combine formalization with illustrative cases to demonstrate why periodicity modeling facilitates language modeling and reasoning, thereby elucidating the motivation behind the development of FANformer.
The essence of periodicity lies the repetitive manifestation of certain invariance under transfor- mations, which can be strictly defined through invariance under group actions in Abstract Algebra Dummit and Foote [2004). Let X be a set and G be a group acting on X. An element æ E X is said to be periodic with respect to the action of G if there exists non-identity element p E G such that p-2 x = I, where denotes the group action. The element is called a period of æ. Periodicity implies that x invariant under the action of the cyclic subgroup generated by p. denoted by (p). For example, f(a) = f(a + T)) can be seen as a specific instance of the abstract dcfinition p æ. where x f.
T, and the group action translation. When the input a and the group G extended to higher dimensions or non-temporal domains, the manifestation of the period T also changes accordingly. Crucially, for many reasoning tasks, the underlying operation inference rule remains invariant across structurally similar subproblems, that is, for all inputs belonging to certain equivalence class, the same functional rule is applied, which precisely reflects periodic invariance.
Consider addition as an illustrative case: let / represent the addition operation, a denote the digit position index, and let the period T correspond the positional shift in place value. The reasoning proceeds as:
Example: 357 286=
Digit-wise operations: units (7 6 13 write 3, carry 1): tens (5 88+8+14 14 + write 4, carry 1): hundreds (3 +3+ 2 1 6 write 6).
Result: 643
Thus, the periodicity of addition is manifested in the repeated application of the same rule across different digit positions, where the rule itself remains invariant under positional shifts. can be extended to other reasoning, such logical reasoning, and the scenario is analogous: when a neural network extracts a feature applicable to specific conditions or premises, it repeatedly applies the same invariant rules across analogous subproblems. Such periodic invariance allows for enhancing both the learning efficiency and generalization capability of neural models by reducing redundancy and reinforcing conceptual regularities.
3 FANformer
We will provide a detailed description of FANformer for sequence modeling and adopt decoder-only model to illustrate the architecture.
Given an input sequence {si, si). where s; denotes the i-th token and represents the length of sequence s, it is first mapped to the input embedding X [x] X2, xi] E Rxd,, where d, represents the model's hidden dimension. The embedding is subsequently fed into the model to obtain the final output X with each n-th layer of FANformer processing ~ where n E [1, .). The core each FANformer layer lies in revised attention module that incorporates a modified FAN layer, referred to the ATtention-Fouur Fourier (ATF) module.
3.1 ATF
The attention mechanism serves as a core component of Transformer architectures, enabling dynamic interaction between tokens through query-key-value (QKV) projections. While effective for general sequence modeling, its standard implementation exhibits limitations capturing periodie patterns due to the inherent locality of linear projections in the time domain. To address this, propose
3

---

FANformer
Multihead
def FAlLayer_ WLF, pl:
XLDI (B, L, depl, XLDI 18, L, de(3-24p1)
LineartX K_Fl.sptin de(1-2xp11)
Food Foward Network
Concal
return ConcaticosiK.pi, sinlXpl, X.E)
def ATFXX, YLORS V.P, (8, L, di, XLFt L, MLORS (d, 3dl, KLF: Id, delt-p1) XLF FMlLayrr_IX, KLF, pl OKLF LXnear(X_F, NLOKVI Q_F, K.F, MLF @KV_F.splusidd d, dl) return Softmox(10_F F K.F.T) sqrt(d)) V_F
Nomaatation
Heodd
Lincar
Multi-head ATT
(o 2Dl (8p- KPO
def HulsslfeadkTTIX, NLOKY, WLol:
Heads [ATFIX, KLOK1,, N.F1, pl ror in range(())
Concat [Hedss]
The pseudocode of Multi-head ATF, where p the hyperparameter that controls the proportion of periodicity modeling for Xp.
Figure 2: Left: The illustration of FANformer's architecture. Right:
the ATF module, which incorporates the operations of FAN into the QKV projection process to explicitly model periodicity in the frequency domain. Specifically, given the input X E gx first calculate X E gixda as:
X p. FANLayer' (X) = [cos (W,, (V,
Bp)],
(1)
where FANLayer' represents a variant of the original FAN layer (i.e., Eq. (TO) with the activation function a in Eq. (T)) replaced by the identity function, i.e., (xx x, in this paper, and hyperpa-
rameter p is defined as the proportion of dwp, dn
On this basis, we employ the linear transform to X
compute QKV projections, i.e., QF, KF, V R
follows:
[Qp. KF. V X Wo, WK, Wy],
where Wq. WK Wy E Rdd xd, are learnable parameters. Similar to the standard attention mecha- nism, the computation of ATF is defined as:
QFKP
ATF(X Q: WK, Wv) softmax
VF,
vdn
where QF, Fr, VF are computed using the input X via Eq. [ and Eq. (2.. To enhance the model's capacity, we extend the ATF module to multiple heads. Given input X E RRt the Multi-head ATF first projects X into independent heads through the ATF module. For the i-th head, we have:
Head; ATF(X/ Q W Wt),
where W W Ki Wi E Rddxxd, are learnable parameters for query, key, and value projections respectively, with de da/k. The outputs of all heads concatenated and linearly transformed:
MuliHHeadATF(() [Head, |...||HHdd,, Wo,
(5)
where Wo Xd is the learnable parameter of projection matrix. Note that ATF(X) is mathematically equivalent to (FAntiinnFANI.... Laee'' (X)) (the detailed derivations are provided in Appendix D. This equivalence enables a simple yet effective implementation of Multi-head ATF as shown in Figure 2 which can seamlessly incorporate various advancements in traditional attention mechanisms, such as FlashAttention [Dao al. 2022
3.2 Overall Architecture
The FANformer model comprises N stacked FANformer layers, where each FANformer layer consists of Multi-head ATF module and a ferd-forwwrd network (FFN) module. Following the previous work [Groeneveld et al. 2024 Touvron et al.) [2023a]. we employ SwiGLU (Ramachandran et al.) 2018
Shazeer 2020 and pre-norm [Zhang and Sennrich, 2019 as the enhancements Transformer-baat
LLMss. Specifically, the n-th FANformer layer can be defined as:
Y MullittaadATF((NomMXX + X",
(6)
X = FFN(Norm(Y")) + Y".
(7)

---

where the MultiHeadATF module is computed via Eq. 5 and the FFN module, which leverages the SwiGLU activation, is expressed as:
FFN(X) (SWish(XW11) & XW2) W3.
(8)
where W1, W2 E Rd, xdy, W3 E xd are learnable parameters, @ denotes element-wise multiplication, and dy is the intermediate dimension. The overview of FANformer's architecture is illustrated in Figure
Evaluation
We begin with the implementation details of our experiments (Section 4.1 followed by compre- hensive evaluation of FANformer from three distinct perspectives: First, we investigate the scalability of FANformer by examining its performance trends on language modeling tasks with respect to model size and training tokens (Section 4.2). Second, we evaluate the capabilities the pre-trained FANfomeer-1B IB model across multiple downstream tasks (Section 4.3). Third, conduct in in-depth empirical analysis FANformer, including ablation study, learning efficiency, reasoning mechanism, representational capacity, and more (Section 4.4h. See Appendix A for more experiments.
4.1 Implementation Details
The experiments are conducted on 80 A100 GPUs. We build FANformer upon the foundation of OLMo [Groeneveld et al. 2024], it provides solid pretraining framework of LLMs, with the hyperparameter p set to 0.25 by default. For pretraining FANformer- IB, we randomly sample IT training tokens from OLMo's 's training data, i..., Dolma (Soldaimi al. 2024). For other experiments, we train models on a smaller sample of Dolma, i.c., Dolma vI 6-sample [AllenAl] 2023), with roughly 10B tokens. The detailed pretraining and experimental setups are provided Appendix M
4.2 Scalability of FANformer
We explore the scalability of FANformer compared with Transformer to investigate performance trends in the construction of much larger models.
Setup. We follow OLMo's s configuration and vary the FFN's intermediate dimension d to keep the number of parameters consistent for all models in this experiment. For scaling up model parameters, we adopt Dolma vI v1_6-v____vv 1_6-sample as training data and train LLMs from 268M to 7B. We compare FANformer with the standard Transformer and variant of FANformer, termed ATT which uses MLP layer instead FAN layer in FANformer. For scaling up training tokens, we train IB LLMs on the first 200 billion of our sampled IT tokens.
Results. As shown in Figure 3 the scaling law [Kaplan et al. 2020 empitically aligns well with the results obtained from our FAN- former, underscoring its superior scalability properties. Figure 33 (left) reveals that the im- plementation of FAN consistently surpasses the performance of the standard Transformer across range of model sizes. This finding highlights FA Aformer's enhanced scalability terms of parameter efficiency, as it achieves comparable performance with only 69.2% of the parameters required by the standard Trans- former. Notably, the scaling curve of Trans- former+ATM closely overlaps with that of the standard Transformer, indicating that merely revising attention mechanisms using MLP Layer is insufficient. This observation further emphasizes that FANformer's performance gains are not at- tributable to network depth increase, but rather to its special architectural design. Figure 3 (right) demonstrates that FANformer achieves performance parity with the standard Transformer while utilizing significantly fewer training tokens. Specifically, FANformer requires only 159.6B training
Transtemer Tanatormer ATM nootormer
Teansfomer
TANtorme
2.70
3.0
2.
2,65
20h Tewr
2.7
33% Tever Farane
#arrmmeters aoMe)
V'TANTT SCANE
Figure 3: Language modeling loss of scaling up model parameters and training tokens. Left: we train LLMs from 268M to 7B parameters. Right: we evaluate LLMs every 20B tokens up to 200B tokens.
5

---

Table I: Zero-shot performance of FANformer-IB versus other comparable open-source LLMs
core tasks from the downstream evaluation suite following OLMo. The results of baselines are taken
from the previous works (Groeneveld et al. 2024 Ye et al. 2024 Dong al. 2024b].
Models
Param.
Tokens
ARC-C
ARC-E
BoolQ
Hella.
OBQA
PIQA
SCIQ
Wino.
Avg.
LLMs around parameters
Qwen2.5-1.5B
1.5B
18T
41.2
75.5
74.0
50.2
52.4
75.7
94.7
63.3
65.9
RI-Distill-Qwen
1,5B
18T+
36.2
54,4
69.1
41.8
35.4
65.1
89.5
55.3
$5.9
Llama-3.2-1B
9T.
31.4
47.8
74.5
92.3
60.7
60.4
Timy--uma-vl. (3T)
1.1B
3T
34.8
53.2
64,6
58.7
43.6
71.1
90.5
58.9
99.4
OLMo-1B
1.1B
2T
34.5
58.1
60.7
62.5
46.4
73.7
88.1
58.9
60.4
TLMs tokeens
OpenLLaaA-vv-33B
3B
IT
33.9
67.6
65.7
70.0
26
76.7
92.9
62.9
62.0
Stablel.M-v2-35
3B
IT
32.4
67.3
64.6
68.6
26.4
76
89.5
62.1
60.9
TmyLLama-vI.I (IT)
L.B
33.1
52.5
37.8
70.4
86,4
55.2
55,4
PANformer-18
1.18
17"
43.8
72.5
64.9
64.7
48.2
75.5
94.8
61.
65.6
tokens to match the performance of the standard Transformer trained on 200B tokens, representing 20.3% reduction in training resource requirements. These findings suggest that FANformer exhibits superior utilization efficiency in terms of both model parameters and training tokens compared to the standard Transformer architecture.
4.3 Performance of FANformer-1B
We pretrain FANformer- IB on trillion tokens and report zero-shot performance on set of monsense downstream tasks, following previous work [Brown et al. 20200 Touvron et al. 2023b Groeneveld et al. 2024 inter alia].
com-
Setup. The downstream evaluation suite consists of core commonsense tasks, including ARC-C
[Clark al. 2018), ARC-E [Clark et al. 2018), BoolQ [Clark et al.) 2019], HellaSwag [Zellers et al.
2019]. OBQA Mihaylov et al. 20188
PIQA Bisk et al. 2020 SCIQ [Welbl et al.) 201 and
WinoGrande (Sakaguchi et al. 2020]. We compare pretrained FA FANformer-
open-source
LLMs with comparable model parameters or training tokens, including Qwen2.5- 1.5B TTam 2024
R1 RI-Distill-Qwen1..
DeepSeek-Ale al. 2025]. Llama-3 3.2- IB (Dubey al. 2024 Timy Llama-
vI.1-IB [Zhang et al. 2024). OLMo- Groeneveld et al. 2024 LapenLLMM--22
[Geng and
Liu 2023
and Stab
Tow 2023
Results. Table presents the evaluation results of our pre-trained FANformer- on downstream tasks. It is evident that FANformer- 1B surpasses LLMs with comparable parameter sizes, such Llama-3.2-1BB Tlmma-l 1.1-3, and OLMo-IB, while utilizing significantly fewer training data. Compared to the base model OLMo- IB, FANformer- 1B achieves relative improvement of 8.8% in the average performance of downstream tasks using only half the training data. On these tasks, FANformer- IB also demonstrates performance comparable to Qwen2.5-1.55 which is the current SOTA L.MM around IB. For L.LMs training on IT tokens, FANformer- 1B even exceeds L.MMs with three times the parameters, showing average relative performance improvement of 6.0-7.9% across all tasks. Moreover, while R1-Distill-wwen1 capabilities based on its reported performance, it exhibits significantly weaker general performance on these commonsense downstream tasks. This observation shows shortcomings of distillation, highlighting the necessity of the pre-training stage and the importance of research into more efficient model architectures.
shows notable improvements reasoning
4.4 Further Analysis
4.4.1
Ablation Study and Variant Analysis
Setup. We compare FANformer to other variant architectures, including 1) the above-mentioned Transformer+ATMM 2) Transformer+ATLH which use two linear transforms to compute QKV pro- jection, 3) FANformer (original FAN) that employs Eq. (TO) (original FAN layer) instead of Eq. (T) FANformer, 4) Transformer (FFN +- FAN) where the FFN is replaced with FAN [Dong al.
2024a and 5) standard Transformer their ablations.

---

Table 2: Results of ablation study and variant analysis LLMs with 1B parameters trained Dolma v1 1-6-sample dataset (about 10B tokens). The complete experimental results can be found in Table and Table of of Appendix.
Training
V2 Eval
V2 Eval
V3 Eval
V3 Eval
DownStream Avg Acc. t
Variants
Param.
Loss |
Loss !
PPL
Loss I
PPL
Transformer
1.0x
2.889
3.33
30.20
3.07
24.28
53.10
Transformer (PFN < FAN).
1.0x
2,880
3.31
29.79
3.05
23.96
53.95
Same Parameter"
Transformer ATM
1.0x
2.890
3.33
30.31
3.07
24.36
53.69
Transformer ATI.
1.0x
2.882
3.31
29.68
3.05
23.97
53.46
FANformer (original FAN)
1.0%
2.893
3.34
30.64
3.07
24.50
53.61
FANforerr
1.0x
2.863
3,30
29.40
3,04
23.62
55.19
Same Dimension
Transformer ATM
1.06x
2.886
3.33
30.18
3.06
24.28
52.86
Transformer ATI.
1.06x
2.879
3.31
29.76
3.05
23.94
54,23
FANformer (original FAN)
1.04x
2.887
3.34
30.57
3.07
24.39
53.13
FANformer
1.04x
2.856
3.29
29.22
3.03
23.47
54.88
Results. From Table 2 we have the following findings: 1) FANformer consistently outperforms other variant architectures in both scenarios of the same parameter and same dimension all evalua- tion metrics. 2) The performance of Transformer+ATM and Transformer+ATT is notably inferior to that of FANformer, indicating that the core improvement stems from the ATF module designed. 3) Although Transformer (FFN + FAN) yields some improvement, this enhancement inferior to the gains achieved by FANformer, suggesting that integrating periodicity modeling within attention is more advantageous than FFN on language modeling. 4) Incorporating activation functions such GELU into the attention mechanism tends degrade model performance. Specifically, FAN- former (original FAN) and Transformer+ exhibit weaker performance compared to FANformer and Transformer+ATL. likely because these activation functions suppress certain features, thereby hindering subsequent attention operations.
4.4.2 Training Dynamics
We perform comparative analysis of the loss trends during the training process between our FANformer and Transformer, as illustrated Figure 4 The ex- perimental results indicate that the loss of FANformer decreases more slowly in the early stages compared to Transformer, which we hypothesize may be due to the initial lack of established periodic modeling. As the training progresses and periodic modeling gradually improves, FANformer demonstrates a faster conver- gence rate, with its loss decreasing more rapidly than that of Transformer. Intuitively, once the core of se- mantic knowledge established, the model's inherent periodic modeling lets new concepts attach to existing representations instead of being learned from scratch, accelerating subsequent learning. This result suggests that the model progressively learns from the data, the learning efficiency of our FANformer notably sur- passes the standard Transformer.
Transformer
FANformer
Steps
Figure 4: Training loss of FANformer and
Transformer on early training steps, with the complete training loss provided in Figure 7 of Appendix.
4.4.3 Case-based and Rule-based Reasoning
Setup. Following the work (Hu et al. 2024]. we evaluate the case-based and rule- based reasoning of Transformer and our FANformer on two tasks, including: (1) Modular addition: e (a + b) mod 113 with a, E [0. 112]: (2) Linear regression:
+ 2b + with a, b [0, 99). We finetune pretrained LLMss, i.e., OL IB and FANformer- IB, on each task dataset for 500 epochs and their performance is measured via the Leave-Square- method (10 samples per test point).
7

---

Transforier
Trans former
FINforerr
10
1.0
0.8
14
-08
0.8
0.6
0.6
16
-04
104
-02
0.2
0.0
000
000
(a) Modular Addition Task
(b) Linear Regression Task
Performance of FANformer and Transformer on modular addition and linear regression tasks, where the darkened regions indicate areas where the model performance approaches zero, signifying the emergence of the "hole" as described in the work [Hu et al. 2024].
Figure 5:
Results. As shown in Figure 10 of Appendix, both Transformer and our FANformer achieve near- perfect accuracy on the training set of modular addition and linear regression tasks, approaching approximately 100%. However, critical divergence emerges in their performance on the test sets. Specifically, as shown Figure Transformer exhibits a pronounced failure generalize, resulting in "black hole" in the center of the figure, indicating that its accuracy on the test dataset drops to nearly zero. This observation is consistent with the findings reported the work [Hu et al. 2024]. In contrast, FANformer demonstrates marked improvement in addressing the "hole" issue. In the linear regression and modular addition tasks, there no obvious hole observed, further corroborating hypothesis that, relative to the Transformer-beee based model, FANformer possesses stronger tendency to learn underlying rules, thereby achieving superior generalization performance.
4.4.4 LLMs' Proficiency in Logical Reasoning
Setup. Following the work [Wang et al. 2024), we adopt the ULogic dataset to systematically evaluate LLMs their ability to capture underlying inferential logic, where ULogic is constructed after these LLM's training data cutoff, which prevents data contamination [Dong et al., [022cc]. We leverage the Law of Con-Contraaiition [Priest et al. 2006], each rule is paired with negated- conclusion variant; a response correct only the model accepts the original and rejects its flip. We evaluate FA OLMo- Qwen2.5-1.5B, and GPT-4 the two most challenging levels of oogii for stress-testing.
Results. Table 3 summarizes the results on ULogic dataset, with illustrative case presented in Figure 13 of Appendix. FANformer-
Table 3: Average performance of different LLMs on ULogic.
IB substantially outperforms both Qwen2.5-1.5B and OLMo- IB: OL naively labels every rule True, and Qwen2.5-1.55 yields largely contradictory answers, whereas FANformer-18 can affirm the original rule and reject its negated counterpart. These
Model
Ace (%)
GPT-4
65.1
OLMa-1B
0.0
results highlight FANformer's superiority in logic reasoning and underscore the architectural advantages conferred by the periodicity- aware design of FANformer.
Qwen2.5-1.55
7.1
FANformer- IB
38.2
4.4.5
Representational Capacity
Setup. We explain the advantage of FAN- former architecture from another perspective:
the model's ability to learn complex functions. larger Lipschitz constant is fireqeetly linked to greater representational capacity [La torre et al.) 2020 Bartlett al.) (2017), as enables neural networks to approximate more intricate functional mappings by loosening smoothness constraints in parameter space. This relationship stems from the theoretical framework where less restrictive Lipschitz
MLP Transformer
Punformer
Number
Figure 6: Representational capacity MLP, Trans- former, and FANformer across various layer depths.

---

conditions permit sharper decision boundaries and richer expressive power. Formally, L satisfies:
Vx, y E R", [ (2) - Fmmodel (v)ll - L - yll-
Results. We evaluate the representational capacit of MLP, Transformer. and FANformer across various layer depths via computing their Lipschitz constant L. Although representational capacity rises for all three architectures as additional layers are stacked, the growth rates diverge substantially, FANformer exhibits the steepest trajectory and attains the largest Lipschitz constant L at each depth, consistently surpassing Transformer by 27.3% in deeper configurations. These findings indicate that FANformer has markedly richer expressiveness and superior ability to model complex functions.
5 Related Work
Large Language Models The rapid advancement of LLMs has revolutionized NLP and Al research
[Radford, 2018 Dubey et al., 2024 DeepSeek-AI et al.. 2025]. The emergence of GPT-3 [Brown] et al. 2020], with 175B parameters, showcased remarkable few-shot prompting abilities, suggesting that scaling laws (Kaplan et al. 2020] could unlock emergent capabilities. Recent notable LLMs
like PaLM [Chowdhery et al. 2023], LLaMA [Touvron et al. 2023b], GPT-4 [Chowdhery et al.,
2023]. and DeepSeek [Bi et al. [2024] further pushed the boundaries of model size and performance.
Moreover, the open-source release of OLMo [Groeneveld et al. 20224 has provided valuable resources
for the community, enabling more accessible training of L.LMS.
Advances in Transformer Architecture Recent advancements in Transformer architecture focus on overcoming two main limitations: computational inefficiency in long-context processing and limited expressiveness of attention mechanisms. To address long-context inefficiencies, techniques such sparsity [Child al. 2019] and local attention (Beltagy al., 2020). along with innovations in query mechanisms [Shazeer] 2019 Aiissle et al., 2023 and inference acceleration [DeepSeek-A] al. 2024b), have been proposed. Hardware-level optimizations like FlashAttention [Dao et al. 2022 further reduce GPU memory access overhead. To enhance the expressiveness of attention mechanisms, methods like Probabilistic Attention Keys (Nguyen et al.] 2022] and Selective Attention Leviathan et al. 2024] improve semantic relationship capture and refine attention by suppressing irrelevant features. Additionally, approaches like Differential Transformer address attention noise in long contexts YYe et al., 20044) Different from previous work, we improve language modeling by addressing the challenge of periodicity modeling in Transformers, which can seamlessly incorporate the aforementioned works for revising the attention mechanism, as demonstrated in the derivation provided in Appendix L
Fourier-based Networks Previous research on Fourier-basee Networks was aimed at solving some
domain-specific applications [Zuo and Cail 2005 Tan 2006 Chen et al. 2022 Li et al.
202 Some
studies specifically explored the use of sinusoidal activations (e.g.. cosine Silvescu 1999 Ngoom
and Marin 202T] or sine [Parascandolo et al. 2016 Sitzmann et al., 2020 to approximate periodic
patterns LLu 2013). However, these approaches lacked generalizability beyond narrow domains due
to rigid frequency parameterization and limited scalability [Uteuliyeva et al. 2020 Liu et al. 2020)
Recent work (Dong et al., 2024a] addresses these problems using FAN to introduce Fourier Principle into neural networks, but its adaptation to L.MMs remains an open challenge. FNet [Lee-Thorp et al., 2022) replaces self-attention with Fourier Transform to achieve linear complexity, but it sacrifices the performance of LMs. In contrast, we employ effective periodicity modeling to improve LLMs.
6 Conclusion
We propose FANformer, novel LLM architecture that enhances learning efficiency by adapting Fourier Analysis Network into attention mechanism for effective periodicity modeling. Experiments show that FANformer outperforms Transformer when scaling model parameters and training to- kens, achieving better performance with 31% fewer parameters and 20% fewer tokens. Pretrained FAAfformer-1BB surpasses open-source LLMSs of comparable size or training scale on various down- stream tasks. The discovery of FANformer's enhanced scalability, learning efficiency, learning advantages, and representational capacity suggests potential pathways for developing more efficient and high- high-performance language models.

---

7 Acknowledgement
This research supported by the National Key R&DD Program under Grant No. 2023YFB4503801, the National Natural Science Foundation of China under Grant No. 62192733, 62192730, 62192731, the Major Program (JD) of Hubei Province (No.2023BAA024).
References
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrIPS, 2020.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen, A survey of large language models. CoRR. 1wt.22333.1222.. 2023, doi: 10.48550/ 2.2.V.2333..8223. Utt/https: //do:: 100 40w.00/85500xx/. 2303. 18223
Shervin Minaee, Tomás Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. CoRR, abs/2402.06 0w..400..66666 0619,,666666 2024. doi: 10.48550/RRXI.2402.000 166... Utt. org/10. 48550 20550/arXiv..o 2402.06196
OpenAI. GPT-4 technical report. CoRR, abs/2303.08 08774, 2023.
DeepSeek-AL, Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, and Wangding Zeng. Deepseek- v3 technical report. CoRR, abs/2412.19437, 2024a. doi: 1044850/ARXX..2412..1k2 URL https://doi orroorg//// 4855048855iii 2412. 19437
Dirk Groeneveld, Iz Beltagy, Evan Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha. Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff. Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models. In ACL (1), pages 15789-15809. Association for Computational Linguistics, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford. Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.0834 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia
10

---

Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR, 0tts/2203.155566 2022. doi: 10,48550ARXXIV220.. 155566 URL https: org/16 48550/arXiv. 2203.. 15556
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Mcier-Hellsterr Hellstern. Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1-2400 13, 2023. URL htttps:////mm org http://appr/////22211222
Gyorgy Buzsaki. Rhythms of the Brain. Oxford university press, 2006.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Arnaud Zalta, Spase Petkoski, and Benjamin Morillon. Natural rhythms of periodic temporal attention. Nature communications, 1 1051, 2020.
Mohammadreza Edalati, Fabrice Wallois, Javad Safaie, Ghida Ghostine, Guy Kongolo, Laurel J Trainor, and Sahar Moghimi. Rhythm in the premature neonate brain: Very early processing of auditory beat and meter. Journal of Neuroscience, 43(15):2794-2800 2023.
Lexia Zhan, Dingrong Guo, Gang Chen, and Jiongjiong Yang. Effects repetition learning on associative recognition over time: Role of the hippocampus and prefrontal cortex. Frontiers in human neuroscience, 12:277, 2018.
Yihong Dong. Ge Li, Yongding Tao, Xue Jiang, Kechi Zhang, Jia Li, Jing Su, Jun Zhang, and Jingjing Xu. Fan: Fourier analysis networks. CoRR, abs/2410.02675, 2024a.
Ziyin Liu, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how to fix it. In NeurlPS, 2020.
Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei Ma, Jue Chen, Binhua Li, Zhi Jin, Fei Huang, Yongbin Li, and Ge Li. RL-PLUS: countering capability boundary collapse of Ilms in reinforcement learning with hybrid-policy optimization. CoRR, abs/2508.000222. 2025a.
Yihong Dong, Zhaoyu Ma, Xue Jiang, Zhiyuan Fan, Jiaru Qian, Yongmin Li, Jianha Xiao, Zhi Jin, Rongyu Cao, Binhua Li, Fei Huang, Yongbin Li, and Ge Li. Saber: efficient sampling with adaptive acceleration and backtracking enhanced remasking for diffusion language model. CoRR, abs/2510.1 18165, 2025b.
Yi Hu, Xiaojuan Tang, Haotong Yang, and Muhan Zhang. Case- Case-based or rule-based: How do transformers do the math? In ICML. OpenReview.net, 2024.
Siyuan Wang. Zhongyu Wei, Yejin Choi, and Xiang Ren. Can LLMs reason with rules? logic scaffolding for stress-testing and improving LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages
Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v w/202/aacl-oo2oo.g.gee acl-long.444
URL
Fabian Latorre, Paul Rolland, and Volkan Cevher. Lipschitz constant estimation of neural networks 2020.
via sparse polynomial optimization. In ICLR. OpenReview.net, 2020.
11

---

David S. Dummit and Richard M. Foote. Abstract Algebra. John Wiley & Sons, Hoboken, NJ, 3rd edition, 2004, See Sec. 1.7: a group action is *free* iff only the identity fixes any point. Equivalently, if non-identity p E G satisfies p x = then x is * **eriodii*** point under the G-action (invariant under (p)).
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficiett exact attention with io-awareness. In NeurIPS, 2022.
Hugo Touyron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang. Angela Fan. Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288. 2023a.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. In ICLR (Workshop). OvenReview.net, 2018.
Noam Shazeer. GLU variants improve transformer. CoRR, abs/2002.05202, 2020.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In NeurlPS, pages 12360- 1,371, 2019.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Au- thur, Ben Bogin, Khyathi Raghavi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge. and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In ACL (1), pages 15725-15788. Association for Computational Linguistics, 2024.
AllenAI. Dolma, 2023. URL A
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023b.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CORR, abs/1 /d/11803.0555, 2018
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL-HIT (1), pages 2924-2936. Association for Computational Linguistics, 2019.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? ACLC pages 4791-4800. Association for Computational Linguistics, 2019.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In EMNLP, pages 2381-2391 Association for Computational Linguistics, 2018.
12

---

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense natural language. In AAAI, pages 7432-7439, AAAI Press, 2020.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsoureing multiple choice science questions. In NUT@ ENNLP, pages 94-106. Association for Computational Linguisties, 2017.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An
adversarial winograd schema challenge scale. In AAAI, pages 8732-8740. AAAI Press, 2020.
Qwen Team. Qwen2.5: A party of foundation models, September 2024, URL https://qwen.mm github. (t/pobb.../blgg//e/2225/)
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Hcafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.21188, 2024.
Peiyuan Zhang. Guangtao Zeng. Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. CORR, abs/2401 02385,238522 2024.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023, URL https //github, ctw
Jonathan Tow. Stableim alpha v2 models, 2023. URL. [https://hugginggaace co/
stabilitya//stabeelmbaaass alpeeellphpaaaaa stablelm-base-allha--33
Tianzhu Ye, Li Dong, Yuging Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. CoRR, abs/2410.052555 2024.
Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih- Yang Liu, Mathhijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, and Pavlo Molchanov. Hymba: hybrid-head architecture for small language models. CoRR, 0bs/2411.13676 2024b.
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization memorization: Data contamination and trustworthy evaluation for large language models. In ACL (Findings). pages 12039-12051 Association for Computational Linguistics, 2024c.
Graham Priest, Jeffrey Beall, and Bradley Armour-Garb. The law of non-contradiction: New philosophical essays. Clarendon Press, 2006.
Peter Bartlett, Dylan J. Foster, and Matus Telgarsky. Spertrally-normmllzzedd margin bounds for neural networks. In NIPS, pages 6240-6249, 2017.
13

---

Alec Radford. Improving language understanding by generative pre-tratee traininniig..... 2018.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, Alex X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu,, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang. Shiyu Wang. Yaohui Wang, Yongji Wang. Tong Wu, Y. Wu, Xin Xie, Zhenda Xie. Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang. Yuxiang You. Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou, Deepseek LLM: scaling open-source language models with longtermism. CoRR, (55/2401.029544 2024.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR, 1et//9904.1000 2019.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. CoRR, abs/2004.05150, 2020.
Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs//911.0021. 2019.
Joshua Ainslie, James Lee- Leo-Th Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In EMNLP, pages 4895 4901. Association for Computational Linguistics, 2023.
DeepSeck-AI, Aixin Liu. Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao. Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo. Dejian Yang, Deli Chen, Dongjic Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, Hao Zhang. Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song. Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang. Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng. Tao Wang. Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao. Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, and Xiaowen Sun. Deepseek-v2: A strong, economical, and efficient mixture-of-expert language model. CORR, abs/2405.044 04434, 2024b.
Tam Minh Nguyen, Tan Minh Nguyen, Dung D. D. Le. Duy Khuong Nguyen, Viet-Anh Tran, Richard G. Baraniuk, Nhat Ho, and Stanley J. Osher. Improving transformers with probabilistic attention keys. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 16595-16621. PMLR, 2022.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Selective attention improves transformer. CoRR, abs/2410.02703, 2024.
Wei Zuo and Lilong Cai. Tracking control of nonlinear systems using fourier neural network. In Proceedings, 2005 IEEE/ASME International Conference on Advanced Intelligent Mechatronics. pages 670-675. IEEE, 2005.
HS Tan. Fourier neural networks and generalized single hidden layer networks in aircraft engine fault diagnostics. 2006.
Hanlong Chen, Luzhe Huang, Tairan Liu, and Aydogan Ozcan. Fourier imager network (FIN): A deep neural network for hologram reconstruction with superior external generalization. Light: Science & Applications, 2022.
14

---

Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhat- tacharya, Andrew M. Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In ICLR. OpenReview.net, 2021.
Adrian Silvescu. Fourier neural networks. In IJCNN" 99.. International Joint Conference on Neural Networks. Proceedings (Cat. No. 99CH136339,, volume I, pages 488-491, IEEE, 1999.
Marieme Ngom and Oana Marin. Fourier neural networks function approximators and differential equation solvers. Statistical Analysis and Data Mining: The ASA Data Science Journal, 14(6): 647-661, 2021.
Giambattista Parascandolo, Heikki Huttunen, and Tuomas Virtanen. Taming the waves: sine as activation function in deep neural networks. 2016.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances neural information processing systems, 33:7462-74733 2020.
Shuang Liu. Fourier neural network for machine learning. In 2013 international conference machine learning and cybernetics, volume 1, pages 285-290. IEFE, 2013.
Malika Uteuliyeva, Abylay Zhumekenov, Rustem Takhanov, Zhenisbek Assylbekov, Alejandro J. Castro, and Olzhas Kabdolov. Fourier neural networks: A comparative study. Intell. Data Anal., 24(5):1107-11 2020.
James Lee-Thorp. Joshua Ainslie, llya Eckstein, and Santiago Ontañón. Fnet: Mixing tokens with fourier transforms. In NAACL-HLT, pages 4296-4313, Association for Computational Linguistics, 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2312.00075,, 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In SC, page 20. IEEE/ACM, 2020.
Shen Li. Getting started with distributed data parallel, 2018. URL. https: org/ futorials/intermediaa http://meeieaeeeppppt.trrrraaaaa tt..oraa.....mll
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster). OpenReview.net, 2019.
Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, and Xun Zhou. dlpper-conncctions CoRR, 19662440..1966, 2024.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. AAAI, 2011.
Carri-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124, 2019.
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In @WP@ JJCNLP. Asian Federation of Natural Language Processing, 2005.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In MLCW, volume 3944 of Lecture Notes in Computer Science, pages 177-190. Springer, 2005.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631-1642. ACL, 2013.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-though prompting elicits reasoning in large language models, Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately present the paper's contributions and scope, with claims that are fully supported by the results and discussion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper thoroughly discusses limitations in Appendix J.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Section offers theoretical explanations for why periodicity facilitates lan- guage modeling and reasoning, Appendix H contains formal proofs of our methodological claims, and Appendix E. presents theoretical analysis of the computational costs associated with our approach.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We include detailed descriptions of our methodology. model architectures, hyperparameters, and training procedures both the main text and appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
17

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have publicly released our source code at hhtpp:::///// hatp::///nnnmoouu...mm 4open science/r/FANformer with comprehensive instructions for data acquisition and experiment reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht //nips.cc/pll /ttp:///dddeeeoeeeiiiieieeee ssw.nsolccyysysooccooooccoooiiii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all implementation details and experimental setups in both the main text and appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We follow previous work OLMo to report the result of no bias expectation, which reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the computational resources used for experiments implementation details section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't iudn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:////eerrrrssscccppp////ssss
Answer: [Yes]
Justification: Our research fully complies with the NeurIPS Code of Ethics, with careful consideration all ethical guidelines throughout our work.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is foundational research model architecture design that is not tied particular applications deployments.
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper focuses on fundamental architectural improvement for language models rather than releasing high-risk models or datasets. While we do release a language model, it is primarily intended as demonstration of our technical approach and doesn't 't introduce novel risks beyond those already present in existing language models. Our work doesn't 't involve scraped datasets or image generators that would require special safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All benchmark datasets are used in accordance with their respective terms of use, and we've ensured compliance with the licensing requirements for third-party code and models incorporated in implementation. Where applicable, we include URLs to the original sources our references and implementation details.
Guidelines:
The answer NA means that the paper does not existing assets.
20

---

The authors should citep the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. thccode com/datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the GitHub repository for our work, including detailed descriptions, usage instructions, and licensing information in our repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We developed and released 1.0B LLM based on proposed novel LLM architecture, which achieves SOTA performance.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy htttoss///eerrrsssccc///ofrrrrrr what should or should not be described.
for
22

---

A Training Loss Curves of OLMO and FANformer
We present the training loss curves for OLMO and FANformer trained on 1 trillion tokens (i.e., 250K steps) in Figure 7
OLMe OL DiNN Mo-i5 OLMC-15 OLM0-14 14
01M4-11
CLNoS OU CLM63 CLM62 OLNo-
TAlrSine-4 [AMpire-) in fAMlamen in Eitbrrnn
LOss
Final Loss: 2.4944
Final Loss: 2.3769
50k
Step
Step
(a) Training loss of OLMO-1B -B
(b) Training loss of FANformer IB
Figure 7: The training process of OLMO and FANformer. The data Figure (a) is sourced from the publicly available results of OLMO (https://wand..aill)
B Preliminary Knowledge
Fourier Analysis Network (FAN) Dong et al. 2024a enhances neural networks by introducing Fourier principles for effective periodicity modeling. The core component of FAN is its layer design, which combines periodic basis functions with standard linear transformations. Given a input X, the FAN layer is defined as:
FANLayer(X)
(i,
Bp)
(10)
where W, and We are learnable projection matrices, Bş is a bias term, a denotes an activation function, and represents concatenation. Compared to MIPP layer, FAN layer explicitly encodes periodic patterns through Fourier series while maintaining
modeling capabilities.
C Experiments on Code Generation and Math tasks
We conduct experiments code generation tasks (i.c., HumanEval and MBPP) and human-written math tasks (i.e., GSM8K) compared to our baseline, OLMo Grreneveld et al. 2024]. The results show that our FANformer achieves clear and consistent improvements compared with OLMo on all three benchmarks.
Table 4: Comparison of LLMs coding and math benchmarks.
LLMs
Training Tokens
HumanEval
MBPP
GSM8K
OLMo-IB
3T checkpoint
5.2
3.1
8.9
FANformer-1 1B
IT from scratch
6.3
5.4
15.7
D Effect of hyperparameter p
We systematically investigate the impact of hyperparameter which controls the proportion of periodicity modeling in FANformer, on model performance across its value range. The experimental results from the IB-scale FANformer (as shown in Figure 8) demonstrate that our model exhibits
23

---

strong robustness in termss of training loss and downstream task accuracy, with relatively small perfor- mance fluctuations. Furthermore, regardless of the variation p-values. FANformer consistently outperforms the standard Transformer (horizontal baseline). Analysis of experimental results from models of different scales (300M, IB, 3B) (as shown Figure reveals clear trend: larger models tend to exhibit higher optimal values. This observation suggests that more powerful FANformers are better equipped to extract more intricate latent periodicity features.
2.890
55.00
Performaan Downstreal lasks
2.885
54.75
2.880
54.50
2.875
54.25
54.00
2.860
53.50
2.855
53.25
2.850
53.00
0.15
0.2
0.25
0.3
0.35
Hyperparameter
Figure Effect of hyperparameter in FANformer on its training loss and downstream task performance, where the red dashed line represents the training loss of Transformer, while the blue dashed line denotes the performance on downstream tasks of Transformer.
FANformer-300M
3.11
3.10
3.09
0.10
0.15
0.20
0.25
0.30
0.35
FANformer-1B
2.865
2.860
2.855
0.10
0.15
0.20
0.25
0.30
0.35
FANformer-3B
2.754
2.752
Less
2.750
2.748
0.25
0.30
0.35
Hyperparameter
Figure 9: The impact of hyperparameter on FANformer models of varying sizes.
24

---

Table 5: Evaluation results of OLMo-IB-SFT and FANformer- -B--mer--B--FT on MMLU, AlpacaEval. ToxiGen, and TruthfulQA (Info+True). Higher values are better for MMLU, AlpacaEval, and TruthfulQA, while lower values are better for ToxiGen.
Model
MMLU
AlpacaEval %win f
ToxiGen
TruthfulQA (Info+True)
0-shot f
% Toxic 4
Accuracy 1
OLMo-IB-SFT
24.3
1.90
2.8
55.8
LANformer-IB-FY'
26.7
2.51
10.4
83.0
E
Instruction Following with SFT
E.1 Models
FANormer--B--FT. Our pretrained model on trillion tokens, fine-tuned using supervised fine- tuning (SFT) the
dataset.
JLMo-IB-SFTI IB parameter version of OLMo, pre-trained on 3 trillion tokens and fine-tuned dataset. Model available using supervised fine-tuning (SFT) on the teeftulu--see full--3-sft-olmoo2-mixxxxx datases /
For both models, we follow the tokenizer's chat template for prompt formatting when available.
E.2 Evaluation Setup
We evaluate the models on four benchmarks: MMLU, TruthfulQA, AlpacaEval, and ToxiGen. The evaluation is conducted using the open-instruct evaluation suite, which has been widely adopted for evaluating instruction-tune language models. Below, describe the setup for each benchmark.
MMLU We use the official MMLU evaluation script with O-shot prompting. The maximum token length is set to 2048, and we not employ chain-of ofah-hof-houught (CoT) reasoning. The evaluation reports the average accuracy across test examples.
AlpacaEval We use the AlpacaEval VI benchmark with the default setup from the official repository The evaluated models generate responses for 805 prompts, and GPT-4 is employed to compare the responses with those from the reference model (gpt-4-1106) 106-preview). Given the updates in the AlpacaEEall repository, the default models have changed over time. Currently, the default setup annotator as the annotator and 1144-110--prriiew as the
uses the weighted_ caadd_alpaaaaaaee _aal__pp___aa reference model. Therefore, our evaluation aligns with the current default configuration in the official AlpacaEval repository.
ToxiGen For ToxiGen, we focus on the prompts specifically designed to elicit toxic language ('hateful' prompts). To reduce evaluation costs, we use subset of 500 prompts per group for testing. The toxicity classifier used is toxigen_roberte We report the percentage of generations classified as toxic by the classifier.
TruthfulQA For TruthfulQA, we use the generation setting with the default QA prompt format, including 6 in-context examples. The judge model for evaluating truthfulness and informativeness is wwweenii/ruthffllle wwwp:llpa-tuuut-j-e--ee-eee tion suite and OLMo's evaluation. We report the percentage of responses deemed both informative and truthful.
which is adopted in the open-ins instrrcttttt evalua-
F Computational Cost Analysis
We analyze the computational overhead of FANformer compared to standard Transformer under two settings: (I) when the number of parameters is matched, the computational cost remains unchanged; (2) when the FFN dimensions remain fixed, the increase cost is negligible.
25

---

To ensure fair comparison, we maintain the same number of parameters in FANformer as in the original Transformer by varying the FFN's intermediate dimension d during our experiments. Under this configuration, the computational cost remains equivalent between FANformer (same parameter) and Transformer. We further investigate the setting where FFN dimensions remain unchanged (Table Even for FANformer (same dimension), the computational cost increases only slightly compared to the Transformer. Specifically, the ratio of additional computation can be expressed as:
1.5LSD2
1.5D
1.5D
24D 45 -
0.0625,
(11)
L X (24SD² + DS2D) 2SDV
24D
where I. is the number of layers, S is the sequence length, D) is the hidden dimension, and V is the vocabulary size. In practice, the actual additional overhead is much lower than 0.0625, as the term scales inversely with sequence length. This means the overhead diminishes rapidly sequences grow longer. We summarize the detailed comparison computational costs between Transformer and FANformer in Table
Table 6: Computational cost of Transformer vs. FANformer (p = 0.25).
Model
SclfAAttentinnAAE FLOPS
FEN FLOPs
Output FLOPss
Total FLOPs
Transformer
1 (8SD2 4)
L 16SD
2SDV
L (24SD 45*D) + 2SDV
FANfomerr (Some Param)
1. (8SD) 482D + 1.5DP)
Lx (169D2 1.589))
2SDV
1x (24812 4.45P) +8DV
FANfomner (Same
(8S1 48 L.5DD)
168D
28DV
(25.58D* (48+D) 2SDV
G Inference Speed and GPU Memory Usage
We conduct experiments on the inference speed and GPU memory usage of FANformer relative to Transformer in deployment settings, and have added the experimental results to revised manuscript. The results show that it add little latency. The configuration of benchmark test: we run for 20 iterations single GPU of A100 80G with fixed sequence length of 4096 tokens and float16 precision.
Table 7: Inference speed and memory usage comparison (Sequence Length=4096).
Metric
OLMo-1B
FANformer-1B
Difference
Forward Pass Time
141.49 ms
142,88 ms
+1.39 ms (+0.98%)
Allocated Memory
4642.69 MB
4738.86 MB
+96.17 MB (+2.1%)
Peak Memory
6610.70 MB
6706.88 MB
+96.18 MB (+1.5%)
H Different with Mamba
First, our approach is fundamentally different from SSMs. SSMs model periodicity along the sequence dimension, while our FANformer models it along the feature dimension. Second, our motivation is distinct from that of Mamba. Mamba [Gu and Dao 2023 is primarily developed to overcome the quadratic computational complexity of Transformers and improve inference efficiency, while our approach is designed to improve learning efficiency and performance of Transformers through effective periodicity modeling.
I Detailed Results of Ablation Study for Section 4.4.1
In ablation study, we report the average results across various tasks V2 Validation Sets, V3 Validation Sets, and Downstream tasks, with the specific tasks detailed in Section M.3 The complete results are detailed in Table g and Table g
J Extended results of Section 4.4.3
The training and testing performance metries, including loss and accuracy, for case-based and rule-based reasoning are presented in Figure [ and Figure [2 respectively.
26

---

Table 8: The detailed results of ablation study (Part One). All models keep the same number of parameters and are pretrained on Dolma v1 6-sample dataset (about 10B tokens).
Transformer
Transformer
Transformer
FANforeer
Activation
FANformet
3.10
3.12
PPL
22.38
22.52
22.18
22.63
21.87
ollen
PPL
Lass
26.40
26,54
3.78
26.22
3.27
1.29
26.78
3.25
25.85
Loss
1.90
3.90
3.89
3.91
3.87
PP1
49.58
49.64
49.11
50.05
47.83
3.21
3.19
3.21
34.59
23.03
22orc
PPL
36.24
V2 Validation Sets
m262. wiki
manosphere
Loos
PPT
PPL
3.47
23.17
52.21
23.13
3.48
32.46
22.90
31,65
3,46
3.48
23.29
32.62
3:45
31 48
22.18
mo4_cn
Loss
PPL 30.53
3.02
20.52
3,02
3.01
20.22
3,03
20.76
2.95
19.91
2.76
2.76
2.74
2.77
2.73
13.84
15.30
3.64
twitte
PPI 60.25
60118
wikitest 103
Loss 3.33
PPL 28.03
25.07
3:33
27.15
3.30
3,35
28.48
3.29
26.88
Los: 3.33
3.33
3,31
3.34
3.30
Average
PPL 30,20
30.31
29.08
30.564
29. 40
of
PPL 24.80
Los
3.1
24.86
3.31
24.60
3,30
25.04
1.22
24.24
3.15
dolna books
34.98
3.54
33.06
3.52
dolmu_common-crnw
dolma_pes20
PPL 17:45
3.86
2.85
17.35
17.09
2.84
2.85
17.53
16.88
2.83
dolma nediit
PPL
Loss
31.13
144
31,35
3.44
30.94
3,43
1.45
31.42
30.54
3.43
V3 Validation Sets
dolma_wiki
Stack stack
Loos
PP1
4:13
20,89
1.42
4.10
3.04
141
3,03
4.06
1,40
4.13
1.42
3.01
201
4.01
1.39
3.15
m282 s2one
PPL
40.35
40.61
40.22
40.56
39 50
pile
Loss
PP1
2.74
15:44
2.73
15.35
2.72
15.16
2.75
18.58
2.70
14:92
wikneett_ 103
PPL 28.21
Los
3.34
28.11
3.34
27.33
3.31
3.35
28.57
27.03
3.30
Loos
3.07
3.07
3,05
3.07
3.04
P9L
24.28
23.62
piga
65.15
hellaswng
33.87
winegrande
52.80
49.96
$1.72
spenbook_gga
ACC
23.00
28.20
28.00
28.20
29000
sciq arc. racy
ACC
70.30
72.10
69.00
67.20
71
Downstream Benchmarks
cupa
ACC
ACC
62.00
45.44
66.00
46.14
87.19
65.00
66.00
47.02
45.61
66.00
ne
ACC
51.26
52.35
52.71
48.74
57.04
commimest bank
ACC
42.86
41.07
46.43
53.57
44 64
ACC
$1.05
81.22
81.22
81.22
B1.17
59.11
Average
53,10
55.19
27

---

Table 9: The detailed results of ablation study (Part Two), All models keep the same dimension and are pretrained on Dolma v1 6-sample dataset (about 1OB tokens).
Transformr
Transformer AlM
Transfomer AIL
FANformer Activation
FANformer
4cham
Loss
2.68
14,60
2,68
14.54
2.67
14,43
2.68
14.63
2,66
domainn
22.38
3.08
64_c0
gub
Less
PPL
3.90
40.58
3.90
49.41
3.89
48.97
50.11
3.91
47.79
3.87
ice
PPL
Loss
24.59
1.20
3.30
24.62
24.22
3.19
24.90
3.21
23.09
5.17
m242 store
Loss
PPL
35.34
1.56
3.58
35.73
35.1
3.56
35.
3.58
34,58
3.54
3.30
Valisaaiin Sets
22.88
manosphere
31.78
me4_es
Loss
PP1
20.53
3.02
3,02
20.51
3,01
20.22
3,03
20.61
2.99
19:86
pile
PPL
Loss
2.76
15.84
2.76
15.78
2.74
15.54
2.77
15.90
2.72
15.24
Toss
1.68
3,67
3.67
5.73
5,63
PPL
39,68
39.19
39.15
41.67
37,82
twineAFF
60.97
4.07
wikitext_
27.96
27.29
Average
PPL 30.20
3.33
30.18
29.76
3,31
3.34
30.57
29.22
e4_en
PPL. 24,80
Loss
3.21
3.21
34.78
24.52
3,20
24.82
3.21
24.00
5.18
books
34.9%
3,56
35.10
34.41
3.56
3.56
3.52
dolms_pesIo
PPL
17.45
17.37
17.12
17.44
dolma_reddit
Loss
PPL.
3.44
31.13
3.44
31,22
3.43
30.883
3:44
511 $1.28
3.41
30.31
V3 Validation Sets
dolma wiki
dolma_stack
PPL
PPL
Loss
Toss
4.13
3.04
20.89
1.42
4109
20.78
3.03
1.41
4,07
20.61
3,03
1.40
4.13
20,88
5.04
1.42
20.10
4.02
1.39
3.00
3.16
3.70
PPL
40.35
40.50
39.99
40.56
39.17
pile
Loss 2.74
PP1
15:44
2.73
15.39
2.72
15.11
2.74
15.50
2,70
14,87
wikitest_IBB 1IB
PPL 28.21
Loss 3.34
3.34
28,12
3,31
27.46
28.36
3.25
3.2
27.12
Average
3.07
24.28
3.06
23.94
3.05
3.07
3.03
piua
hellaswag
33.96
winogrande
ACC
52.80
51.62
12
51.07
openbook_qp
ACC
28.00
23.00
28.80
28.60
28.40
seig arv, casy
ACC
70.30
7090
70.40)
70.20
70.30
Dowzstream Benchmarks
ACC
45.44
48.60
47.02
44.91
43.95
copa
ACC
62.00
07.00
67.00
63.00
69100
ric
ACC
51.26
$1.99
54.87
$4.51
$487
commitment bunk
ACC
42.86
32.14
41.07
37.50
39.24
mape
ACC
81.01
50.92
Average
52.86
28

---

Transformer
FANformer
1
0.9
0.8
0.7
0.5
0.3
0.1
Training Accuracy
Test Accuracy
Training Accuracy
Test Accuracy
Modular Addition
Linear Regression
Figure 10: Training accuracy and test accuracy of FANformer and Transformer on modular addition and linear regression tasks.
2.75
vareer
225
3.7
1.25
0.8
125
6.5
2500
3000
7500
17000
32303
12000
31500
2020
4200
5302
3000
23000
12000
24032
(a) Training loss on modular addition task
(b) Training loss linear regression task
Figure 11: Training loss of FAN and Transformer case-based and rule-based reasoning.
K Case for Section 4.4.4
We present case of FANformer and the baselines under logical reasoning stress-test in Figure 13
L Formal Proof of ATF (X)
=Attention (FANLayerFF (X))
Proposition 1. For an input sequence representation X E Bx the ATF operator satisfies
ATF(X) Attention FANLayer (X)).
Proof. By the definition of ATF via Eq. we have
QrKKp VF. Vdk
ATF(X) softmax
Substituting QF, KF, VF from Eq.
2 into Eq. 33) yields
(X W Q (XXAxR)) (X#Wr)
ATF X) = softmax
Vdg
29

---

Tansfarret
Tuar Acor
6.
Trxefumer
6.3
100
200
200
500
100
200
200
802
500
Epach
Sposh
(a) Training accuracy on modular addition task
(b) Test accuracy modular addition task
Turnlorres
FA TA
500
Fpach
Epoch
(c) Training accuracy on linear regression task
(d) Test accuracy of linear regression task
Figure 12: Accuracy of FAN and Transformer during training and testing on case-based and rule- based reasoning.
Matching standard attention, for any input Z, multi-head attention (single head shown for clarity) is defined as
ZW (Wo(ZWk) WK)
Attention(2 ( softmax
(ZWv)
Vize
Thus, we have
ATF(X) Attention(X (X
(12)
Finally, substituting X5 = FANLayer (X) via Eq. I into the above formula establishes the claim:
ATF(X) = Attention(FANLayy (X)).
30

---

Case Study
True False? Please predict whether the input rule very likely to be true, and also explain why. Please note that the rule need not necessanlly but just very likely to be true.
Examples:
Input: If Show Y was produced at Time Period Z1, Person X died at Time Period 22, and Time Period Z1 earlier (han Time Perlod Z2, then Person X had no chance to access Show Y.
Output: False. Because Show Y was available before Person X died.
Input: Person X lives Region Z and Animal Y inhabits the same Region Z, then Person X sccess Animal
Output: True. Because person and animal exist in the same region.
Original Conclusion
Flipped Conclusion
Input: If Organization X conducts an activity during Time Period 21 and Technology Y was invented during Time Period Z2, and Time Period Z2 earlier than Time Period Z1, then Organization X can access Technology Y.
Input: If Organization X conducts an activity during Time Period Z1 and Technology Y was invented during Time Period Z2, and Time Period 22 earlier than Time Period 21, then Organization X cannot access Technology Y.
OLMo-1B
OLMo-1B
Output: True. Because Organization X can access Technology Y.
Output: True. Because Organization X cannot access Technology Y.
Qwen2.5.11
@wen2.5-1155
True. Because person and animal exist the same region.
Output:
Output: True. Because and animal exist in the same region.
FANformer
FANformer
True. Because technology was invented before the activity was conducted.
Output:
Output: False, Because Technology Y was invented before Organization X conducts the activity.
Figure 13: Case of FANformer and the baselines under logical reasoning stress-test.
M
Comprehensive Experimental Details
M.1 Detailed training settings of FANformer
We train FANformer- B using the ZeRO optimizer strategy (Rajbhandari et al. 2020) via PyTorch's ' DDP framework Li 2018]. Following OLMo [Groeneveld et al. [2024], we use a constant global batch size of approximately 4M tokens (2048 instances, each with sequence length of 2048 tokens). To improve throughput, we employ PyTorch's s amp module with the bfloat16 format. We employ the Adam W optimizer [Loshchilov and Hutter 2019 for the model's 's training process. The learning rate for all LLMs is set to 4.0e-4. We warm up the learning rate over 2000 steps 8B tokens) and then decay in a cosine manner from there down
tenth of the peak learning rate over the remainder of training. We employ Flash Attention [Dao et al.) [022)] to accelerate the model training and inference processes, leveraging its ability optimize memory usage and computational efficiency. The total GPU computational cost for pre-training FANformer-1 B amounts to approximately 47, 47,600 GPU hours.
M.2 Detailed Setup for Section 4.2
For different model sizes Figure the hidden dimension, number of layers, and number of heads are listed in Table I0
M.3 Validation Set And Downstream Tasks
Following Zhu et al. 2024 we use V2 Validation Sets, V3 Validation Sets, and Downstream tasks to evaluate our approach. The specific tasks included V2 validation sets, V3 validation sets, and downstream tasks are listed in [a
31

---

Table 10: Model size and setup used for FANformer in Section 4.2 where Transformers follows the
setups of previous work OLMo [Groeneveld et al., 2024
Model
Size
Hidden Dim.
Num Layers
Num Heads
Weight Tying
FANformer-3000
268M
1024
16
16
True
FANformer-600M.
604M
1536
16
16
True
FANformer- IB
1.1B
2048
16
16
True
FA Aformer-3B
2.6B
2560
24
20
False
FANformer-7B
6.7B
4096
24
32
False
M.4 Detailed Setup of Case-based and Rule-based Reasoning.
Following the work (Hu al. 2024 focus on binary operations that take two numbers, a and b, as inputs. Denoting as the target label, the constructed datasets are in the form of D = {(a,, b;), c:) for two mathematical tasks: modular addition and linear regression. The tasks are defined as follows:
Modular addition. The input to the model is "a + =", and the output is c, where mod P. The values of and range from 0 to 112. The constant 113
c c= (a bb b) here.
Linear regression. This task involves the model learning linear regression function. The input given by "(a, b) =", and the output c, where m + n + p. The values of a and b range from 0 to 99. The constants are set as m = 1, 2, and = 3.
Luave-Square-Out The work [Hu et al. 2024 employs the Leave- Suuee-Squree-utt method to evaluate the generalization ability of the Transformer [Dong et al.) 2024c]. In this approach, square test set is created to isolate the test samples from the training samples. For instance, consider the center of the square at (ak, by) with a side length of 4g- The square test set is defined as 4 S a, S az + 2. bk lt < S by + ), and all remaining samples from the training set. This division creates a "hole" in the center the training set, which is more challenging for the model compared a random split. Since there are no similar cases in the training set to aid the model in solving the problem, this method tests whether the model has truly learned the underlying rules. In the experiments of the work (Hu al. ]024], they found that Transformer-basee models fail to generate correct answers for the test set in the "hole". Therefore, we use this method to assess the generalization ability of FANformer.
Th = [((a, b,), aş
Settings We finetune both the Transformer and FANformer models each dataset for 500 epochs. The batch size is set to 336, and the learning rate is initialized at 10 10-4, warm-up ratio 0.01 is used, and we apply cosine decay adjust the learning rate throughout the training process.
During generation, we the model temperature 0.5 and sample 10 generations evaluate the accuracy on each test point. The square center (ax, by) is (50, 50) for linear regression and (56, 56) for modular addition.
Following the work [Hu et al. 2024]. we apply the Leave-Square-Our method to each dataset. Specifically, extract square comprising 441 samples (from a total of approximately 10,000 samples) with a side length of 20 to form our test set, leaving the remainder as the training set. is important note that, despite removing small portion of training samples, we ensure that all tokens present in the dataset appear in the training set. This precaution is to prevent the models from failing simply due encountering unseen tokens. We then proceed to finetune Transformer and FANformer models using this specific training-tes test split for each dataset.
M.5 Assessing LLMs Proficiency in Capturing Inferential Rules
Analysis Setup Following experimental setup proposed by Wang et al. (2024), we adopt the ULogic framework to systematically evaluate LLMs their ability to capture underlying inferential logic. Specifically, we leverage a curated probing subset comprising 1, 1,104 diverse rules drawn from their rule base. These diles-manually verified by the original authors sithors-span span range of lengths,
32

---

Table 11; Validation Set And Downstream Tasks.
V2 Validation Sets
2-smaal-4chan-validane
v2- v2-small-c4_ 1.00_ddmains-vvlidatton
22-small-c4_co-vvlidatiin
2-ssaall-gabb-aaiidatton
12-small-ice-validattin
v2- small-m2222 Saaallmm22__ikivvaliddee
wiki-validation
v2 -mall-lllmae wwwlll--mnosspspaalllleel validation
v2- 2-smmlll-me en-validation
v2-small-pilee -2-small-pile-validation
v2-
103-validation
V3 Validation Sets
v3-small-c4_ -ecll--A__ee-lliaattinnn
v3
bookk-validation
v3 ----mmll--olmme common crawl-validation
v3-small-dolma_peet wwwe--eeeleaaaiiioone
v3 www.allloolmasstae---aaaaaa vallidttion
v3-small-ice-validatio
33-small-2222_$22oevvviiiii
V3-small-pile--alidation
v3-small-wikitex___// hatalll-iikieex_(O--vviiidddtdii 103-validatio
Downstream Benchmarks
piqa Bisk et al. 2020
hellaswag Zellers et al. 2019
winogrande (Sakauuchi et al.
2020
openbook qa Mihaylov al.
sciq Welbl et al. 2017
arc_easy Clark et al. 2018
copa [Roemmele et al. [201]
commitment, bank DD Marneffe al.
2019
mrpe
Dolan and Brockett
2005
rtc Dagan et al. 2005
sst2
Socher et al.
2013
polarities, and structural patterns, ensuring broad coverage and high quality. The evaluation framed as binary entailment classification task, where the model must determine whether given rule expresses valid logical entailment. We employ a two-shot Chain-of- Chain-of-Thought (CoT) prompting method [Wei et al. 2022], in which each input includes one correct and one incorrect example to minimize label bias. The model is prompted not only to make a binary judgment but also to justify its reasoning, with appended instruction such as "and also explain why."
To further enhance the reliability of the evaluation, we incorporate the Law of Non- Contradiction [Priest et al., 2006), which posits that statements of the form "If X, then Y" and "If X, then not Y" cannot simultaneously be true. Accordingly, for each original rule, we construct flipped version by negating its conclusion. A rule is considered correctly classified only if the model affirms the original rule as true and rejects the flipped version as false, as illustrated below. evaluate the base model of FANformer- 1B. BB OLMo-1BB and Qwen2.5-1.5B. as well as GPT-4 on the two most challenging levels ULogic for stress-testing (i.e., Length and Length 4).
If Premise, then Conclusion Conclasionnooiginaal. If Premise, then Conclusion_ ffippeddd
True
False
33

---

N Limitations
Our work has several limitations, which aim to address in our future work:
First, due to constraints in computational resources, we only pretrain the FANformer- IB on trillion tokens. However, our experimental results regarding FANformer's scalability indicate that FANformer demonstrates favorable scaling behavior during training, suggesting that increasing the model size and training tokens could lead to more significant performance improvements. To explore this further, we plan to seek additional computational resources to train larger-scale language models.
Second, work is orthogonal to the existing approaches for revising the attention mechanism. i.c.. our work can seamlessly incorporate them, as verified the derivation provided in Appendix L There are numerous variants of attention mechanisms, as discussed in the related work (Section 5) such as Flash Attention [Dao et al., 2022], MQA [Shazeer] 2019). and MLA [DeepSeek-Al et al. 2024b]. In this work, only incorporate Flash Attention for necessary acceleration, while leaving the exploration of other approaches for future work.
Third, although we have observed that enhancing the ability of language models to model periodic patterns can improve language modeling performance, the underlying mechanisms responsible for this improvement remain underexplored. To the best of our knowledge, it has hardly been studied the role of periodicity or the potential periodic behaviors LLMs on language modeling. Therefore, in future work, we will conduct a more comprehensive investigation into the fundamental mechanisms of periodicity in language modeling.
34


Reviews:
[{"review_id": "iPPy1WNOlj", "reviewer": "Reviewer_txRr", "summary": "The paper's primary contribution is introducing periodicity as a mechanism to enhance language modeling and reasoning capabilities in large language models. The authors provide both theoretical foundations and practical implementations for this approach.", "questions": "--How to detech if a setting suits to use periodicity modeling? Is there a preliminary step to determine whether to introducting a FANformer vs a regular transformer?\n\n----While the study demonstrates that enhancing a language model's ability to model periodic patterns improves performance, the underlying mechanisms responsible for this improvement remain mysterious. Why there is naturally a periodic structure in reasoning and generalization?\n\n--Limited testing scope - the approach has only been tested on a few datasets.\n\n--The scope of the claims may be limited by the experimental setup. It remains unclear how well this new method generalizes to other settings.\n\n--Results may depend on implicit assumptions that need to be articulated.", "limitations": "--Computational Resource Constraints: Due to limits in computational resources, the researchers were only able to pretrain FANformer-1B model. The result would be more convincing if it is observed on larger models. \n\n--FANformer architecture is designed to be orthogonal to other existing approaches for revising the attention mechanism, meaning it can seamlessly incorporate them. However, in this work, only Flash Attention was incorporated for necessary acceleration. The exploration and integration of other attention variants, are left for future research.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Contribution and strengths:\n\n--Incorporating Fourier transforms into Transformer architectures—and potentially future LLMs—is a challenging problem. On the other hand, it is well known that such integration could be a breakthrough for handling data with inherent periodicity. This paper explores new model architectures to address this challenge and demonstrates strong empirical performance.\"\n\n--The paper includes a wide range of downstream tasks, which makes the proposed architecture particularly appealing.\n\n--The authors provide detailed descriptions of their model architectures, hyperparameters, and training procedures, ensuring reproducibility of their experimental results.\n\n--The paper also offers theoretical explanations for why periodicity facilitates language modeling and reasoning, supported by formal proofs of their methodological claims, and computational cost analysis.\n\n\n\nWeaknesses:\n--Computational Resource Constraints: Due to limits in computational resources, the researchers were only able to pretrain FANformer-1B model. The result would be more convincing if it is observed on larger models. \n\n--FANformer architecture is designed to be orthogonal to other existing approaches for revising the attention mechanism, meaning it can seamlessly incorporate them. However, in this work, only Flash Attention was incorporated for necessary acceleration. The exploration and integration of other attention variants, are left for future research.", "was_revised": true, "final_justification": "I wish to maintain my ratings.", "rebuttal": "Thank you for your in-depth review and valuable feedback. We will respond to your comment point-by-point below.\n\n> How to detech if a setting suits to use periodicity modeling? Is there a preliminary step to determine whether to introducting a FANformer vs a regular transformer?\n\nThere is currently no precise method to determine if a setting is suitable for using periodicity modeling. We recommend considering FANformer as a more powerful default option to use.  Because various forms of real-world data contain either explicit or implicit periodic patterns, and we believe that these tasks will benefit from FANformer. When a task is completely aperiodic, FANformer essentially degenerates to a standard Transformer. To date, we have not yet identified specific scenarios where it performs poorly. However, the boundaries of FANformer's generalizability remain unknown, and we leave this exploration for future work.\n\n\n> While the study demonstrates that enhancing a language model's ability to model periodic patterns improves performance, the underlying mechanisms responsible for this improvement remain mysterious. Why there is naturally a periodic structure in reasoning and generalization?\n\nThanks for your question. As described in Limitation Section, \"although we have observed that enhancing the ability of language models to model periodic patterns can improve language modeling performance, the underlying mechanisms responsible for this improvement remain underexplored. To the best of our knowledge, it has hardly been studied the role of periodicity or the potential periodic behaviors of LLMs on language modeling. Therefore, in future work, we will conduct a more comprehensive investigation into the fundamental mechanisms of periodicity in language modeling.\" We will try to explain this question from two perspectives:\n\nFirst, based on the strict definition of periodicity, we will give some intuitive observations and explanations.\n\n1. **The essence of periodicity lies in the repetitive manifestation of certain invariance under transformations**, which can be strictly defined through invariance under group actions in Abstract Algebra. *(Let $X$ be a set and $G$ be a group acting on $X$. An element $x \\in X$ is said to be periodic with respect to the action of $G$ if there exists a non-identity element $p \\in G$ such that $p \\cdot x = x$, where $\\cdot$ denotes the action of the group $G$ on the set $X$. The element $p$ is called a period of $x$ under the action of $G$. Periodicity manifests as the invariance of $x$ under all elements of the cyclic subgroup generated by $p$, denoted by $\\langle p \\rangle$. If the group operation in $G$ is written multiplicatively, then $\\langle p \\rangle =$ {$p^n \\mid n \\in \\mathbb{Z}$}. For every element $g \\in \\langle p \\rangle$, we have $g \\cdot x = x$.)* For example, $f(a) = f(a+T)$ can be seen as a specific instance of the abstract definition $p \\cdot x = x$, where $x=f$, $p=T$, and the group action is translation. When the input $a$ and the group $G$ are extended to higher dimensions or non-temporal domains, the manifestation of the period $T$ also changes accordingly. In simple terms, **for inputs belonging to the same domain (or having the same properties), using the same rule $f$ to calculate, its resulting output implies invariance. It's typical for most reasoning questions to take this form.** \n2. Intuitive Explanation: Assuming that certain knowledge has already been learned within the semantic feature space, the emergence of new concepts benefits from the inherent periodicity by preferentially establishing connections with existing knowledge, rather than requiring de novo learning.\n3. In Section 4.4.3, we experimentally explore that FANformer facilitates the rule-based learning paradigm of math reasoning in natural language, effectively mitigating the occurrence of \"holes\" inherent in the case-based reasoning of Transformer (Hu et al., 2024, ICML).  In Section 4.4.4, under the stress test of logical reasoning [Wang et al., 2024], FANformer-1B demonstrates superior performance compared to OLMo-1B and Qwen2.5-1.5B. \n\nIn summary, the core of periodicity lies in the recurrence of a system's certain rule under specific transformations, which is common and important in reasoning and language modeling.  \n\nSecond, for the generalization, we believe that some generalization properties can be expressed as periodicity. This is because many real-world features are inherently periodic, which makes it reasonable to generalize based on this characteristic. Furthermore, periodic extension is a valid and parsimonious assumption that complies with Occam's razor. But we must acknowledge that the extent of its generalization capability is likely highly dependent on the specific real-world task.\n\n> Limited testing scope - the approach has only been tested on a few datasets. & The scope of the claims may be limited by the experimental setup. It remains unclear how well this new method generalizes to other settings. & Results may depend on implicit assumptions that need to be articulated.\n\nThank you for your valuable questions about test scope, generalization ability, and implicit assumptions. To ensure the rigor and comprehensiveness, our evaluation process follows the widely recognized previous work, especially the evaluation framework established by the work OLMo (Outstanding Paper at ACL 2024), which ensures that our experimental settings and benchmark selection are fair and representative. Specifically, we conduct experiments on 8 core tasks of commonsense in Table 1, 35 tasks of language modeling in Table 2, and 4 tasks of instruction following in Table 4. In addition to adhering to OLMo's evaluation framework, we further validated 1) scalability of FANformer in accordance with scaling laws in Figure 3, 2) generalization capability of FANformer on case-based and rule-based reasoning tasks in Figure 5 and Figure 10, 3) logical reasoning ability of FANformer-1B under the stress test in Table 3 and Figure 13. To further validate the model's generalization and practical effectiveness, we conduct additional experiments on code generation tasks (i.e., HumanEval and MBPP) and human-written math tasks (i.e., GSM8K) compared to our baseline, OLMo,  and have added the experimental results to our revised manuscript. The results show that our FANformer achieves clear and consistent improvements compared with OLMo on all three benchmarks. \n\n| LLMs | Training Tokens | HumanEval | MBPP | GSM8K |\n|-|-|-|-|-|\n| OLMo-1B | 3T checkpoint  | 5.2     | 3.1       | 8.9        |\n| FANformer-1B  |  1T from scratch   | **6.3**     | **5.4**       | **15.7** |\n||"}, {"review_id": "Zt4CEqSnVD", "reviewer": "Reviewer_BuMP", "summary": "This paper introduces FANformer, a novel large language model (LLM) architecture that improves learning efficiency and performance by enhancing the Transformer attention mechanism with periodicity modeling via Fourier Analysis Network (FAN). The authors argue that periodic patterns, ubiquitous in human reasoning, are poorly modeled by standard Transformers. FANformer modifies the attention mechanism to operate partially in the frequency domain using sine and cosine projections, resulting in better representation of periodic features. Extensive experiments show FANformer achieves comparable or superior performance to baseline Transformers with significantly fewer parameters and training tokens. It also demonstrates better rule-based reasoning, logical inference capabilities across several downstream tasks.", "questions": "1. Explanation of Validity\nAuthors should provide more detailed explanations as to why periodic modeling can enhance the capability of reasoning and why it can improve the efficiency of learning, rather than merely illustrating through experiments. Intuitively, modeling in the frequency domain would decrease the model efficiency.\n\n2. Limited Scope of Evaluation Benchmarks\nThis paper primarily evaluates standard common-sense question-answering tasks and synthetic mathematical reasoning datasets, lacking systematic testing in more real-world task scenarios.\n\n3. Real-world Applicability:\n Have you benchmarked the inference speed and GPU memory usage of FANformer relative to Transformer in deployment settings? Would the frequency-domain calculations add significant latency?\n\n4. Generalization Beyond Commonsense Tasks:\nHave you evaluated FANformer in other structured or periodic domains (e.g., code generation) where periodicity may be even more prevalent?\n\n5. How to balance the model's capability in periodic modeling and its ability in non-periodic task modeling?", "limitations": "YES", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Innovation of Approach:\nThe use of a modified Fourier-based projection within the QKV structure of attention (the ATF module) is an original solution. By embedding periodic patterns directly into attention computation, the model diverges meaningfully from standard Transformer variants. While previous work explored Fourier Analysis Network, this paper proposes a principled and scalable adaptation.\n\nSignificance:\nThis work has high potential impact. Periodicity is underexplored in LLMs, and this approach opens a novel and elegant path for improving Transformer-based architectures, especially in domains where pattern repetition and rule-based reasoning are essential.\n\nClarity and Fluency:\nThe writing is generally clear, and the motivations are well articulated. However, a few sections (e.g., the mathematical formalism in Section 2) may be too dense for readers unfamiliar with group theory or frequency-domain analysis. Pseudocode and diagrams are helpful but could be further clarified.\n\nWeaknesses:\n1. The theoretical advantages of periodic modeling were not explained in further detail.\n2. While promising, the reliance on sinusoidal expansions could introduce constraints in non-periodic linguistic domains. Furthermore, there is insufficient analysis of the model’s behavior in handling irregular or non-periodic sequences — a potential limitation for broad NLP applicability. While generalization is claimed, more rigorous checks (e.g., across languages and tasks) would have bolstered the argument.\n3.  While periodic tasks are ideal tests for FANformer, real-world linguistic periodicity is more nuanced; extrapolation from synthetic setups (e.g., modular arithmetic) to natural language warrants caution.\n4. Lack of Cross-Domain Validation– All tasks use English and a narrow band of benchmarks. Cross-linguistic or multimodal validation is absent.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and detailed feedback. We will respond to your comment point-by-point below.\n\n> Generalization Beyond Commonsense Tasks: Have you evaluated FANformer in other structured or periodic domains (e.g., code generation) where periodicity may be even more prevalent? & Limited Scope of Evaluation Benchmarks This paper primarily evaluates standard common-sense question-answering tasks and synthetic mathematical reasoning datasets, lacking systematic testing in more real-world task scenarios.\n\nThanks for your suggestions. We conduct experiments on code generation tasks (i.e., HumanEval and MBPP) and human-written math tasks (i.e., GSM8K) compared to our baseline, OLMo (which is the Outstanding Paper at ACL 2024),  and have added the experimental results to our revised manuscript. The results show that our FANformer achieves clear and consistent improvements compared with OLMo on all three benchmarks. \n\nMoreover, we conduct systematic testing on 35 language modeling tasks, and FANformer also achieves notable and stable improvements across these tasks compared to OLMo and the variants of FANformer, as shown in Table 6 and Table 7 of Appendix. \n\n| LLMs | Training Tokens | HumanEval | MBPP | GSM8K |\n|-|-|-|-|-|\n| OLMo-1B | 3T checkpoint  | 5.2     | 3.1       | 8.9        |\n| FANformer-1B  |  1T from scratch   | **6.3**     | **5.4**       | **15.7** |\n||\n\n> Explanation of Validity Authors should provide more detailed explanations as to why periodic modeling can enhance the capability of reasoning and why it can improve the efficiency of learning, rather than merely illustrating through experiments. Intuitively, modeling in the frequency domain would decrease the model efficiency.\n\nMany real-world features are inherently periodic. For example, general reasoning can be understood as a form of periodicity under group actions as analyzed in Section 2. However, existing LLMs lack dedicated approaches to process these periodic characteristics. Our Fanformer is designed to effectively model such features. For periodic latent features, effectively modeling them improves LLM's learning efficiency, as demonstrated in our experiments.\n\n> Real-world Applicability: Have you benchmarked the inference speed and GPU memory usage of FANformer relative to Transformer in deployment settings? Would the frequency-domain calculations add significant latency?\n\nThanks for your questions. We conduct experiments on the inference speed and GPU memory usage of FANformer relative to Transformer in deployment settings, and have added the experimental results to our revised manuscript. The results show that it adds little latency. \n\nThe configuration of the benchmark test: we run for 20 iterations on a single GPU of A100 80G with a fixed sequence length of 4096 tokens and float16 precision.\n\n| Metric              | OLMo-1B     | FANformer-1B | Difference               |\n|---------------------|-------------|--------------|---------------------------|\n| Forward Pass Time   | 141.49 ms   | 142.88 ms    | +1.39 ms (+0.98%)        |\n| Allocated Memory    | 4642.69 MB  | 4738.86 MB   | +96.17 MB (+2.1%)         |\n| Peak Memory         | 6610.70 MB  | 6706.88 MB   | +96.18 MB (+1.5%)        |\n||\n\n\n> How to balance the model's capability in periodic modeling and its ability in non-periodic task modeling?\n\nThe periodicity ratio hyperparameter can be used to balance the model's capability in periodic modeling and non-periodic task modeling. In this paper, we fixed the periodicity ratio to 1/4 and used the same setting in all experiments, and achieved stable and clear improvements. The experimental results in Figure 8 and Figure 9 of Appendix show that, regardless of the variation in p values, 1) FANformer exhibits strong robustness in terms of training loss and downstream task accuracy, with relatively small performance fluctuations; 2) FANformer consistently outperforms the baselines. These findings indicate that FANformer maintains good robustness with respect to the hyperparameter p."}, {"review_id": "gPqe4ewQKm", "reviewer": "Reviewer_GTjC", "summary": "This paper introduces a new method that integrates the Fourier basis into the attention module, primarily motivated by the periodicity in human data (as they claim). They demonstrated that transformers with this periodicity component outperform alternative transformers that lack this architecture. They did ablation studies to show that the benefit primarily stems from the periodicity component. They conducted extensive experiments comparing their method to existing transformers on a range of datasets, including those that incorporate reuse. They showed that FANformer scales up, performs okay, and takes fewer parameters.", "questions": "From Section 4.4.3, it appears that FAN introduces more holes in unwanted regions around 99/98 in the linear regression task (figure 5 right most plot). Does FANFormer introduce new problems that transformers do not have?\n\nThere are a small number of holes in the modular addition task, but why does the transformer have holes shaped like a square? \n\nEquation 1: what is p with an overline and p without an overline? \n\nHow is the tokenization done? Why does FANformer need a smaller number of training tokens? \n\nThe authors observe that effective periodicity modeling improves the learning efficiencies of LLMs. Does it induce loss in some other aspect, mainly when knowledge is not described in a periodic form? \n\nHow are the Lipschitz constants evaluated? \n\nLine 27:\n“Transformer-based models are also known for their immense demand for data and computational resources during training [Kaplan et al., 2020; Hoffmann et al., 2022; Chowdhery et al., 2023]. In comparison, humans can accomplish similar learning tasks with far fewer resources. This discrepancy suggests that existing LLM architectures still suffer from low learning efficiency.”\nThis is a valid observation, but it remains unclear whether the proposed FAN model addresses this gap in learning efficiency. The connection between periodicity and improved learning efficiency is not substantiated in the paper.", "limitations": "The authors included limitations, but a discussion with other large models that take periodicity into account is currently lacking.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "__Strengths__ \n\nThe paper's strength lies in its extensive experiments and evaluations on a wide range of datasets, as well as investigations into the influence of hyperparameters on the loss. The organization and structure of the paper is also very clear. \n\n__Weaknesses__ \n\nWhat is currently lacking is a comparison with alternative models that also employ periodicity in their architecture, which have also claimed that these models perform better than transformers, as it is not the first paper to find ways to cook periodicity into its architecture. For example, how does Fanformer compare to state space models (SSMs), such as mamba, with recurrence that naturally support periodic dynamics?\n\nI found the motivation for introducing periodicity rather unconvincing.\nThe paper claims that periodicity facilitates structured knowledge acquisition. However, this is conceptually vague and poorly grounded. Knowledge, particularly in domains like language, does not naturally exhibit periodic structure. While one might metaphorically describe some mental representations as \"wave-like,\" structured knowledge is not inherently represented as waveforms.\n\nLine 31:\nThe authors cite Lake et al. and Buzsáki (2006) as part of the motivation for periodicity. However, these references tangentially relate to the claim. Lake et al. emphasize human learning efficiency through intuitive theories and compositional representations, periodicity is never mentioned. Buzsáki’s work hypothesizes that rhythmic activity underlies some brain functions, but this remains speculative. Furthermore, most supporting evidence pertains to animal studies and does not directly suggest that periodicity is a principle governing human learning processes, especially not in high-level domains like language.\n\nMoreover, pattern recognition—which is a plausible explanation for human generalization—is not equivalent to with periodicity.\nLastly, it remains completely unclear how periodicity should manifest in language modeling. Mathematical functions like x mod 5 may be periodic, but this kind of structure is far removed from the kind of compositional and hierarchical patterns present in natural language.", "was_revised": true, "final_justification": "This paper proposes a novel architecture and shows that in some domains, the new architecture performs better than the transformer-based architecture in a number of evaluation experiments. Although the motivations for periodicity is a bit of a stretch, the new architecture would be a contribution to the community and I would recommend an acceptance.", "rebuttal": "Thank you for your thorough review and detailed feedback. We will respond to your comment point-by-point below.\n\n> How does Fanformer compare to state space models (SSMs), such as mamba, with recurrence that naturally support periodic dynamics?\n\nFirst, our approach is fundamentally different from SSMs. SSMs model periodicity along the sequence dimension, while our FANformer models it along the feature dimension. Second, our motivation is distinct from that of Mamba. Mamba is primarily developed to overcome the quadratic computational complexity of Transformers and improve inference efficiency, while our approach is designed to improve learning efficiency and performance of Transformers through effective periodicity modeling.\n\n\n> From Section 4.4.3, it appears that FAN introduces more holes in unwanted regions around 99/98 in the linear regression task (figure 5 right most plot). Does FANFormer introduce new problems that transformers do not have?\n\nSorry for the confusion, we find that although the test metric on the training set was close to 1, there is still a small amount of data that was not fitted when displayed. We continue to extend the training and find that this problem can be effectively solved. The final conclusion does not change. Transformer still shows overfitting on the training set, while FANformer facilitates the rule-based reasoning paradigm, mitigating the occurrence of \"holes\" inherent in the case-based learning of Transformer. Moreover, we have not yet observed specific scenarios where FANformer performs poorly. We leave this for our future work.\n\n> There are a small number of holes in the modular addition task, but why does the transformer have holes shaped like a square?\n\nThe square shape of the hole in Transformer is due to the experimental setting. Following work [1], we randomly dug out the middle square part in the training set and used it as the test set, and the rest was used as the training set.\n\nReference:  \n[1] Case-based or rule-based: How do transformers do the math?  ICML 2024\n\n> Equation 1: what is p with an overline and p without an overline?\n\nThe input X is multiplied by a weight matrix W, which is conceptually split into two parts: W_p and W_p̄. The part of the output corresponding to W_p (i.e., W_p X) is fed into both cos and sin functions to create periodic features, while W_p̄ undergoes a linear transformation. The overline notation signifies the remaining part of the feature transformation.\n\n\n> How is the tokenization done? Why does FANformer need a smaller number of training tokens?\n\nTokenization is performed using a standard LLM tokenizer. We have not made any special modifications or optimizations to this process. In the standard pre-training process for LLMs, a massive text corpus is segmented into sequences of tokens, which are then fed into the model for training. The total number of tokens processed by the model is the precise measure of the training data used. In our comparative experiments (Figure 1, right), we observed that FANformer uses significantly fewer total training tokens to reach the same performance level as the traditional Transformer baseline. Therefore, our statement that FANformer \"requires fewer training tokens\" is a direct description of this experimental result. It demonstrates that FANformer has higher learning and data efficiency, achieving comparable or even better results with less training data. This proves the advantage of our new architecture in improving learning efficiency.\n\n\n> The authors observe that effective periodicity modeling improves the learning efficiencies of LLMs. Does it induce loss in some other aspect, mainly when knowledge is not described in a periodic form?\n\nThis is a good question! Our current results suggest that introducing periodic modeling improves performance compared to existing models, but whether this will lead to performance loss is unknown. However, we have not yet observed specific scenarios where FANformer performs poorly. From our perspective, we retain the existing modeling approach while also adding frequency-domain modeling. This allows for some signals to be converted to the frequency domain. We believe this approach will not result in excessive information loss, but will compensate to some extent for the lack of periodic signals. \n\n> How are the Lipschitz constants evaluated?\n\nThe Lipschitz constant L is defined as $\\forall x, y \\in \\mathbb{R}^n, \\quad \\lVert f_{\\text{model}}(x) - f_{\\text{model}}(y) \\rVert \\leq L \\lVert x - y \\rVert$. We approximate L using a small perturbation $\\epsilon=1e-7$, computing $\\lVert f_{\\text{model}}(x)-f_{\\text{model}}(x+\\epsilon) \\rVert$. We perform 10 calculations and take the average to approximate L. The reproducible code of experiments is provided in the anonymous code project.\n\n> Line 27: “Transformer-based models are also known for their immense demand for data and computational resources during training [Kaplan et al., 2020; Hoffmann et al., 2022; Chowdhery et al., 2023]. In comparison, humans can accomplish similar learning tasks with far fewer resources. This discrepancy suggests that existing LLM architectures still suffer from low learning efficiency.” This is a valid observation, but it remains unclear whether the proposed FAN model addresses this gap in learning efficiency. The connection between periodicity and improved learning efficiency is not substantiated in the paper.\n\nThanks for your comment. We demonstrate the connection between periodicity and improved learning efficiency through the following results. FANformer effectively improves the periodicity modeling performance, and FANformer also improves learning efficiency as evidenced by: 1) FANformer-1B outperforms an open-source LLM of similar size with fewer training tokens, and even surpasses an LLM with three times the number of parameters using the same training tokens. 2) FANformer consistently outperforms the Transformer, achieving comparable performance with only 69.2% of the model parameters or 79.7% of the training tokens. 3) By observing the training process, we find that FANformer's learning efficiency significantly improves compared to the Transformer as the model continuously learns from the data.\n\nThese results, i.e, outperforming other models with fewer resources, demonstrating superior scaling-law performance, and achieving faster convergence, collectively confirm that FANformer's effective periodicity modeling is the key mechanism driving these measurable improvements in learning efficiency."}, {"review_id": "jYU3aKnlDh", "reviewer": "Reviewer_CN5c", "summary": "The paper introduces a periodic component into transformer attention layer. The change is inspired by recent Fourier based network development as well as backed by the argument of periodicity importance (motivation section 2). After describing the network changes, the authors test the architecture on LM benchmarks against the natural baselines with controlled variations (4.4.1) as well as language models with similar size and capacities. The model achieves clear superior performance compared to baselines on common sense reasoning, language modelling as well as controlled math tasks and logical reasoning. \n\nThe paper tries to find elements that can explain the improvements: the paper shows the advantage of learning curve as well as the scaling properties. Finally, the paper also shows the model expressiveness advantage measured by Lipschitz measure.", "questions": "1. Do authors have tested different p value for the architecture. when p = 0 it goes to a linear layer and one can also only have fourier layers so it seems interesting to see what this parameter balances in terms of downstream performance as well as characteristics.\n\n2. Do authors test on other benchmarks? As authors have mentioned, the improving mechanism is unknown so it is possible that other domains also simply improve and degradations are very useful to get noticed.\n\n3. It is surprised to see that the model only gets pretrained but the performance exceeds some models that are fine-tuned or distilled. Do authors have comments on this please?", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength: Despite that the change is not straightforward, the paper is easy to follow where authors motivate the approach and explain clearly the changes that are brought to the existing architecture. The paper not only mentions the relevant work in the fourier network field but also connects with other attention improvement where the paper discusses the complementary roles of the current approach. The paper has extensive evaluations well organised. In terms of performance, the proposed model is evaluated against its natural baselines as well as compared with state-of-the-art models of similar size and show convincing improvements. Beyond that, the paper also makes a step to try to investigate where comes the improvement. Although the exact mechanism is still very difficult to find, the paper investigates into training curve, controlled task, scalability as well as representational power, laying a solid ground for future research to be built on top of the existing work. The work makes significant improvement on the seemingly well matured transformer architecture, so the contribution I judge to be significant to the field as well. \n\nWeakness: There are some natural questions that I have when reading the paper. The paper can be even more extensive although it is definitely good on the thoroughness for the experiments.", "was_revised": true, "final_justification": "I would continue to recommend the paper as \"strongly accept\". \n\nWhile I agree with the reviewers and authors that there is no clear motivation why periodicity modelling brings performance. I think the phenomenon and the modelling itself is exciting enough for the community given how popular transformer architecture is. It shows a modelling area that brings further fruits in a currently thought \"mature\" technology and architecture which bears high impact.", "rebuttal": "Thank you for your in-depth review and valuable feedback. We will respond to your comment point-by-point below.\n\n> Do authors have tested different p value for the architecture. when p = 0 it goes to a linear layer and one can also only have fourier layers so it seems interesting to see what this parameter balances in terms of downstream performance as well as characteristics.\n\nYes, we present results for different p-values in Appendix C. But we fixed p = 0.25 and used the same setting in our experiments by default, which achieves stable and clear improvements. The experimental results in Figure 8 and Figure 9 of Appendix C show that, regardless of the variation in p values, 1) FANformer exhibits strong robustness in terms of training loss and downstream task accuracy, with relatively small performance fluctuations; 2) FANformer consistently outperforms the baselines. These findings indicate that FANformer maintains good robustness with respect to the hyperparameter p.  \n\nAs you point out, p=0 is an interesting baseline comparison, and the model degenerates to a purely linear layer at this value. The results for p=0, shown in Table 2 (i.e., the baseline ATL mentioned in this paper), perform significantly worse than FANformer.\n\n> Do authors test on other benchmarks? As authors have mentioned, the improving mechanism is unknown so it is possible that other domains also simply improve and degradations are very useful to get noticed.\n\nThanks for your questions. We conduct additional experiments on code generation tasks (i.e., HumanEval and MBPP) and human-written math tasks (i.e., GSM8K) compared to our baseline, OLMo (which is the Outstanding Paper at ACL 2024),  and have added the experimental results to our revised manuscript. The results show that our FANformer achieves clear and consistent improvements compared with OLMo on all three benchmarks. \n\nMoreover, we conduct systematic testing on 35 language modeling tasks, and FANformer also achieves notable and stable improvements across these tasks compared to OLMo and the variants of FANformer, as shown in Table 6 and Table 7 of Appendix. \n\n| LLMs | Training Tokens | HumanEval | MBPP | GSM8K |\n|-|-|-|-|-|\n| OLMo-1B | 3T checkpoint  | 5.2     | 3.1       | 8.9        |\n| FANformer-1B  |  1T from scratch   | **6.3**     | **5.4**       | **15.7** |\n||\n\n\n> It is surprised to see that the model only gets pretrained but the performance exceeds some models that are fine-tuned or distilled. Do authors have comments on this please?\n\nThat's a great point. While techniques like fine-tuning and distillation are powerful for adapting a model to a specific task, they often come with the risk of the phenomenon called catastrophic forgetting or knowledge degradation, i.e, the model may lose some of the general capabilities acquired during pre-training. This underscores the critical importance of research on fundamental model architecture to improve learning efficiency."}]

Author rebuttal:

Meta-review: This paper introduces a Transformer variant, FANformer, that integrates Fourier-based periodicity modeling into attention. Empirical studies  show that FANformer achieves comparable performance (both on pretraining metrics as well as on some downstream tasks) with fewer parameters or fewer training tokens than baseline Transformers. As motivation it is claimed that standard transformers poorly capture periodic patterns in reasoning and language modeling, however, the sinusoidal transformation is applied on feature dimension and not along token axis which makes the argument unclear. Nevertheless, all the reviewers leaned positive, citing novelty, clarity, and strong empirical evidence. Main concerns focused on limited scale (experiments at ~1B parameters), generalization to broader, and motivation/explanation of why adding the sinusoidal terms is a good idea. We thank the reviewers and authors for engaging during the rebuttal period to improve the paper. The rebuttal provided additional evaluations (HumanEval, MBPP, GSM8K) and clarifications that strengthened the paper. The two reviewers (BuMP and GTjC) with borderline score also explicitly recommend accepting this paper in final response. Overall, despite unclear motivation, the work demonstrates meaningful improvements over mature Transformer baselines and thus should be accepted.
Decision: Accept (poster)