Pre-trained Large Language Models Use Fourier Features to Compute Addition
Authors: Tianyi Zhou
Deqing Fu
Vatsal Sharan
Robin Jia
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
arithmetic task
Fourier analysis
Primary area: interpretability_and_explainability

Abstract: Pre-trained large language models (LLMs) exhibit impressive mathematical reasoning capabilities, yet how they compute basic arithmetic, such as addition, remains unclear. 
This paper shows that pre-trained LLMs add numbers using Fourier features---dimensions in the hidden state that represent numbers via a set of features sparse in the frequency domain. 
Within the model, MLP and attention layers use Fourier features in complementary ways: MLP layers primarily approximate the magnitude of the answer using low-frequency features, while attention layers primarily perform modular addition (e.g., computing whether the answer is even or odd) using high-frequency features.
Pre-training is crucial for this mechanism: models trained from scratch to add numbers only exploit low-frequency features, leading to lower accuracy.
Introducing pre-trained token embeddings to a randomly initialized model rescues its performance.
Overall, our analysis demonstrates that appropriate pre-trained representations (e.g., Fourier features) can unlock the ability of Transformers to learn precise mechanisms for algorithmic tasks.

Pre-trained Large Language Models Use Fourier Features to Compute Addition
Tianyi Zhou Deqing Fu Vatsal Sharan Robin Jia
Department of Computer Science University of Southern California
Los Angeles, CA 90089
(echou029,deqineee weeaingnu.aaaaaannn.....o vsharan,robinn Sareiiaauuuuu... @usc.
Abstract
Pre-trained large language models (LLMs) exhibit impressive mathematical reason- ing capabilities, yet how they compute basic arithmetic, such as addition, remains unclear. This paper shows that pre-trained LLMs add numbers using Fourier features-dimensiin dimensions in the hidden state that represent numbers via set of fea- tures sparse in the frequency domain. Within the model, MLP and attention layers use Fourier features in complementary ways: MLP layers primarily approximate the magnitude of the answer using low-frequency features, while attention layers primarily perform modular addition (e.g., computing whether the answer is even or odd) using high-frequency features. Pre-training is crucial for this mechanism: models trained from scratch to add numbers only exploit low-frequenc, features, leading to lower accuracy. Introducing pre-trained token embeddings to randomly initialized model rescues its performance. Overall, our analysis demonstrates that appropriate pre-trained representations (e.g., Fourier features) can unlock the ability of Transformers to learn precise mechanisms for algorithmic tasks.
Introduction
Mathematical problem solving has become crucial task for evaluating the reasoning capabilities of large language models (LLMs) [20, 7, 23, 13]. While LLMs exhibit impressive mathematical abilities [34, 17, 1, 45, 40, 4, 11], remains unclear how they perform even basic mathematical tasks. Do LLMs apply mathematical principles when solving math problems, or do they merely reproduce memorized patterns from the training data?
In this work, we unravel how pre-trained language models solve simple mathematical problems such as "Put together 15 and 93. Answer: ". Prior work has studied how Transformers, the underlying architecture LLMs, perform certain mathematical tasks. Most studies [5, 14, 42, 2, 12, 28, 18, 36] focus on Transformers with limited number of layers those trained from scratch; [19] analyzes how the pre-trained GPT-2-small performs the greater-than task. Our work focuses on a different task from prior interpretability work-enteger addition and shows that pre- pre-trained LLMs learn distinct mechanisms from randomly initialized Transformers.
In $3, we show that pre-trainn trained language models compute addition with Fourier features dimensions in the hidden state that represent numbers via a set of features sparse the frequency domain. First, we analyze the behavior of pre-trained I.LMs on the addition task after fine-tuning, which leads to almost perfect accuracy on the task. Rather than merely memorizing answers from the training data, the models progressively compute the final answer layer by layer. Next, we analyze the contributions of individual model components using Logit Lens [3]. We observe that some components primarily approximate the answer-they promote all numbers close to the correct answer in magnitude- while other components primarily classify the answer modulo m for various numbers m. Then,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

we use Fourier analysis to isolate features in the residual stream responsible for the low- low-frequeney "approximation" and -eee-freuenccy "classification" subtasks. Identifying these features allows us to precisely ablate the ability of the model to perform either approximation or classification by applying a low-pass or high-pass pass filter, respectively, to the outputs of different model components. We find that MLP layers contribute primarily to approximation, whereas attention layers contribute primarily to classification.
In $4, we show that pre-training is crucial for learning this mechanism. The same network trained from scratch with random initialization not only shows no signs of Fourier features, but also has lower accuracy. We identify pre- pre-trained token embeddings as key source of inductive bias that help the pre-trained model learn a more precise mechanism for addition. Across the pre-trained token embeddings of many different pre-trained models, Fourier analysis uncovers large magnitudes of components with periods 2, 5, and 10. Introducing pre-trained token embeddings when training the model from scratch enables the model to achieve perfect test accuracy. Finally, we show that the same Fourier feature mechanism present not only in models that were pre-trained and then fine-tuned, but also in frozen pre-trained LLMs when prompted with arithmetic problems.
Overall, our work provides mechanistic perspective on how pre-trained LLMs compute addition through the lens of Fourier analysis. It not only broadens the scope from only investigating few-layer Transformers trained fit particular data distribution to understanding LLMs as whole, but also hints at how pre-training can lead to more precise model capabilities.
2 Problem Setup
Task and Dataset. We constructed a synthetic addition dataset for fine-tuning and evaluation purposes, Each example involves adding two numbers I 260, chosen because the maximum number that can be represented by a single token in the GPT-2-XL tokenizer is 520. For each pair of numbers between and 260, we randomly sample one of five natural language question templates and combine it with the two numbers. The dataset is shuffled and then split into training (80%), validation (10%), and test (10%) sets. More details are provided in Appendix F. In Appendix C.3, we show our that results generalize to a different dataset formatted with reverse Polish notation.
Model. Unless otherwise stated, all experiments focus on the pre-trained GPT-2-XL model that has been fine-tuned on our addition dataset. This model, which consists of 48 layers and approximately 1.5 billion parameters, learns the task almost perfectly, with an accuracy of 99. 79% on the held-out test set. We examine other models in $4.2 and $4.3.
Transformers. We focus on decoder-only Transformer models [41], which process text sequentially, token by token, from left to right. Each layer in the Transformer has an attention module with output Attn (8) and an MLP module with output MLP(() Their outputs are added together to create continuous residual stream h [9], meaning that the token representation accumulates all additive updates within the residual stream, with the representation 7,00) in the b-th layer given by:
h (f-1) Attn (9) MLP(()
The output embedding W projects the residual stream to the space of the vocabulary; applying the softmax function then yields the model's prediction. We provide formal definitions in Appendix A.
3 Language Models Solve Addition with Fourier Features
In this section, we analyze the internal mechanisms of LLMs when solving addition tasks, employing a Fourier analysis framework. We first show that the model initially approximates the solution before iteratively converging to the correct answer ($3.1). We then show that the model refines its initial approximation by computing the exact answer modulo 2, 5, and 10, employing Fourier components of those same periods ($3.2). Finally, we demonstrate through targeted ablations that the identified Fourier components are causally important for the model's 's computational processes ($3.3). Specifically, we show that MIP layers primarily approximate the magnitude of the answer, using low- frequency features, while attention layers primarily perform modular addition using tig-frreueecc components.

---

Lagerasse Prodiction Acourscy
Loglts Acrees Dack MIP Lopers
Tostle Acrnes Earh ATTY Lowess
Pood muth
honge Jearety
1.0
20
0.8
200
121
10
Frediction
0.6
10
110
100
Loghs
04
02
-20
0.0
15
20
30
32
32
12
Layer
Layer Iodes
Indio
(a) Accuracy change
(b) MLP logits
(c) Attention logits
Figure 1: (a) Visualization of predictions extracted from fine-tuned GPT-2-XL at intermediate layers. Between layers 20 and 30, the model's accuracy is low, but its prediction is often within 10 of the correct answer: the model first approximates the answer, then refines it. (b) Heatmap of the logits from different MIP layers for the running example, "Put together 15 and 93. Answer: 108". The y-axis represents the subset of the number space around the correct prediction, while the æ-axis represents the layer index. The 33-rd layer performs mod 2 operations (favoring even numbers), while other layers perform other modular addition operations, such as mod 10 (45-th layer). Additionally, most layers allocate more weight to numbers closer the correct answer, 108. (c) Analogous plot for attention layers. Nearly all attention modules perform modular addition.
3.1 Behavioral Analysis
Our first goal is understand whether the model merely memorizes and recombines pieces of information learned during training, or it performs calculations to add two numbers.
Extracting intermediate predictions. To elucidate how LLMs perform computations and progres- sively refine their outputs towards the correct answer, we extract model predictions at each layer from the residual stream. Let L denote the number of layers. Using the Logit Lens method [3], instead of generating predictions by computing logits W (*.), predictions are derived through W* where é E [4]. We compute the accuracy of the prediction using each intermediate state (e) If the models merely retrieve and recombine pieces of information learned during training, certain layers will directly map this information to predictions. For instance, [25] demonstrates that there is a specific MLP module directly maps a country to its capital.
LLMs progressively compute the final answers. Figure la instead shows that the model progres- sively approaches the correct answer, layer by layer. The model capable of making predictions that fall within the range of + and +10 relative to the correct answer in the earlier layers, compared to the exact-match accuracy. This observation implies that the Transformer's layer-wise processing structure is beneficial for gradually refining predictions through series of transformations and updates applied to the token representations.
3.2 Fourier Features in MLP & Attention Outputs
Logits for MLP and attention have periodic structures. We now analyze how each MLP and attention module contributes to the final prediction. We transform the output of the attention and MLP output at layer l into the token space using W (ttn( and W ML)(() each layer, thereby obtaining the logits for each MLP and attention module. We use the running example "Put together 15 and 93. Answer: 108" to demonstrate how the fine-tuned GPT-2-X0 XI. performs the computation. As illustrated in Figure lb and Figure Ic, both the MLP and attention modules exhibit a periodic pattern in their logits across the output number space, e.g., the MLP in layer 33, outlined in green, promotes all numbers that are congruent to 108 mod 2 (in Figure 20 in the appendix, we zoom into such layers to make this clearer). Overall, we observe two distinct types of computation within these components. Some components predominantly assign a high weight to numbers around the correct answer, which term approximation. Meanwhile, other components predominantly assign high weight to all numbers congruent to a + mod e for some constant c, which we term classification.
Logits for MLP and attention are approximately sparse in the Fourier space. It is natural to transform the logits into Fourier space to gain a better understanding of their properties such as the periodic pattern. We apply the discrete Fourier transform to represent the logits as the sum of

---

Attention Output in Fourier Space for Layer 33
Attention Output in Fourier Space for Layer 40
T=2.00
sin
60
T=5.00
sin
cos
CoS
T-10.00
T-5:00
Magnitude
Magnitude
40
T=10.00
I-10.03
T=10.00
T-5.00
T-1.98
T+3.33
T+2.50
T-2.49
T-2.02
T-9.6:
T=0.01
T-2.00
0
0
50
100
150
200
250
50
100
150
200
250
Fourier Component
Fourler Componen:
(a) Logits for the MLP output in Fourier space
(b) Logits for the attention output in Fourier space
Figure 2: The intermediate logits in Fourier space. We annotate the top-10 outlier high-frequency
Fourier components based their magnitudes. T stands for the period of that Fourier component.
(a) The logits in Fourier space for the MLP output of the 33-rd layer, i.e., _(33) The component with period 2 has the largest magnitude, aligning with the observations in Figures 1b and 20a. (b) The logits in Fourier space for the attention output of the 40-th layer, i.c., A The components with periods and 10 have the largest magnitude, aligning with the observations Figures Ic and 20b.
sine and cosine waves of different periods: the k-th component in Fourier space has period 520/ /kk and frequency k/520 (see Appendix A for more details). Let 2 denote the logits in Fourier space. Figure 2 shows the Fourier space logits for two layers from Figure 1b and Figure Ic that have a clear periodic pattern. We find that the high-frequency components Fourier space, which we define as components with index greater or equal to 50, are approximately sparse as depicted in Figure 2. This observation aligns with [28], which found that a one-layer Transformer utilizes particular Fourier components within the Fourier space to solve the modular addition task.
In Figure 3, we show that similar sparsity patterns in Fourier space hold across the entire dataset. We compute the logits in Fourier space for the last 15 layers, i.e., Σ(e) and Z(O) where & E [32, 47), for all test examples and average them. We annotate the top-10 outlier high-frequency components based on their magnitude. The MLPs also exhibit some strong low-frequency components; the attention modules do not exhibit strong low- wwwwereeenccy components, only eeg-ffre components.
Heatmop ML. Outputs in Fourier Space for Each Layer
Hestmap of ATTN Outpots Fourier Space for Each Loyer
Peried
Dostra
Teried
200
Site
Pirted 2
Site
Farto 2:31
Peiteo 2.10
100
PERtO 2:10
150
1.0
300
200
Parted 3,00
Perivall (,00
Paitoe 5,00
Peritdl 5,00
Peried 0.01
10
30
31
25
28
41 Laysr Trales
20
10
12
41
4
45
16
20 321 24
36
20
10
41
42
41
44
45
Layse Trales
(a) Logits for MLP output in Fourier space
(b) Logits for attention output Fourier space
Figure 3: Analysis of logits in Fourier space for all the test data across the last 15 layers. For both the MLP and attention modules, outlier Fourier components have periods around 2, 2.5, 5, and 10.
Final logits superpositions of these outlier Fourier components. The final logits, C(4), are the sum of all (()) and (() across layers I [T.]. Figure elucidates how these distinct Fourier components contribute to the final prediction, for the example "Put together 15 and 93. Answer: 108".

---

Logits for Top Fourier Components
Summation of the Logits for Top Fourier Components
20
Persod 520.00
Partod 47.27
15
Period 10.06
10
Period, 5. 3.00
Maqnitude
Fariod 2.
Magaltude
5
-5
-10
-15
108
100
120
125
320
Number Space
Number Spare
(a) Final logits for top Fourier components
(b) Summation the Top-5 Fourier components
Figure 4: Visualization of how sparse subset Fourier components can identify the correct answer. (a) Shows the top-5 Fourier components for the final logits. (b) Shows the sum of these top-5 Fourier components, highlighting how the cumulative effect identifies the correct answer, 108.
We select the top-5 Fourier components of E(L) based their magnitudes and transfer them back to logits in number space via the inverse discrete Fourier transform (Figure 4a). The large-period (low-frequency) components approximate the magnitude while the small-period (high-frequency) components are crucial for modular addition. Figure 4b shows that aggregating these waves is sufficient predict the correct answer.
Why is high-freueency classification helpful? The Fourier basis comprises both cos and sin waves (see Definition A.3). By adjusting the coefficients of cos and sin, the trained model can manipulate the phase of the logits in Fourier space (number shift in number space), aligning the peak of the wave more closely with the correct answer. As shown in Figure 4a, consider a wave with period of 2. Here, the peak occurs at every even number in the number space, corresponding to the mod 2 task. In contrast, for components with a large period such as 520, the model struggles to accurately position the peak at 108 (also see Figure 14 in the appendix for the plot of this component with period 520 in the full number space). This scenario can be interpreted task-a classification task among 520 classes-which is challenging for the model to learn accurately. Nevertheless, even though the component with period of 520 does not solve the "mod 520" task precisely. it does succeed in assigning more weight to numbers near 108. The classification results from the high-frequency components can then provide finer-grained resolution distinguish between all the numbers around 108 assigned large weight by the lower frequencies. Due to this, the low- frequency components need not be perfectly aligned with the answer to make accurate predictions.
solving "mod 520"
3.3 Fourier Features are Causally Important for Model Predictions
In the previous section, we demonstrated that there are outlier Fourier components in the logits generated by both the MLP and attention modules, as shown Figure 3. We also illustrated that, in one example, the low-frequency components primarily approximate the magnitude, while the high-frequency components are crucial for modular addition tasks, as depicted in Figure 4. this section, through ablation study conducted across the entire test dataset, we show that both types of components are essential for correctly computing sums. Moreover, we reveal that the MLP layers primarily approximate the magnitude of the answer using low- wwwffreuenccy features, whereas the attention layers are responsible for modular addition using gigh-frequency features.
Filtering out Fourier components. To understand the role various frequency components play for the addition task, we introduce low-pass and high-pass filters For an intermediate state h, and a set of frequencies I {...... ) the filter F(h; returns the vector that is closest L2 distance to h subject to the constraint that the Fourier decomposition of W at every frequency Y We show in Appendix that this has a simple closed-form solution involving a linear projection. We then apply either a low-pass filter by taking I' to be all the components whose frequencies are greater than the frequency of the 7-th component for some threshold (i.e., removing high-frequency components), and hagh filter by taking I' to be all the components whose frequencies are less
5

---

Table 1: Impact of Filtering out Fourier Components on Model Performance. Removing low-
frequency components from attention modules (blue) or high-frequency components from MLP modules (red) does not impact performance
Module
Fourier Component Removed
Validation Loss
Accuracy
None
Without Filtering
0.0073
0.9974
ATTN & MLP ATTN
eew-Freuenccy
4.0842
0.0594
Low-Frequenge
0.0352
0.9912
MLP
Low-Frequency
2.1399
0.3589
ATTN & MLP
High-Frequency
1.8598
0.2708
ATTN
High-Frequency
0.5943
0.7836
MLP
High-Frequency
0. 0.1213
0.9810
than the frequency of the 7-th component (i.e., removing low-frequency components). As in the previous subsection, take the high-frequenc threshold 50 for the following experiments (see Appendix B for more details).
Different roles of frequency components in approximation and classification tasks. We evaluated the fine-tuned GPT-2-XL model on the test dataset with different frequency filters applied to all the output of MLP and attention modules. The results, presented in Table 1, indicate that removing low- frequency components from attention modules or high-frequency components from MLP modules does not impact performance. This observation suggests that attention modules are not crucial for approximation tasks, and MIP modules are less significant for classification tasks.
Eliminating fighffreyeency components from attention results in a noticeable decrease in accuracy. Furthermore, removing high-frequency components from both the attention and MLP modules simultaneously leads to an even greater reduction in accuracy. This finding corresponds with observations from Figure I and Figure 3, which indicate that both MLP and attention modules are involved in classification tasks due to the presence of high- fighffrequency components in the logits. As shown in Table 1, the approximation tasks are primarily performed by the MLP modules, with contributions from the attention modules as well.
The errors induced by these ablations align with our mechanistic understanding. Ablating low- frequency parts of MLPs leads to off-by 10, 50, and 100 errors: the model fails to perform the approximation subtask, though it still accurately predicts the unit digit. Conversely, ablating high- frequency parts of attention leads to small errors less than 6 in magnitude: the model struggles to accurately predict the units digit, but it can still estimate the overall magnitude of the answer. See Figure 21 in the Appendix for more details. These observations validate hypothesis that low-frequency components are crucial for approximation, while high- figh-frequency components are vital for classification. The primary function of MLP modules is to approximate the magnitude of outcomes using low- wwwfrreuency components, while the primary role of attention modules is to ensure accurate classification by determining the correct unit digit.
4 Effects of Pre-training
The previous section shows that pre- pre-trained LLMs leverage Fourier features to solve the addition problem. Now, study where the models' reliance Fourier features comes from. In this section, we demonstrate that LLMs learn Fourier features in the token embeddings for numbers during pre-training. These token embeddings models trained from scratch achieve lower accuracy, but adding just the pre-trained token embeddings fixes this problem. We also show that pre-trained models leverage Fourier features not only when fine-tuned, but also when prompted.
important for achieving high accuracy the addition task:
4.1 Fourier features in Token Embedding
Number embedding exhibits approximate sparsity in the Fourier space. Let WE E RRxD where p = 521 and D the size of the token embeddings, denote the token embedding for numbers. We apply the discrete Fourier transform to each column of E to obtain a matrix V e D where

---

Chustering Token Embeddings GPT2
Number Embedding in Fourier Space: Fine-tuned GPT-2-XL
Equlaalat under 2112
Magnitude LF
T=10.00
T-2.00
20
T- 1=10.00
T-5.00
Group 7182
Mullipia 22
T=2.50
T+3.33
T-3.33
1
T-2.50
0
50
100
150
200
250
Fourler Componen:
(a) Number embedding in Fourier space
(b) Number embedding clustering
Figure 5: (a) Number embedding in Fourier space for fine-tuned GPT-2-XL. T stands for the period of that Fourier component.Ch Visualization of token embedding clustering of GPT-2 using T-SNE and k-means with 10 clusters. The numbers are clustered based on their magnitude and whether they are multiples of 10.
each row represents a different Fourier component. Then we take the L2 norm of each row to yield p-dimensional vector. Each component j in this vector measures the overall magnitude of the j-th Fourier component across all the token embedding dimensions. Figure 5a shows the magnitude of different Fourier components in the token embedding of GPT-2-XI. We see that the token embedding has outlier components whose periods are 2, 2.5, 5, and 10. Therefore, similar to how the model uses different Fourier components represent its prediction (as shown in Section 3.2), the token embeddings represent numbers with different Fourier components. Figure 15 in the Appendix shows that the token embeddings other pre-trained models have similar patterns the Fourier space. This suggests that Fourier features are a common attribute in the token embedding of pre-trained LLMs. In Figure 5b, we use t-SNE and k-means to visualize the token embedding clustering. We can see that numbers cluster not only by magnitude but also by their multiples of 10.
4.2 Contrasting Pre-trained Models with Models Trained from Scratch
Hoatmap of MLP Outpots Pourler Space for Each Layer
Heatmap of ATTN Occputs in Fourier Space for Each Layer
8300
1000
200
800
0300
200
600
200
600
2000
100
100
Lagaz
Layer
(a) Logits for MLP output in Fourier space
(b) Logits for attention output in Fourier space
Figure 6: Visualization of the logits Fourier space on the test dataset from the last 15 layers for GPT-2-XL model trained from scratch. For both the MLP and attention modules, there are no outlier Fourier components, in contrast with the clear outlier components in the fine-tuned model (Figure 3).
To understand the necessity of Fourier features for the addition problem, we trained the GPT-2-XI. model from scratch on the addition task with random initialization. After convergence, it achieved only 94.44% test accuracy (recall that the fine-tuned GPT-2-XL model achieved 99.74% accuracy).
7

---

Fourier features are learned during pre-training. Figure 6 shows that there are no Fourier features in the intermediate logits of the GPT-2-XL model trained from scratch on the addition task. Furthermore, Figure 7a shows that the token embeddings also have no Fourier features. Without leveraging Fourier features, the model merely approximates the correct answer without performing modular addition, resulting in frequent off-by-one errors between the prediction and the correct answer (see details in Figure 23).
Number Embedding in Fourter Space: GPT-2 Trained From Scratch
Validation Accuracy per Epoch
sin
Sahizoom Aramiy
Magnituse
2
0.
1.5
1
Wibbett Protrozzed
0
10th piner srchidding
100 150
250
106
200
600
900
Fourter Companent
Tpsch
(a) Embedding: GPT-2-XL Trained from Scratch
(b) Validation Accuracy Comparison for GPT-2
Figure 7: (a) The number embedding in Fourier space for GPT-2-XL trained from scratch. There are no high-frequency outlier components, in contrast with the pre-trained embeddings (Figure Sa). (b) Validation accuracy of GPT-2-smal trained from scratch either with or without pre-trained token embeddings. We show the mean and the standard deviation of the validation accuracy across random seeds. GPT-2-small with pre- pre-trained token embedding consistently achieves 100% accuracy, while GPT-2-small without pre-trained token embedding only achieves less than 60% accuracy.
Pre-trained token embeddings improve model training. We also trained GPT-2-small, with 124 million parameters and 12 layers, from scratch the addition task. GPT-2-small often struggles with mathematical tasks [26]. This model achieved a test accuracy of only 53.95% after convergence. However, when we freeze the token embedding layer and randomly initialize the weights for all other layers before training on the addition task, the test accuracy increases to 100%, with a significantly faster convergence rate. This outcome was consistently observed across five different random seeds, as illustrated in Figure 7b. Following Section 3.2, to validate that the model learn to leverage the Fourier feature solve addition, we analyze the logits in Fourier space for all the test data across all 12 layers In Figure 8, we can clearly observe that, with solely the pre-trained number embedding, the Fourier features appear in the MLP and attention modules' output for most layers. This demonstrates that given the number embeddings with Fourier features, the model can effectively learn to leverage these features to solve the addition task.
4.3 Fourier Features in Prompted Pre-Trained Models
Finally, we ask whether larger language models use similar Fourier features during prompting.
Pre-trained LLMs use Fourier features to compute addition during in-context learning. We first test on the open-source models GPT-J [43] with 6B parameters, and Phi-2 [21] with 2.7B parameters on the test dataset. Without in-context learning, the model cannot perform addition tasks. Therefore, we use 4-shot in-context learning to test its performance. Their absolute errors are predominantly multiples of 10: 93% of the time for GPT-J, and 73% for Phi-2. Using the Fourier analysis framework proposed in Section 3.2, we demonstrate that for Phi-2 and GPT-J, the outputs of MLP and attention modules exhibit approximate sparsity in Fourier space across the last 15 layers (Figure 9 and Figure 19). This evidence strongly suggests that these models leverage Fourier features to compute additions.
Closed-source models exhibit similar behavior. We study the closed-source models GPT-3.5 [33], GPT-4 [34], and PaLM-2 [16]. While we cannot analyze their internal representations, we can study whether their behavior on addition problems consistent with reliance on Fourier features. Since closed-source L.MS are instruction tuned and perform well without in in-context learning, we conduct error analysis with O-shot. Most absolute errors by these models are also multiples of 10: 100% of the time for GPT-3.5 and GPT-4, and 87% for PaLM-2. The similarity in error distribution to
8

---

Hestmap of MLP Outputs Fourier Space Each Layer
Eleatmap ATTN Outputs Fourier Space for Bach Layer
300
500
173
175
400
150
000
Megeratt
Magahade
100
80
100
100
25
Layer Lacke
Lases Index
(a) Logits for MLP output in Fourier space without pre-trained number embeddings
(b) Logits for attention output in Fourier space without pre- pre-trained number embeddings
Beetmmp Outputs Space
Hestmap of ATTN Outputs in Fourler Space for Bach Layer
500
Cosaao
Persot:
Stoc
Perosh mM
450
(Neted: 2.50
Perted 2,09
2000
Magnttada
300
800
750
Perine
Ieeash (1))
Pettre 3:00
Perize: Posaah 10.00
Peeissh 0.01
2
Lager Lodes
Laper talox
(c) Logits for MLP output in Fourier space with pre- trained number embeddings
(d) Logits for attention output in Fourier space with pre-trained number embeddings
Figure 8: Analysis of logits Fourier space for all the test data across the 12 layers. (a,b) GPT-2- small trained from scratch fail to learn to leverage the Fourier feature to solve addition. (c,d) However, with solely the pre-trained number embeddings, GPT-2-small. is able to learn leverage the Fourier features to solve the addition as the fine-tuned models.
Healmap of MLP Outputs in Fourier Space for Each Layer
Hestmap of ATTN Outputs in Fourier Space for Each Layer
Costoa
Costoe
Sine
Pwrtod
10
Sine
Pertod
Perion 2.50
400
Pertod 2.50
400
Compenente
Peroot:
Pwrine:
Pwiind
Magunide
Maguzude
Prrmod: (0)
200
700
Perton 3,20
Perautt mm
Perton
Prraatt 5.00
17
22
27
26
27
24
27
28
Lapse Index
Lapsc Index
(a) Logits for MI.P output in Fourier space
(b) Logits for attention output Fourier space
Figure 9: For Phi-2 (4-shot), analyzed the logits in Fourier space for all the test data across the last 15 layers. For both the MLP and attention modules, the outlier Fourier components have periods around 2, 2.5, 5, and 10, similar to the fine-tuned EPT-2-XL logits (Figure 3).
that of open-source models leads us to hypothesize that Fourier features play a critical role their computational mechanism.

---

Related Work
Learning mathematical tasks. Previous studies primarily explore what pre-trained LMs can achieve on arithmetic tasks, with less emphasis the underlying mechanisms [29, 37]. For instance, [22] demonstrates that small Transformer models can effectively learn arithmetic by altering the question format and utilizing a scratchpad method [30]. [19] identifies activation patterns for the "greater-than" operation in GPT-2, and [5] focuses on the enumeration and selection processes in GCD computation. In this paper, dive into the specific roles of MLP and attention layers in solving mathematical tasks. Our research analyzes these components' distinct contributions to integer addition tasks.
Mechanisms of pre-trained LMs. Recent studies have significantly advanced our understanding of the underlying mechanisms of pre-trained Transformer models. For instance, research on "skill neurons" by [44] and "knowledge neurons" by [8] underscores the development of specialized neural components that encode task-specific capabilities or hold explicit factual information in the pre-trained LMs, enhancing model performance on related tasks. [25] and [15] discuss how MLPs and FFNs transform and update token representations for general language tasks. In contrast, we show that the pre-trained LMs use multiple layers compute addition by combining the results of approximation and classification. Additionally, [46] demonstrated the capacity of GPT-2 to consolidate similar information through pre-training the model weights, which aligns with our observations on the importance of pre- training in developing effective number embedding and arithmetic computation strategies LMs.
Fourier features in Neural Networks. Fourier features are commonly observed in image models, particularly in the early layers of vision models [32, 31, 10]. These features enable the model to detect edges, textures, and other spatial patterns effectively. Recently, Fourier features have been noted in networks trained for tasks that allow cyclic wraparound, such as modular addition [28, 27], general group compositions [6], or invariance to cyclic translations [38]. [28] demonstrates that learning Fourier features can induce 'grokking" [35]. Furthermore, [24] provides a mathematical framework explaining the emergence of Fourier features when the network exhibits invariance to finite group. We extend these insights by observing Fourier features in tasks that do not involve cyclic wraparound. [39] found that by selecting problem-specine Fourier features, the performance of MIPPs can be improved on computer vision-related task.
Conclusion
In this paper, we provide a comprehensive analysis of how pre-trained L.MSs compute numerical sums, revealing nuanced interplay of Fourier features within their architecture. Our findings demonstrate that LLMs do not simply memorize answers from training data but actively compute solutions through combination of approximation and classification processes encoded in the frequency domain of their hidden states. Specifically, MLP layers contribute to approximating the magnitude of sums, while attention layers contribute to modular operations.
Our work also shows that pre-training plays a critical role in equipping LLMs with the Fourier features necessary for executing arithmetic operations. Models trained from scratch lack these crucial features and achieve lower accuracy; introducing pre-trained token embeddings greatly improves their convergence rate and accuracy. This insight into the arithmetic moblem-solving capabilities of LLMs through Fourier features sets the stage for potential modifications to training approaches. By imposing specific constraints model training, could further enhance the ability of LLMs to learn and leverage these Fourier features, thereby improving their performance in mathematical tasks.
7 Limitations
We note that our contributions are limited by the size of the dataset. As the maximum number that can be represented by one token for GPT-2-XL is 520, we analyze on the dataset whose operands are less than 260. However, as the Fourier features commonly exist in many different pre-trained models as shown Section 4, we believe different models still use Fourier features, possibly with a more complicated strategy.
10

---

Acknowledgments
DF and RJ were supported by a Google Research Scholar Award. RJ was also supported by an Open Philanthropy research grant. VS was supported by NSF CAREER Award CCF-2239265 and an Amazon Research Award. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the funding agencies.
References
[1] Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024.
[2] Yu Bai, Fan Chen, Haiquan Wang. Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. ArXiv, 0bss2300.006337. 2023.
[3] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.088 /2, 2023.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] François Charton, Can transformers learn the greatest common divisor? arXiv preprint arXiv:2308. 2023.
[6] Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. In International Conference on Machine Learning, pages 0243-6261 PMLR, 2023.
[7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021.
[8] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.0888 086966. 2021.
[9] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/transffommr-ccine cirruttrrrrrrrrreeeeeeeeeeeeseee pubp:ittsppp//222111 www.///wwwo/inde....i...
[10] Pierre-ftiennee Fiquet and Fero Simoncelli. A polar prediction model for learning represent visual transformations. Advances in Neural Information Processing Systems, 36, 2024.
[11] Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz. Philipp Petersen, and Julius Berner. Mathematical capabilities of chatgpt. Advances in Neural Information Processing Systems, 36, 2024.
[12] Deqing Fu, Tian-Qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn higher-order optimization methods for in-context learning: A study with linear models, 2023.
[13] Deqing Fu, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. Isobench: Benchmarking multimodal foundation models on isomorphic representations, 2024.
[14] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? case study of simple function classes. ArXiv, abs/2208.01066, 2022.
[15] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint 2203 2022.

---

[16] Google. Palm 2 technical report, 2023.
[17] Gemini Team Google, Gemini: A family of highly capable multimodal models, 2023.
[18] Jiuxiang Gu, Chenyang Li, Yingyu Liang, Zhenmei Shi, Zhao Song, and Tianyi Zhou. Fourier circuits in neural networks: Unlocking the potential of large language models in mathematical reasoning and modular arithmetic, 2024.
[19] Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater- than?: Interpreting mathematical abilities in a pre-trained language model. arXiv preprint arXiv:2305.00586 2023.
[20] Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
[21] Mojan Javaheripi. Sebastien Bubeck, Marah Abdin, Jyoti Anejaand Caio Cesar Teodoro Mendes, Allie Del Giorno Weizhu Chen, Ronen Eldan, Sivakanth Gopi, Suriya Gunasekar, Piero Kauff- mann, Yin Tat Lee, Yuanzhi L, Anh Nguyen, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Michael Santacroce, Harkirat Singh Behl, Adam Taumann Kalai, Xin Wang, Rachel Ward, Philipp Witte, Cyril Zhang, and Yi Zhang. Phi-2: The surprising power of small language models, 2023.
[22] Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.0338/ 2023.
[23] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024.
[24] Giovanni Luca Marchetti, Christopher Hillar, Danica Kragic, and Sophia Sanborn. Har- monics of learning: Universal fourier features emerge in invariant networks. arXiv preprint arXiv:2312.085500 2023.
[25] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec-style vector arithmetic, arXiv preprint arXiv:2305.1613 2023.
[26] Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. Numglue: A suite of fundamental yet challenging mathematical reasoning tasks. arXiv preprint (0.Xi::224.0366000 2022.
[27] Depen Morwani, Benjamin L Edelman, Costin-Andrei Oncescu, Rosie Zhao, and Sham Kakade. Feature emergence via margin maximization: case studies in algebraic tasks. arXiv preprint arXiv:2311.07566 2023.
[28] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023.
[29] Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations of transformers with simple arithmetic tasks. arXiv preprint arXiv:2102. 13019, 2021.
[30] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.0011: 2021.
[31] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. An overview of early vision in inceptionvl. Distill, 5(4):e00024-000, 2020.
[32] Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23)33311-33 1997.
[33] OpenAl. Introducing ChatGPT. https: tttps://openat. com contt/lg///c/acgggttttt 2022. Accessed: 2023-09-10.
12

---

[34] OpenAl. Gpt-4 technical report, 2023.
[35] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:220..02177, 2022.
[36] Alethea Power, Yuri Burda, Harrison Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. ArXiv, 02222777, 2022.
[37] Jing Qian, Hong Wang. Zekun Li, Shiyang Li, and Xifeng Yan. Limitations of language models in arithmetic and symbolic induction. arXiv preprint arXiv:2208.0500 2022.
[38] Sophia Sanborn, Christian Shewmake, Brumo Olshausen, and Christopher Hillar. Bispectral neural networks. arXiv preprint TrXiv:2209.03416, 2022.
[39] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
[40] Avijit Thawani, Jay Pujara, Pedro A Szekely, and Filip llievski. Representing numbers in nip: a survey and vision. arXiv preprint arXiv:2103. 13136, 2021.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv:1706.03762, 2017.
[42] Johannes von Oswald, Eyvind Niklasson, E. Randazzo, João Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference Machine Learning, 2022.
[43] Ben Wang and Aran Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: /ettss//githull com www./iiiggollll//eesttttitn seniehh-rransee transformer- jax, May 2021.
[44] Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons in pre-trained transformer-based language models. arXiv preprint a.X0v:2211.07349, 2022.
[45] Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Proceedings of the 2017 conference on empirical methods in natural language processing. pages 845 854, 2017.
[46] Zeyuan Allen Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction, Xiviv preprint arXiv:2309, (tti:2399..1316,, 2023.
13

---

Appendix
Roadmap. In Appendix A, introduce some formal definitions that used in our main content. In Appendix B, we show why we separate the Fourier components into the high-frequency part and the low-frequency part and why we choose T be 50. In Appendix C, we show our observation generalizes to another format of dataset, another arithmetic task and other models. In Appendix D, we provide more evidence that shows the Fourier features in the model when computing addition. In Appendix E, we provide more evidence that shows the GPT-2-XL trained from scratch does not use Fourier feature to solve the addition task. In Appendix F, we give the details of our experimental settings.
A Formal Definition of Transformer and Logits in Fourier Space
We first introduce the formal definition of the Transformer structure that we used in this paper. Definition A.1 (Transformer). An autoregressive Transformer language model G : X - y vocabulary Vocab maps token sequence ability distribution y E y C RRVocabi that predicts next-token continuations of x. Within the Transformer, the i-th token is embedded as a series of hidden state vectors ((((), beginning with ()) = -emb (x4) + pis(i) E TRD. Let WU (R. VooablxDD denote the output embedding. The final output = softmax(WU (b)) is read from the last hidden state. In the autoregressive case, tokens only draw information from past tokens:
[Z1y...., ZN] E X, E Vocab a prob-
n1 (e-1) + Actn (9 + ML.P(()
where
Attn, (1) = Attn
[1 (e(e-))
1
hi )) and MLP == MLP( (9) n (((1))).
h22
In this paper, we only consider the output tokens be numbers. Hence, we have the unembedding matrix E RRXD where is the size of the number space. As we are given the length-N input
sequences and predict the (N 1)-th, we only consider 4 N (((-1) Atta ( MLP N For
simplicity, we ignore the subscript N in the following paper, so we get Eq. (1).
Definition A.2 (Intermediate Logits). Let Attn W Attn denote the intermediate logits of
(f)
the attention module at the f-th layer. Let (()) MLP == WUML)(() denote the intermediate logits of the MLP module at the E-th layer. Let L(0) = ( denote the logits on intermediate state h!))
Throughout the model, h undergoes only additive updates (Eq. (1)), creating continuous residual stream [9), meaning that the token representation accumulates all additive updates within the residual stream up to layer
To analyze the logits in Fourier space, we give the formal definition the Fourier basis as follows:
Definition A.3 (Fourier Basis), Let denote the size of the number space. Let X == (0, 1..... (p We denote the normalized Fourier basis F as the X p matrix:
2ar 1)). Let W/: == p-1'
Y
2
sin (wi x
c08
(W)
F==
E RRPx
sin (wz x
p-1
1 cos (W(p- 11/2 x)
The first component F F[0] is defined as a constant component. For i E [0,p 11. F(i] is defined as the kith component in Fourier space, where = 1 J. The frequency of the k-th component is fo = .. The period of the k-th component is Tk = p-1
14

---

We can compute the discrete Fourier transform under that Fourier basis as follows:
Remark A.4 (Discrete Fourier transformer (DFT) and inverse DFT), We can transform any logits RP Fourier space by computing ū F We transform ù back to u by FT.
Next, we define the logits in Fourier space.
Definition A.5 (Logits in Fourier Space), Let L(L), Litt and MLP denote the logits (Definition
A.2). The output logits before softmax in Fourier space is defined as: [(L) = F(..,, The logits of the MLP and attention modules in Fourier space are defined as:
((E) Atta
F.FFFL.
Aten and
[(E) LMPP F
=F.L
(L)
MLP
We ignore the first elements in _1D), 210 and ZLC Astn "MPP for the Fourier analysis in this paper as they the constant terms. Adding constant to the logits will not change the prediction.
Let T E denote a constant threshold. The low-frequency components for the logits Fourier space are defined as [ 2r]. The high-frequency components for the logits in Fourier space are defined as _ [2T :]. For the following analysis, we choose T 50 (the specific choice of r 50 is explained in Appendix B).
Next, we propose the formal definition of low-pass www-passhhigh-ep filter that is used in the following ablation study.
Definition A.6 (Loss-pass / Hass Filter). Let x E RR denote the output of MLP attention modules. Let F denote the Fourier Basis (Definition A.3). Let T & R denote the frequency threshold. Let W RRx denote the outpul embedding. For low-pass filter, we define diagonal binary
ifi FFrr high-pass filter, we define diagonal binary
matrix B E 10,1 as big
otherwise"
1 il -Note that retain the constant component,
matrix B E {0, 11pxP as bis
otherwise
bi, = 0. The output of the filter F(x): DRD + RD is defined by the following objective function:
min | 1aa-vlk v11
subject to BFW y =
The solution the above optimization problem is given by a linear projection.
Remark A.7. The result of the optimization problem defined in Definition A.6 is the projection of to the null space of BFWU, N (BFWU) denote the null space of BFWU. have
F(x) NN(EWW) -N(FFWU) x
15

---

B Fourier Components Separation and Selection of т
Hestmap of MLP Outputs in Fourier Space for Each Laver
Heatmap of ATIN Outpats Fourier Space for Each Laver
Costoe
Costoe
24
30
120
90
80
100
Magnibdia
130
THEMS: 8722
Magettudi
50
Peradd52..0 52.05.000
1.00
Forads 22.00
00
Peroul 10000
40
Paroal 100.00
Deraal: 175.5
Forad: 20
Poradi 760
Pernnd Fermii-50..0 $80.00
Perwi--5n.00 530.00
Perill 320.00
Per 530.00
(a) Logits for MLP output Fourier space
(b) Logits for attention output in Fourier space
Figure 10: We analyzed the logits in Fourier space for all the test data across the last 15 layers. For both the MLP and attention modules. We only plot the first 50 Fourier components (a) The MLP exhibits some outlier low-frequency Fourier components. (b) The attention module's low-frequency Fourier components are not as obvious as the ones in MLP.
Following Definition A.6, define single-pass filter as follows:
Definition B.1 (Single-Pass Filter). Let x E RDD denote the output of ML.P or atlention modules. Let F denote the Fourier Basis (Definition A.3). Let ny E R denote the 7-th Fourier component (Definition A.3) that we want to retain. Let W E Ry xb denote the output embedding. We define
0 if - -
diagonal binary matrix B E 100,1VXV as bys =
otherwise.
The oulput of the filter Fy(x): RDD - RR is defined the following objective functioon
min || vll
y
subject to
BFW y =0
Remark B.2. The result of the optimization problem defined in Definition B.1 is the projection of to the null space of BFW Let (BFW denote the null space of BFW". We have
F,(x) = ((B)=N(BFFU) -N(BEWU)T T
For the single pass filter, only retrain Fourier component and analyze how this component affects the model's 's prediction. The residual stream is then updated as follows:
( = 1 (e-1) (Attm((1))) + +F, (MLL(((()))
We evaluated the fine-tuned GPT-2-XL. model the addition dataset with the Fourier components period 520 and Given that Ti : V-1 (Definition A.3), we retained only the Fourier components with 7 1 and 260, respectively.
As shown in Figure 1la, with only one frequency component, whose period 2, the model accurately predicts the parity with 99.59% accuracy. As depicted in Figure 1lb, with single frequency component of period 520, the model fails to accurately prediet with 96.51% accuracy. We consider the frequency component with a period of 2 as the model's 's prediction for the mod 2 task, and the frequency component with a period of 520 as its prediction for the mod 520 task. Figures 11 and 12 suggest that the model effectively learns the mod 2 task, as it involves two-class classification, but struggles with the mod 520 task, which requires classifying among 520 classes. As the model does not need to be trained to converge to the optimal for these low-frequency components as explained at the end of Section 3.2, predicting with the period-520 component leads to predictions that normally distributed around the correct answers.
16

---

Parily Comparison: Bataining Ouly Partad-2 Frequency Componnnis
Asswer Comparison: Rotaininng Ouiy Period-520 Frequency Components
Connent Parlty
99 59%
Inowmedi
96.61%
Predictio Type
Powdiciio Type
Parily
20
50
80 100
100
Percentage
(a) Retaining Period-2 Frequency Components
(b) Retaining Period-520 Frequency Components
Figure 11: The prediction analysis when predicting with only one Fourier component. (a) Retaining only the Period-2 Fourier component makes the prediction 99.59% accurate for mod task (b) Retaining only the Period-520 Fourier component makes the prediction 96.61 % inaccurate for the mod 520 task.
Distribution: Retaining Period-520 Frequency Components
500
Frequeacy
400
200
100
-150
-100
-50
50
150
Error
Figure 12: Retaining only the Period-520 Fourier component makes the model's predictions normally
distributed around the correct answers.
The Fourier components with larger periods present greater difficulty in solving the corresponding modular addition task compared to those with smaller periods. As demonstrated in Figure 12, compo- nents with large periods serve primarily as approximations of the correct answer. Consequently, categorize the Fourier components into low-frequency and high-frequency groups. The low-frequency components approximate the magnitude of the answer, whereas the high-frequency components are employed to enhance the precision of the predictions.
elucidate the contribution of these distinct Fourier components our final prediction and the rationale behind their separation, consider the example: "Put together 15 and 93. Answer: 108". We selected the top-10 10 Fourier components of [(L) based on their magnitudes and converted them back logits in the numerical space by multiplying with FT. We plotted the components with components index less than 50 in Figure 13a and those with components index greater than 50 Figure 13b. Leveraging the constructive and destructive inference for different waves, the components with low periods assign more weight to the correct answer, 108, and less weight to numbers close to 108. These high-frequency (low-period) components ensure prediction's 's accuracy at the unit place. For the low-frequency (large-period) components, the model fails to precisely learn the magnitude of the factor between the cos and sin components, which results in failing to peak at the correct answer. Thus, the oow-frequency (large-period) components used to approximate the magnitude of the addition results.
In reference Figure
17

---

Top Components
Logits Top Fourner Compeeeetic
Axtaa botuas Avcal meeas
UNGTRCT DO
12b
100
315m
1330
32.
Kinbar
(a) Final Logits for Components whose index Less (b) Final Logits for Components whose index Greater than 50
than 50
Figure 13: Visualization of tte individual final logits for the top- 10 Fourier components with example "Put together 15 and 93. Answer: 108" (a) The components index less than 50. (b) The components index greater than 50.
Logits Top Componests
-100
Figure 14: Visualization of the Fourier component whose period 520 analysis for the final logits.
18

---

Does Fourier Features Generalize?
C.1 Token Embedding for Other LMs
We first show that other pre-trained LMs also have Fourier features in their token embedding for the numbers [0, 520].
Number Embeddingg in Fourier Space: Pre-trained GPT-2-XL
Number Embedding in Fourier Space: Fine-tuned GPT-2-XL
sin
COS
COG
Nagnitude 1
Magnituds
Tu:t0.00
Tw.500
Tw10
T-2.00
T-2.00
Tw5.000
2-2.0
2
T-2.60
Tw2 40
3-3333
Tw2.50
250
Component
Component (b) fine-tuned GPT-2-XL
(a) pre-trained GPT-2-XL
Number Embadding Fourler Space: Pre trained RoBERTo base
Number Embedding in Fouriar Space: Pre brabtrannd RoBERTh base
sin
CDS
cos
2.5
Magnitude
Mogniuus
Tw3
2.00
T1-5800
T-10.0
1.5
T-10.0
T=5.03
Tw0.00
1-2.50
Component
Component
(c) pre-trained RoBERTa
(d) pre-trained Phi2
Figure 15: Number embedding in Fourier space for different pre-trained models.
19

---

C.2 Multiplication Task
key question is whether pre-trained models utilize Fourier Features solely for solving addition tasks or if they generalize to other arithmetic tasks. We hypothesize the latter, knowing that numbers are represented by their Fourier features in the token embeddings after pre-training. Consequently, this Fourier representation should be leveraged in variety of number-related tasks. To validate this hypothesis, we perform a Fourier analysis on the GPT-2-XI. model fine-tuned for the multiplication task.
Considering maximum number of 520 for multiplication would result in an insufficient dataset size. Therefore, we set the maximum allowable product to 10000.
For each pair of numbers where the product does not exceed this limit, we used distinct phrasing for each pair numbers, selecting one template from five available templates. This ensures that every unique pair of numbers between 0 and 260 is presented with consistent phrasing from these templates. We have fixed that typo in the revised version. The different phrasings used include: "What is the product of num1 and num2?", "Find the product of numl multiplied by num2.", "Calculate numl times num2.", "numl multiplied by num2 equals what?", and "Multiplication of numl I with num2." The dataset is then shuffled to ensure randomness and split into training (80%), validation (10%), and test (10%) sets. We finetune the model for 25 epochs with a learning rate of le - 4. Upon convergence, the validation accuracy reaches 74.58%.
As the primary objective is to determine whether the Fourier features are utilized in tasks other than addition, Figure 16 displays the logits in Fourier space for each layer, as in Figure 3. It is evident that the logits are sparse in Fourier space.
Heatmap of MILP Clutputs in Fourter Space for Kach Layer
Heatmap of ATTN Outpats in Pourter Space for Kach Layer
60
Cootir Rite
Costire Sine
>0
35-
400
25
30
Twiisd: 3:49
Pountor Conppiaens
Tweisd: 2:1h
Pountor Compaecc
25
Sogattude
30
300
20-
15
15-
10
Pretod
Prrtad
201
44 45
48
31
706
45
46
Loyer Index
Lagres Index
(a) Logits for MLP output in Fourier space
(b) Logits for attention output in Fourier space
Figure 16: We analyzed the logits in Fourier space for all the test data across the last 15 layers. For both the MLP and attention modules, the outlier Fourier components have periods around 2, 2.5, 3.3, 5, and 10.
20

---

C.3 Same Results for other format
To demonstrate that our observations are not confined a specific description of the mathematical problem, we conducted experiments on another format of addition problem and obtained consistent results. From Figure 17, we can see that there are also periodic structures in the intermediate logits.
Logits Acroes Each MLP Layers
Logits Across Each ATTN Layers
100
100
118
118
108
Number
Logits
108
Numbi
Logits
508
908
88
88
-50
78
78
1100
32
37
42
47
-50
32
37
42
47
68
Layer Index
Layer Index
(a) MLP output logits for the last 15 layers
(b) Attention output logits for the last 15 layers
Figure 17: Heatmap of the logits across different layers. The y-axis represents the subset of the number space around the correct prediction, while the x-axis represents the layer index.
From Figure 18, we can also see the Fourier features for the MLP and attention output. These two experiments validate that our observations are not confined to specific format of the addition problems.
Heatmap MLP Outputs in Fourter Space for Kach Layer
Heatmap of Attn Outputs Pourter Space for Each Laver
Costite
Pertoc:
500
Cositee
Pertod
500
Sine
Sinee
330
700
Fout Compaasii
Fwileer Compa 101
230
Magtinade
200
303
200
400
Terieh
Pertel:
Perted: 0.57
Perted: 07
Peninal: 4
100
1ou
200
100
Peroot: 10
Pontot 10
Perted: 10.00
35
54
20
20
90
42
45
49
47
35
34
30
30
Layer
Layer
(a) Logits for MLP output Fourier space
(b) Logits for attention output in Fourier space
Figure 18: We analyzed the logits in Fourier space for all the test data across the last 15 layers. For both the MLP and attention modules, the outlier Fourier components have periods around 2, 2.5, and 10. (a) The MI.P exhibits some outlier low-freat frequenquncy Fourier components. (b) The attention module does not exhibit any outlier low-frequency Fourier components, but it has stronger high-freyuency components.
C.4 Fourier Features in Other Pre-trained LM
Using the Fourier analysis framework proposed in Section 3.2, we demonstrate that for GPT-J, the outputs of MLP and attention modules exhibit approximate sparsity in Fourier space across the last 15 layers (Figure 19)
21

---

Beutmap of MLP Outputs in Fourier Space For Each Layer
Heotmap of Attn Outputs in Pourier Space for Bach Layer
Costoe
500
Cosine
500
Sinn
Smn
(Nered: Pertedl 2.10
Perted: 2.30
Mounitula
309
Mouiitude
Pirsed: 2.00
300
Pirtedi
Pertod: 3.33
Portod: 9.20
20g
10
200
Pedied: 5.00
Poriod: 5:20
Pedfot 5.00
Poried: 5.00
100
100
Peeied: 10.00
Denrel: 10.00
Prosi: 10,00
Deiiel: 10.0)
14
15
16
17
20
21
23
24
20
27
14
10
16
17
20
23
24
26
(a) Logits for MLP output in Fourier space
(b) Logits for attention output in Fourier space
Figure 19: For GPT-J (4-shot), we analyzed the logits in Fourier space for all the test data across the last 15 layers. For both the MLP and attention modules, the outlier Fourier components have periods around 2, 2.5, 5, and 10.
D Supporting Evidence For the Fourier Features
We selected the layers that clearly show the periodic pattern in Figure lb and Figure Ic and plot their logits in Figure 20.
Logits Weve for MLP of the 33-th Layer
Logits Weve for ATTN of the 40-th Layer
10
20
Logit
Logits
-5
-10
100
108
120
130
140
120
153
130
(a) Logits for 33-th layer's MLP output
(b) Logits for 40-th layer's attention output
Figure 20: The x-axis represents the number space, and the y-axis represents the logits value. (a) The logits wave for the MLP output of the 33rd layer, (333) The MLP favors even numbers. The MLP module favors the answer 15 + mod (b) The logits wave for the attention output the thth
layer, (10) The attention module favors the answer to 15 + 93 mod 10 and 15 + 93 mod 5.
Figure 21 illustrates that the errors resulting from the ablation study (Section 3.3) correspond with our theoretical insights. Removing low-frequency parts from the MI.P results in errors such as off-by 10, 50, and 100. Without these low-frequency components, the MLP is unable to accurately approximate, although it still correctly predicts the unit digit. In contrast, removing high- frequenyycncy components from the attention modules results in smaller errors, all less than 6 in magnitude. These findings support statement that low-frequency components are essential for accurate approximation, whereas fighefreyuency components are key for precise classification tasks. Consequently, the primary function of MLP modules is approximate numerical magnitudes using low-fread fow-freuuency components, and the essential function of attention modules is to facilitate precise classification by identifying the correct unit digit.
22

---

Erroc Distribution Filter out Low Frequency Componects of MLP Error Distribution: Filber out High Frequency Components of ATTN
10
50
Alsooute Error (9))
Absolate Brror Type
100
40
90
200
other 10s
nom- 0s
other
16
1.5 Percentage
201
25
30
10
20
30
40
Percentage
(a) Filtering out low-frequency components of MLP (b) Filtering out high-frequency components attention
Figure 21: (a) Across all the data, the difference between predictions and labels are all the multiple of 10. (b) Across all the test data, the differences between predictions and labels are all below 6.
E More Experiments on GPT-2-XL Trained from Scratch
Following the methodology proposed in Section 3, we plotted the logits of the MLP and attention modules for each layer, as shown in Figure 22. The prediction is solely determined by the 40-th layer MLP. Unlike Figure 3, there no observable periodic structure across all layers.
Logits Across Each MIP Lavers
Logits Across Fach ATTN Lavers
10000
138
400
7500
200
Nlmma
108
Nomibo
0
Logis
98
88
2500
88
-200
78
55000
78
-400
37
42
47
68
37
47
68
Layer Index
Layer Indlx
(a) output logits for last 15 layers
(b) Attention output logits the layers
Figure 22: Heatmap of the logits across different layers. The y-axis represents the subset of the number space around the correct prediction, while the x-axis represents the layer index. The final prediction is solely decided by the 40-th layer MLP.
For the model trained from scratch on the created addition dataset, all of the predictions on the test dataset deviate from the correct answer within shown in Figure 23,
23

---

Error Distribation: GPT2 XI. Trained From Scratch
Absolute Error
other
20
40
80
100
Percentage
Figure 23: Error distribution for GPT-2-XL trained from scratch.
F Details of Experimental Settings
Fine-tuned GPT-2-XL We finetune GPT-2-XL the "language-matth-attsse with 50 epochs and a batch size 16. The dataset consists of 27, 400 training samples, 3, 420 validation samples, and 3, 420 test samples. We use the Adam W optimizer, scheduling the learning rate linearly from 1 X 5 to without warmup.
Train GPT-2-XL from scratch We train GPT-2-XL on the "language-man matttee--tt--adtstt"" dattastt"""" from scratch with 500 epochs and batch size of The dataset consists of 27, 400 training samples, 3, 420 validation samples, and 3, 420 test samples. We use the Adam optimizer, scheduling the learning rate linearly from 10-1 to 0 without warmup.
Train GPT-2 from scratch For both with pre-trained token embedding and without token embedding. with 700 epochs and batch size of 16. The dataset consists of 27, 400 training samples, 3, 420 validation samples, and 3, 420 test samples. We use the AdamW optimizer, scheduling the learning rate linearly from 5 10 10-5 to 0 without warmup. In Figure 7b, we train the model with five different seeds and plot the mean and deviation for them.
train GPT-2 on the "language-math-duee
Create the addition dataset in main content We consider numbers in base 10 up maximum value of 260. For each pair of numbers between O and 260, we used a distinct phrasing for each pair of numbers, selecting one template from five available templates. This ensures that every unique pair of numbers between 0 and 260 is presented with a consistent phrasing from these templates. We have fixed that typo in the revised version. The different phrasings used are: "Total of numl and num2.". "Add together numl I and num2.", "Calculate num + num2.", " "What the sum of numl I and num2?", and "Put together numl and num2.". The dataset is shuffled ensure randomness and then split into training (80%), validation (10%), and test (10%) sets.
Create the addition dataset in Appendix C.3 with different format We consider numbers in base 10 up to a maximum value of 260. We generate all possible pairs of numbers within this range using combinations with replacement. For each pair, convert the numbers to the specified base and create questions formatted as "numl _num2+" with their corresponding answers. The dataset is then split into training (80%), validation (10%), and test (10%) sets.
Experiments Compute Resources All experiments involving fine-tuning and training from scratch in this paper were conducted on one NVIDIA A6000 GPU with 48GB of video memory. The fine-tuning process required less than 10 hours, while training from scratch took less than days. Other experiments, such as those involving Logit Lens,
completed in less than hour.
Licenses for Existing Assets Open Access to Data and Code. For the following models, we use the checkpoints provided by Huggingface. For all the trained models, we use default hyperparameters during all the training but with different random seeds.
GPT-2-XL: http:://wggggiiioocoooooooooooom Modified MIT License
GPT-2: http://wgggiiilcccoooooriiimmmme Modified MIT License
GPT-J: http::/wwggiigiiiccccoooleeeeees Apacheh License
Phi2: http://hnggiigiiicccccocoooooopp MIT License

---

GPT-3.5 and GPT-4: https://chalgpt.com// or https:/bppaniicommm/mddxoopopppp
PaLM-2 https://ii.gogge/diicvve//////
G
Impact Statement
Our work aims understand the potential large language models in solving arithmetic tasks. Our paper is an interpretability paper and thus we foresee no immediate negative ethical impact. We believe improved understanding and enhancement of LLMs can lead more robust Al systems that are capable of performing complex tasks more reliably. This can benefit areas such as automated data analysis, financial forecasting, and more.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
In the introduction (Section 1), explicitly state the observations and their implications.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations is discussed in Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
This is an interpretation paper without any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide the details of experimental settings in Section F, By following these settings, our results can be reproduced. The detail process about the interpretability methods can be found in Section A.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer:[NG
The goal this paper to understand how LLMs compute addition. We believe the code is not central to our contribution.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
welbbmisssooooicyy.............c for more details. Sttpi////uudes//odeSuue
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
httppppcccppbppccccpecceeecllllt ggwidees////eeeeeooeo Seeiiddee/ooeSbmnin CCodeSubmi ssi asionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:[Yes Yes]
Justification: We show all the details in Section F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We run the experiments with 5 random seeds. Figure 7, we show the plot the error bar with the mean and the standard deviation of the validation accuracy.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The detail the computing resourse provided at the end Section F.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http::///.ee..eeeeeeeeeee http://uuurrs..cpuubeeeeee cttppccicccc/cccccccccccccccclct etw.ii/iieetiiiiieelllleelleeeel
Answer: [Yes]
Justification: The authors have read the NeurIPS Code Ethies and made sure the paper followsthe NeurIPS Code of Ethics in every aspect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The potential societal impact is discussed in Section G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper works simple addition task and dataset. We belive there is such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:[Yes]
Justification:
The details are listed in Section F
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We did not introduce any new assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "d7XC9o3XvH", "reviewer": "Reviewer_YRn2", "summary": "This paper analyzes how language models conduct the task of mathematical addition. Specifically, the paper shows that when performing the mathematical addition task, the internal states of language models exhibit periodical patterns, referred to as \"Fourier Features\" in the paper. The paper then conducts Fourier transformations to analyze such features and shows that (1) high-frequency features are more helpful in reaching the correct prediction (2) filtering out Fourier components causes performance degradation, hence these features are causally important for model predictions (3) such feature arises during pre-training, in the token embedding components. Finally, the paper shows that their findings generalize to other pre-trained LLMs (GPT-J, Phi-2, GPT-4, etc.) since they exhibit similar behaviors.", "questions": "### Presentation Suggestions\n\n- Section 3: I know this is mentioned in the existing literature but it would be nice to define \"modular addition\" before you use it here for the first time.\n- L91: I'm confused by $[L]$. Do you mean a range $[0, L)$ or something similar?\n- L114: \"approximately\" is a weird word in this context. I'm not sure what you are trying to say, but to me, it makes perfect sense even if you remove it (similar occurrence in L121).\n\n### Questions\n\n1. How much do you think the choice of positional embedding has played in the creation of Fourier features? To be more specific, if we use learned positional embedding rather than sinusoidal/rotary positional embedding, do you think you'll still observe these features?\n2. I might have missed it but I don't think there's ever an explanation for why attention only learns high-frequency features, while MLP has both. Do you have any insights?\n3. More of a suggestion: Did you look at stronger open-source models such as the Mixtral ones? It's a pity that you had to do in-context learning for the model to be able to conduct arithmetic addition.", "limitations": "The authors should address the limitations their experiment configurations have created, including but not limited to:\n\n- single token case only\n- small number addition only\n- findings are constrained to a single arithmetic operation, and it's not clear whether they'll generalize to others\n\n(To clarify, not asking for extra experiments, but just acknowledging the limitations)", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This is an in-depth analysis of one specific facade of the internal workings of the LLM.\n2. While this does not seem like the first time Fourier features are studied in neural networks, I still find the analysis method quite interesting.\n3. The writing quality is high. Findings are very nicely presented in a progressive and easy-to-follow way.", "weaknesses": "1. While the study is thorough, I find the scope to be quite narrow -- it's only focused on one specific arithmetic task. I'm skeptical about how much this will help downstream model development.\n2. To make the study feasible, the paper focused on the single-token case, which in turn limits the range of addition to <520. This makes the setup very artificial. That said, this is unfortunately an inherent weakness of many similar analysis papers, and I realize it's tricky to reach a good solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s supportive comments and constructive suggestions. \n\n**Q1: The term \"modular addition\" should be defined before its first use.**\n\nWe have added a definition of \"modular addition\" in Section 3.1 to the revised version of our paper for clarity: \"Modular addition is an arithmetic operation in which the sum of two numbers is divided by a modulus, and the remainder of this division is taken as the result. This operation ensures that the resulting value always falls within the range from zero to one less than the modulus, effectively \"wrapping around\" to zero once the modulus is reached.\"\n\n**Q2: Confusion about the notation [L].**\n\n$[L]$ stands for $\\{1,2,3,4….L\\}$. Hence, $h^{(\\ell)}$ where $\\ell \\in [L]$,  refers to one specific hidden state on the residual stream at layer $\\ell$. We have included this definition in Section 3.1 in the revised version.\t\n\n**Q3: The word \"approximately\" is potentially confusing in the context it's used.**\n\nFor line 114, we agree that the term \"approximately\" might cause confusion. We originally included it to indicate that while outlier components have significantly larger magnitudes, there are still small magnitudes for other components. However, we recognize that this might not be necessary for clarity in this context. Therefore, we have removed \"approximately\" from lines 114 and 121 to improve clarity.\n\n\n**Q4: How much do you think the choice of positional embedding has influenced the creation of Fourier features?**\n\nIn our research, we treat each number as a distinct token with a corresponding token embedding. Consequently, the choice of positional embedding does not directly impact our results and observations. Moreover, it is important to note that the pre-trained embeddings have already shown Fourier features. We believe that when it comes to multiple tokens, these features will also enhance model accuracy. However, this may introduce additional complexity due to the increased number of tokens involved. \n\n**Q5: Why do attention mechanisms tend to learn high-frequency features, whereas MLPs capture both high and low-frequency features?**\n\nThank you for your insightful question. While our paper focuses primarily on the interpretability of pre-trained LLMs, we can provide insights into this observation based on existing research and theoretical understandings.\n\n   - **The key to do modular addition is to compute the trigonometric functions.** As shown in [2], to solve modular addition $a + b \\mod p$, the Transformer model tends to address the equivalent problem $\\arg\\max_{c} \\cos(2\\pi(a + b - c)/p)$. The token embedding in transformers represents inputs as $\\cos(wa)$, $\\cos(wb)$, $\\sin(wa)$, and $\\sin(wb)$ with different frequencies $w$. Using trigonometric functions, the model computes $\\cos(w(a + b))$ and $\\sin(w(a + b))$, which are then processed to get $\\cos(w(a + b - c))$ and $\\sin(w(a + b - c))$.\n  - **Attention can compute trigonometric functions.** Assuming the key and query matrices as identity matrices for simplicity, the attention mechanism can be simplified as $ \\text{softmax}(XX^\\top)V $. As $\\cos(wx)$ and $\\sin(wx)$ have been encoded in $X$, the attention can effectively compute the result of $\\cos(w_1 a) \\cdot \\cos(w_2 b)$ with different frequencies $w_1, w_2$. More evidence can be found in [1] Section 5.2. These shows why attention mechanisms are suitable to compute trigonometric functions.\n\n- **MLP can approximate magnitude.**    According to [3], during arithmetic tasks, the information about numbers $a$ and $b$ is processed by MLPs in the later layers of a neural network. If the magnitudes of $a$ and $b$ are represented in specific entries of the hidden state, say $h[i]$ and $h[j]$, an MLP can compute $w[i]h[i] + w[j]h[j]$, which approximates the magnitude of $a + b$. This demonstrates that MLPs can approximate the magnitude of $a+b$. \n\nIn summary, the tendency of attention mechanisms to learn modular addition (high-frequency features) while MLPs primarily learn approximation (low-frequency features) can be attributed to their different structures. We hope these insights help clarify the observation and provide a deeper understanding of the underlying mechanisms.\n\n**References:**\n\n1. Gu, Jiuxiang, et al. \"Fourier circuits in neural networks: Unlocking the potential of large language models in mathematical reasoning and modular arithmetic.\" arXiv preprint arXiv:2402.09469 (2024).\n2. Nanda, Neel, et al. \"Progress measures for grokking via mechanistic interpretability.\" arXiv preprint arXiv:2301.05217 (2023).\n3. Stolfo, Alessandro, Yonatan Belinkov, and Mrinmaya Sachan. \"A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis.\" arXiv preprint arXiv:2305.15054 (2023).\n\n\n**Q6: Consider using stronger open-source models like Mixtral for arithmetic tasks.**\n\nThank you for your feedback. We have indeed tried our experiments on stronger open-source models. However, we found that the predictions were not controllable without in-context learning. The models sometimes generated intermediate steps or repeated the questions even when we appended ‘Answer is’ at the end of our prompts. In-context learning helped ensure that the model understood the task and provided the correct answers. We believe that in-context learning does not change the model’s underlying strategy to solve the tasks but rather helps the model to respond appropriately.Additionally, we emphasize that for closed-source models, there is also evidence supporting the existence of Fourier features. This further validates our findings and suggests that the observed phenomena are not limited to the specific models we tested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o7w2Fq9zir", "reviewer": "Reviewer_XwiY", "summary": "This paper analyzes how language models perform addition, showing that they use Fourier features. Most of the analyses are done on fine-tuned models. First, periodicity is shown with a logit-lens technique on different layers. Fourier analysis shows this nicely in the frequency space, and shows differences between MLP and attention components, the latter exhibiting only high-freq components. This is connected to \"approximation\" of the answer vs \"classification\" per small modulo. Ablation by removing low/high frequency supports a difference in behavior of the two components. The next section shows that models trained from scratch do not have such Fourier features, but plugging in trained number embeddings recovers this behavior. Finally some analysis shows a similar pattern in prompted models.", "questions": "1. Data split seems to allow cases where the same expression (say 12 + 17) appears in both training and test sets, just with different templates. When looking at generalization, especially when fine-tuning a pretrained model, it could be beneficial to make sure no such overlap happens. \n\n2. Section 3.1 assumes that \"If the models merely retrieve and recombine pieces of information learned during training, certain layers will directly map this information to predictions\". I am not convinced by this. I don't see why 'memorization and recombination' cannot be implemented by the full model, with this gradual refinement process as an implementation of memorization. \n\n3. The periodicity analyses in section 3.2 are striking and insightful. That said, one concern is quantitative analysis - I found the example in figure 4 helpful, but would appreciate a quantitative evaluation of the superposition phenomenon. Also, given the residual stream view, one could say that this is trivial. That is, the final logits are always the sum of earlier ones. So, it would be useful to explain the significance of this finding and whether there is anything special in this case of addition compared to the general behavior of the model. Even better, some control test in a quantitative evaluation could make this point more compelling. \n\n4. Section 3.3 is excellent - it shows which components are causally important for which roles. The experiments are well planned and convincing. One comment I have is on line 183, which says \"the approximation tasks are primarily performed by the MLP modules alone\". If I read Table 1 correctly, then, given that removing low-freq from both Attn and MLP leads to an even greater reduction in accuracy, we can conclude that both MLP and Attn are involved in approximation. This is essentially the same argument made in the paper just a few lines above. \n\n5. Since the analysis in section 3 is done on fine-tuned model, I don't think we can conclude that \"The previous section shows that pre-trained LLMs leverage Fourier features to solve the addition problem\" (line 195). The rest of the section in fact gives evidence to this, but the phrasing at this point should be revised. \n\n6. Figure 3 is very nice but other figures aren't as clean, specifically figure 8 (4-shot) and figure 15 (multiplication), figure 17 (other format), and figure 18 (GPT-J). I think the discussion of each of these cases was a bit too optimistic and naive, and the comparison to the main analyzed case should be made more carefully, and differences highlighted. \n\n7. The related work section is well done. Clarifying the difference from [28] early on in the paper could be useful for the reader. And, one can also mention this paper on arithmetic mechanisms in the related work: Stolfo et al., A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis.", "limitations": "Sufficient discussion.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Seemingly novel observations about language models' mechanism of addition computation. \n2. Clear and to-the-point analyses and visualizations.\n3. Both correlational and causal experiments via ablations to raise hypotheses and test them.", "weaknesses": "1. Missing quantitative analysis of the superposition pattern in section 3.2. See below in questions. \n2. While the main experiments in section 3 (and to a lesser extent also 4.1 and 4.2) are quite thorough and convincing, the ones in 4.3 and in some places in the appendix are less detailed and not sufficiently contrasted with the main results. See below in questions. \n3. Potential problem with datasplit having leakage of expressions from training to test. See below in questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the supportive feedback and thoughtful comments.\n\n***Q1: Why 'memorization and recombination' cannot be implemented by the full model, with this gradual refinement process as an implementation of memorization.***\n\nThank you for your feedback. Our assertion is based on prior research and specific empirical observations detailed in Section 3.1. [1,2] indicate that only a few late layers integrate information for prediction in fact-retrieval tasks. However, our findings in the addition task (Figure 1a) demonstrate that the model utilizes over 20 layers to compute the result, suggesting a more complex process than mere retrieval and recombination. This extensive use of layers aligns with the model performing actual computation, progressively refining the output across multiple layers, contrary to what would be necessary if it were merely memorizing and recombining pre-learned facts.\n\nFurthermore, in Figure 1a, we can see the results gradually refining from far from the correct result to accurate as processing progresses through the layers, which clearly show that the model is performing computation. \n\n[1] Meng, Kevin, et al. \"Locating and editing factual associations in GPT.\" \n\n[2] Merullo, Jack, Carsten Eickhoff, and Ellie Pavlick. \"Language models implement simple word2vec-style vector arithmetic.\"\n\n***Q2: Requests quantitative evaluation of the superposition phenomenon (Figure 4) in periodicity analyses and questions the triviality of findings given the model's architecture.***\n\nThanks for your insightful comments. We emphasize that Figure 4 shows different dimensions in the Fourier space for the final logits. Hence, It is not the sum of the logits of earlier ones. The goal of that experiment (Figure 4) is to provide readers with an intuition of how the final logits are expressed with only a few Fourier components and why sparse Fourier components are sufficient to make the prediction. \n\nIn Figures 2 and 3, we perform quantitative analysis in Fourier space to show that each layer’s contribution to the residual stream is sparse in Fourier space. This Fourier space analysis may not intuitively explain how these outlier components form the final logits. Therefore, we believe that by showing the final logits with only a few Fourier components in number space for this specific example, readers can better understand how these outlier Fourier components contribute to the prediction. \n\n***Q3: Critique of the statement that approximation tasks are primarily performed by MLP modules, suggesting involvement of both attention and MLP modules.***\n\nThank you for your comments. Regarding the statement on line 183, \"the approximation tasks are primarily performed by the MLP modules alone,\" we would like to emphasize our findings as follows:\nIn Table 1, we observe that removing low-frequency components from the attention output alone does not lead to a significant reduction in accuracy. However, when we remove low-frequency components from both the attention and MLP modules, there is a larger reduction in accuracy compared to removing them from the MLP modules alone. This suggests that while both attention and MLP modules are involved in approximation, the MLP modules play a more critical role.\nFurthermore, the minimal impact on accuracy from removing low-frequency components from the attention output indicates that the MLP modules are capable of performing the approximation tasks independently. During prediction, the low-frequency components of the attention output do not contribute significantly, as the MLP modules have the ability to handle the approximation effectively on their own.\nTherefore, we conclude that “the approximation tasks are primarily performed by the MLP modules alone,” as the MLP modules are sufficient to achieve the desired approximation even in the absence of low-frequency components from the attention output.\n\n***Q4: Suggestion to revise phrasing regarding the use of Fourier features in pre-trained LLMs due to the analysis being on a fine-tuned model.***\n\nThanks to the reviewer for pointing that out. You are correct. That is one of our observations in Section 4. We have revised that sentence to “The previous section shows that pre-trained LLMs leverage Fourier features to solve the addition problem after fine-tuning”.\n\n***Q5: Figure 8,15, 17,18 are not as clean as Figure 3***\n\nThank you for your detailed feedback. We have updated the appendix in our paper to incorporate more discussion about the figures you mentioned, which will be reflected in the revised manuscript. We put the full version in the official comment. Here is a summary of the updated discussion. \n\nIn the analysis of various figures, Figure 17 highlights the effective learning of the 'mod 10' task by the last few layers, showing more prominent outlier frequency components compared to Figure 3. Figures 8 and 18 exhibit similar outlier components to Figure 3 but with less clarity, indicating that without fine-tuning, models like 4-shot Phi2 and GPT-J (with respective accuracies of 55.44% and 72.21%) struggle to fully utilize Fourier features for arithmetic tasks. This observation suggests potential improvements in model performance through enhanced leveraging of these features. Meanwhile, Figure 15 reflects the complexity models face with multiplication tasks, where despite data expansion and fine-tuning, accuracies remain relatively lower (74.58%) due to inadequate utilization of Fourier features, as also supported by recent studies from McLeish et al. (2024) and Dziri et al. (2024).\n\n***Q6: Clarify the difference from Nanda et al. (2023) and add Stolfo et al. (2023) to related work.***\n\nThank you for your supportive comments. We have added more discussion for Nanda et al. (2023) and Stolfo et al. (2023) to the related work section, which will be reflected in the revised manuscript.  Due to space limits, we put the fully revised version in the official comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c8zT2izwaQ", "reviewer": "Reviewer_wYe3", "summary": "This paper focuses on understanding the mechanisms the LLMs employ to carry out mathematical operations, in particular sum of two numbers.  It demonstrates: a) Models utilize Fourier features, with different parts of the model utilizing different frequency ranges — attention mechanism uses high frequency components while MLP primarily uses low frequencies.  This is further corroborated by conducting filtering studies.  b) Model embeddings and the pre-training process play a critical role in learning these Fourier features.  Models that are not pre-trained do not seem to be able to induce Fourier features and have lower accuracy.  However, such models when seeded with only embeddings from pre-trained models do learn the Fourier features.", "questions": "Lines 154-156 : “We also illustrated that in one example, the high-frequency components primarily approximate the magnitude, while the low-frequency components are crucial for modular addition tasks, as depicted in Figure 4.“  Is it the other way around?", "limitations": "Limitations sufficiently addressed", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Well written paper on understanding basic mechanism of how LLMs carry out the specific task of addition of small numbers.\n* Interesting findings on pre-training vs task specific training from scratch, demonstrating criticality of model pre-training.", "weaknesses": "* Study is limited to addition and that too of numbers where the final sum can be represented by a single token, leaving a gap in understanding of how addition of larger numbers spanning multiple tokens is achieved.\n* With use of tools / functions in conjunction with the LLMs, mathematical operations are often carried out using a calculator or similar tool.   Thus attempting to improve LLMs' arithmetic abilities may not be very fruitful.  However, do the discoveries made in the paper apply only to number processing by LLMs or can they have broader implications, say on logical reasoning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the supportive feedback and thoughtful comments.\n\n***Q1: With the use of tools / functions in conjunction with the LLMs, mathematical operations are often carried out using a calculator or similar tool. Thus attempting to improve LLMs' arithmetic abilities may not be very fruitful. However, do the discoveries made in the paper apply only to number processing by LLMs or can they have broader implications, say on logical reasoning?***\n\nThank you for your feedback. We agree that using a calculator or similar tool can solve arithmetic problems easily. However, we believe that without understanding numbers and performing basic arithmetic, a model cannot fully grasp more complex concepts in physics or mathematics like humans do. With billions of parameters, LLMs should be capable of easily solving number-related tasks such as arithmetic problems and time-series prediction. Thus, we believe improving LLMs' arithmetic abilities is important.\n\nAdditionally, we emphasize that our observations have broader implications beyond arithmetic tasks. One key finding is that number embeddings after pre-training are sparse in the Fourier space for many pre-trained LLMs without fine-tuning (Figure 14). This suggests that the model inherently uses Fourier features to represent numbers and solve number-related tasks. This insight can inspire future research in several ways:\n- Extending this number embedding strategy to handle larger numbers.\n- Adding regularizers to help models learn high-frequency Fourier features to enhance performance on number-related tasks.\n- Exploring the potential for these Fourier features to benefit logical reasoning and other tasks beyond arithmetic.\n\nBy understanding and leveraging these Fourier features, we can improve LLMs' overall capabilities in number processing and related areas, thereby contributing to their broader applicability and effectiveness. More broader implications and future directions can be found in our response to all reviewers.\n\n***Q2: Lines 154-156 : “We also illustrated that in one example, the high-frequency components primarily approximate the magnitude, while the low-frequency components are crucial for modular addition tasks, as depicted in Figure 4.“ Is it the other way around?***\n\nThank you for pointing out that typo. We have revised it to: “We also illustrated that in one example, the low-frequency components primarily approximate the magnitude, while the high-frequency components are crucial for modular addition tasks, as depicted in Figure 4.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ADeHbVRzn", "reviewer": "Reviewer_mHEL", "summary": "The authors utilize discrete Fourier transform to determine which Fourier components play the most important role in computing the addition of relatively small numbers (< 520) in Large Language Models, such as the GPT-2 family, and, _likely_, Phi-2, GPT-J, and others. They found out that MLP modules of GPT-2 use mostly low-frequency Fourier components to approximate the magnitude of the answer. In contrast, MHA modules utilize high-frequency components to approximate the last digit of the answer. Moreover, the model converges to the correct answer layer by layer.\n\nThe paper shows that this mechanism appears after pretraining the model; the weights of the embedding layer play a special role in it.", "questions": "Questions:\n\n- If I understood correctly, Table 1 contains the ablation experiments only for GPT-2-XL. Why didn't you repeat the experiments from Table 1 for other models? E.g. for Phi2, GPT-2-base, etc.\n- How does the mechanism change when you add the bigger numbers, encoded in several tokens instead of one token?\n- You wrote in 91 lines:\n> ℓ ∈ [L].\n\nDoes it mean, that you use the outputs of all layers, preceding layer number L, for prediction?\n- You wrote at 594 lines:\n> We consider numbers in base 10 up to a maximum value 260. For each pair of numbers between 0 and 260, we generate various phrasings of addition questions and their corresponding answers. The different phrasings used are: “Total of num1 and num2.”, “Add together num1 and num2.”, “Calculate num1 + num2.”, “What is the sum of num1 and num2?”, and “Put together num1 and num2.”. The dataset is shuffled to ensure randomness and then split into training (80%), validation (10%), and test (10%) sets\n\nDoes it mean that train, valid and test could contain the addition questions for the same numbers, but in different phrasing? I.e., could the train contain the phrase “Total of 11 and 12.”, while the test contains “Put together 11 and 12.”?\n\n---\n\nSuggestions:\n\n- GPT-2 is a model with absolute position embeddings. What about other types of position embeddings? (see also my question/suggestion about ablation experiments on Phi2 above). It would be interesting to research the connection of the addition mechanism with the position embedding type. See the paper \"**Positional Description Matters for Transformers Arithmetic**\" by Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang.\n- It would also be interesting to research the connection between the Fourier addition mechanism and outlier dimensions (see, for example, the paper \"**Outlier Dimensions Encode Task-Specific Knowledge**\" by William Rudman, Catherine Chen, Carsten Eickhoff, and many other papers about Outlier Dimensions in transformers).\n\n--- \n\nTypos and presentation issues:\n\n- Figures 1 (b) and (c) are very confusing because the word \"Number\" is closer to the color bar, than the word \"Logits\". So, at first, I didn't understand, that the color means the **Logit value**. As a result, I was confused about these diagrams, and it took time for me to sort things out.\n- You wrote in line 165:\n> We show in Appendix A that this has a simple closed-form solution involving a linear projection.\n\nI didn't find this in Appendix A, can you please, elaborate, on what you meant?\n- You wrote in the description of Figure 13: \n> Visualization of the Fourier component whose period is 520 analysis for the final logits.\n\nCan you please, elaborate, on what did you mean?\n- You wrote in line 204:\n> D is the size of the token embeddings, denote the token embedding for numbers\n\nCan you please, elaborate, on what did you mean?\n\n- What does Fig.5(a) show? Are these some average number embeddings?\n- Why does Fig.6 show the legend with the different colors of cos/sin, while only red bars are visible? Maybe the bars are too narrow, so their colors mix together?", "limitations": "There are two small \"Limitation\" and \"Impact\" sections: Appendix G Limitations, and Appendix H Impact Statement.\n\nThe authors addressed the limitations correctly overall. However, I would add that most of the experiments were devoted to the models from the GPT-2 family; very few experiments have been carried out on other models, so any arguments about other models are weaker. It is also a big limitation of this work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper provides interesting insights into the addition mechanism inside of Large Language models:\n\n- It contains original research about the role of Fourier components in small numbers addition inside of GPT-2 family models;\n- it provides some observations that indicate that the same mechanism may appear in Phi-2, GPT-J, and bigger models;\n- it demonstrates how the roles of MHA and MLP complement each other in this mechanism;\n- it shows the connection between this mechanism and the pretraining of the model, especially the embedding layer.", "weaknesses": "- Most of the experiments were devoted to the models from the GPT-2 family; very few experiments have been carried out on other models, so any arguments about other models are weaker;\n- Only small (< 520) numbers were considered;\n- The experimental setup is questionable (see \"Questions\" part of the review);\n- The paper is not easy to follow. Some figures are confusing, and there are many typos (see \"Questions\" part of the review).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the supportive feedback and comments.\n\n ***Q1: Why not repeat the experiments from Table 1 for other models.***\n\nThe goal of this paper is to understand how pre-trained LLMs solve addition tasks. We conduct most of the experiments, such as Figures 1-5, on GPT-2-XL to provide a deep and comprehensive understanding of its mechanisms in solving these tasks. We focus on GPT-2-XL because it offers a balance between model complexity and interpretability, allowing us to draw meaningful insights.\nTo address the generalizability of our findings, we provide evidence for the existence of Fourier features in other pre-trained LLMs and tasks. Specifically, our Fourier analysis of the pre-trained embeddings for different models (as shown in Figure 14) reveals a consistent sparsity in Fourier space, even without fine-tuning. Additionally, the intermediate logits (Figure 8) and final predictions (Section 4.3) further demonstrate this sparsity across models.\n\nThese results suggest that the observed phenomena are not unique to GPT-2-XL but are likely intrinsic to the architecture of pre-trained LLMs in general. Therefore, while repeating all experiments on every other pre-trained LLM would be ideal, we believe the provided evidence sufficiently supports our claims across different models. This approach balances thoroughness with practicality, ensuring that our main findings are robust and broadly applicable without unnecessary redundancy.\n\n\n***Q2: In 91 lines: $\\ell \\in [L]$. Does it mean that you use the outputs of all layers, preceding layer number L, for prediction?***\n\nNo, $[L]$ stands for $\\{1,2,3,4….L\\}$. Hence, $h^{(\\ell)}$ where $\\ell \\in [L]$,  refers to one specific hidden state on the residual stream at layer $\\ell$.\n\n***Q3: GPT-2 is a model with absolute position embeddings. It would be interesting to research the connection of the addition mechanism with the position embedding type. See the paper \"Positional Description Matters for Transformers Arithmetic\"***\n\nThank you for your comments. In our paper, each number is treated as one token, corresponding to one token embedding (as mentioned in line 58). Therefore, the type of positional embedding does not change our results and observations.\nRegarding the paper \"Positional Description Matters for Transformers Arithmetic,\" it's important to note that in the footnote of page 3, they mention: “In this paper, for every dataset used, a space is inserted before each digit. This ensures the tokenizer tokenizes each digit as an individual token.” This approach allows them to leverage the position of each digit to solve arithmetic tasks, which is not a common way to tokenize numbers in the GPT-2 model in real-world scenarios.\n\n\n***Q4: It would also be interesting to research the connection between the Fourier addition mechanism and outlier dimensions (paper \"Outlier Dimensions Encode Task-Specific Knowledge\").***\n\nThank you for your comments. Outlier dimensions are the dimensions in the embedding space that exhibit large variance. In our work, we demonstrate that there are outlier components in the Fourier space that play a significant role in arithmetic tasks. We emphasize that the outlier are in two different space for these two cases. \n\n***Q5: Figures 1 (b) and (c) are very confusing***\n\nThanks for your feedback. We have revised Figures 1(b) and 1(c) to improve clarity by adjusting the placement of labels and color bars.\n\n---\n***For Q6 - Q9, Reviewer mHEL requested further elaboration on the processes, figures, and concepts discussed in the paper. Due to space constraints, we have addressed these details in the official comment.***\n\n\n---\n\n***Q10: Why does Fig.6 show the legend with the different colors of cos/sin, while only red bars are visible?***\n\nThanks for noticing that. We used the same plotting method in Figure 3 and Figure 6. In Figure 3, we use the legend with different colors to show whether the outlier component is sine or cosine. However, for Figure 6, there are no outlier Fourier components and we do not need that legend. We have removed the legend in Figure 6 in the revised version of our paper.\n\n\n***Q11: The authors addressed the limitations correctly overall. However, I would add that most of the experiments were devoted to the models from the GPT-2 family; very few experiments have been carried out on other models, so any arguments about other models are weaker. It is also a big limitation of this work.***\n\nWe acknowledge that this paper primarily focuses on the GPT-2 family. However, we believe that this already constitutes a significant step forward, highlighting our contribution rather than a limitation. As discussed in the related work section, prior studies [1,2,3,4] mainly concentrated on training shallow networks from scratch and analyzing their performance in modular addition. To the best of our knowledge, this is the first work analyzing how pre-trained LLMs solve addition.\nWe delve deeply into understanding the mechanisms by which GPT-2 leverages Fourier features. We then provide evidence of the existence of these Fourier features in other closed-source and open-source pre-trained LLMs. We demonstrate similar Fourier features in both pre-trained number embeddings and the intermediate hidden states. Additionally, these outlier Fourier components in other models exhibit almost the same periods as those in GPT-2. Based on these findings, we conclude that they leverage Fourier features to solve addition.\n\n[1] Morwani, Depen, et al. \"Feature emergence via margin maximization: case studies in algebraic tasks.\"\n\n[2]Nanda, Neel, et al. \"Progress measures for grokking via mechanistic interpretability.\"\n \n[3]Zhong, Ziqian, et al. \"The clock and the pizza: Two stories in mechanistic explanation of neural networks.\" \n\n[4]Gu, Jiuxiang, et al. \"Fourier circuits in neural networks: Unlocking the potential of large language models in mathematical reasoning and modular arithmetic.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Response to All Reviewers:**

We thank reviewers [R1(YRn2), R2(XwiY), R3(wYe3), R4(mHEL)] for their thoughtful and highly supportive feedback! We are glad that the reviewers found the analysis method interesting and  [R1, R3], the observations about the application of Fourier features in language models performing addition tasks insightful and novel [R2, R4], the presentation of our findings easy to follow [R1, R2], and the experimental results compelling, especially regarding the roles of Fourier components in model behavior and the importance of model pretraining [R3, R4]. 


***Q1: Limitations pointed out by the reviewers: focusing on a single arithmetic task (addition) and the single-token case limiting the range to numbers less than 520.***

1. **Scope of Study**: As an interpretability paper, we focused on a single-token case with numbers less than 520 to provide a controlled and clear analysis environment. This setup allows us to dive deep into understanding the fundamental mechanisms at play. It is a starting point to inspire further research and improvements in the model’s performance on arithmetic tasks. These limitations are inherent to many interpretability studies. For example, [1,2] focus on modular addition, a simpler task than general addition, and their analyses are conducted on shallow neural networks or transformers. In contrast, our study investigates pre-trained LLMs on the addition task, offering insights into more complex and realistic scenarios.

2. **Broader Implications**: We emphasize that the observations on pre-trained number embeddings are not limited to addition tasks. The number embeddings after pre-training are sparse in the Fourier space for many pre-trained LLMs without fine-tuning (Figure 14). This suggests that the model does not learn to use Fourier features to embed numbers just for one task, implying that these learned Fourier features will benefit other number-related tasks as well. In this paper, we focus on addition to help readers understand how these Fourier features assist the model in solving addition problems. We also provide evidence that the model uses these Fourier features in multiplication (Figure 15, Section C.2).

3. **Future Research Directions**: The insights and observations from this paper can inspire many interesting future directions:
    - Exploring the potential of adding a regularizer to help models learn high-frequency Fourier features to enhance performance on number-related tasks.
    - Investigating the training dynamics of how models learn to use Fourier features to represent numbers and solve arithmetic problems during pre-training.
    - Understanding why models tend to use these Fourier features to solve problems.
    - Examining whether models still utilize these Fourier features when dealing with numbers tokenized using sub-word tokenizers.
    - Investigating the use of digit-wise tokenizers, such as in Llama, to see if the model leverages Fourier features for arithmetic tasks.
    - Developing strategies to help the model learn Fourier features in both cases mentioned above to improve performance on arithmetic tasks.
    - Improving the model's performance on larger numbers or decimal numbers by finding strategies to correctly embed numbers.

By highlighting these points, we hope to demonstrate the broader implications of our findings and the potential for future research inspired by our work.


**References:**

1. Morwani, Depen, et al. "Feature emergence via margin maximization: case studies in algebraic tasks." arXiv preprint arXiv:2311.07568 (2023).
2. Nanda, Neel, et al. "Progress measures for grokking via mechanistic interpretability." arXiv preprint arXiv:2301.05217 (2023).


***Q2: Concern about data overlap in training and test sets due to the use of the same mathematical expressions with different templates.***

We apologize for the confusion. In line 595, the mention of generating "various phrasings" for addition questions was a typo. In the actual experiments, we used a distinct phrasing for each pair of numbers, selecting one template from five available templates. This ensures that every unique pair of numbers between 0 and 260 is presented with a consistent phrasing from these templates. We have fixed that typo in the revised version.

Meta-review: The paper analyzes how LMs perform addition using Fourier features and with logit lens as well. The work makes a solid niche contribution in our understanding of LMs mechanism of computing addition. While the study is conducted on a very small dataset and mainly the addition property of LMs is analyzed, the work is solid and it contributes to own understanding of LMs.
Decision: Accept (poster)