Pre-trained Large Language Models Use Fourier Features to Compute Addition

Authors: tianyi  zhou
deqing  fu
vatsal  sharan
robin  jia
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 01:26:18

Abstract: Pre-trained large language models (LLMs) exhibit impressive mathematical reasoning capabilities, yet how they compute basic arithmetic, such as addition, remains unclear. 
This paper shows that pre-trained LLMs add numbers using Fourier features---dimensions in the hidden state that represent numbers via a set of features sparse in the frequency domain. 
Within the model, MLP and attention layers use Fourier features in complementary ways: MLP layers primarily approximate the magnitude of the answer using low-frequency features, while attention layers primarily perform modular addition (e.g., computing whether the answer is even or odd) using high-frequency features.
Pre-training is crucial for this mechanism: models trained from scratch to add numbers only exploit low-frequency features, leading to lower accuracy.
Introducing pre-trained token embeddings to a randomly initialized model rescues its performance.
Overall, our analysis demonstrates that appropriate pre-trained representations (e.g., Fourier features) can unlock the ability of Transformers to learn precise mechanisms for algorithmic tasks.

Main paper: 
 introduction: 
 Mathematical problem solving has become a crucial task for evaluating the reasoning capabilities of large language models (LLMs) [20, 7, 23, 13] . While LLMs exhibit impressive mathematical abilities [34, 17, 1, 45, 40, 4, 11] , it remains unclear how they perform even basic mathematical tasks. Do LLMs apply mathematical principles when solving math problems, or do they merely reproduce memorized patterns from the training data?In this work, we unravel how pre-trained language models solve simple mathematical problems such as "Put together 15 and 93. Answer: ". Prior work has studied how Transformers, the underlying architecture of LLMs, perform certain mathematical tasks. Most studies [5, 14, 42, 2, 12, 28, 18, 36] focus on Transformers with a limited number of layers or those trained from scratch; [19] analyzes how the pre-trained GPT-2-small performs the greater-than task. Our work focuses on a different task from prior interpretability work-integer addition-and shows that pre-trained LLMs learn distinct mechanisms from randomly initialized Transformers.In §3, we show that pre-trained language models compute addition with Fourier features-dimensions in the hidden state that represent numbers via a set of features sparse in the frequency domain. First, we analyze the behavior of pre-trained LLMs on the addition task after fine-tuning, which leads to almost perfect accuracy on the task. Rather than merely memorizing answers from the training data, the models progressively compute the final answer layer by layer. Next, we analyze the contributions of individual model components using Logit Lens [3] . We observe that some components primarily approximate the answer-they promote all numbers close to the correct answer in magnitudewhile other components primarily classify the answer modulo m for various numbers m. Then, we use Fourier analysis to isolate features in the residual stream responsible for the low-frequency "approximation" and high-frequency "classification" subtasks. Identifying these features allows us to precisely ablate the ability of the model to perform either approximation or classification by applying a low-pass or high-pass filter, respectively, to the outputs of different model components. We find that MLP layers contribute primarily to approximation, whereas attention layers contribute primarily to classification.In §4, we show that pre-training is crucial for learning this mechanism. The same network trained from scratch with random initialization not only shows no signs of Fourier features, but also has lower accuracy. We identify pre-trained token embeddings as a key source of inductive bias that help the pre-trained model learn a more precise mechanism for addition. Across the pre-trained token embeddings of many different pre-trained models, Fourier analysis uncovers large magnitudes of components with periods 2, 5, and 10. Introducing pre-trained token embeddings when training the model from scratch enables the model to achieve perfect test accuracy. Finally, we show that the same Fourier feature mechanism is present not only in models that were pre-trained and then fine-tuned, but also in frozen pre-trained LLMs when prompted with arithmetic problems.Overall, our work provides a mechanistic perspective on how pre-trained LLMs compute addition through the lens of Fourier analysis. It not only broadens the scope from only investigating few-layer Transformers trained to fit a particular data distribution to understanding LLMs as a whole, but also hints at how pre-training can lead to more precise model capabilities. 
problem setup: 
 Task and Dataset. We constructed a synthetic addition dataset for fine-tuning and evaluation purposes. Each example involves adding two numbers ≤ 260, chosen because the maximum number that can be represented by a single token in the GPT-2-XL tokenizer is 520. For each pair of numbers between 0 and 260, we randomly sample one of five natural language question templates and combine it with the two numbers. The dataset is shuffled and then split into training (80%), validation (10%), and test (10%) sets. More details are provided in Appendix F. In Appendix C.3, we show our that results generalize to a different dataset formatted with reverse Polish notation.Model. Unless otherwise stated, all experiments focus on the pre-trained GPT-2-XL model that has been fine-tuned on our addition dataset. This model, which consists of 48 layers and approximately 1.5 billion parameters, learns the task almost perfectly, with an accuracy of 99.74% on the held-out test set. We examine other models in §4.2 and §4.3.Transformers. We focus on decoder-only Transformer models [41] , which process text sequentially, token by token, from left to right. Each layer ℓ in the Transformer has an attention module with output Attn (ℓ) and an MLP module with output MLP (ℓ) . Their outputs are added together to create a continuous residual stream h [9] , meaning that the token representation accumulates all additive updates within the residual stream, with the representation h (ℓ) in the ℓ-th layer given by: h (ℓ) = h (ℓ-1) + Attn (ℓ) + MLP (ℓ) .(EQUATIONThe output embedding W U projects the residual stream to the space of the vocabulary; applying the softmax function then yields the model's prediction. We provide formal definitions in Appendix A.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations is discussed in Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
language models solve addition with fourier features: 
 In this section, we analyze the internal mechanisms of LLMs when solving addition tasks, employing a Fourier analysis framework. We first show that the model initially approximates the solution before iteratively converging to the correct answer ( §3.1). We then show that the model refines its initial approximation by computing the exact answer modulo 2, 5, and 10, employing Fourier components of those same periods ( §3.2). Finally, we demonstrate through targeted ablations that the identified Fourier components are causally important for the model's computational processes ( §3.3). Specifically, we show that MLP layers primarily approximate the magnitude of the answer, using lowfrequency features, while attention layers primarily perform modular addition using high-frequency components. Our first goal is to understand whether the model merely memorizes and recombines pieces of information learned during training, or it performs calculations to add two numbers.Extracting intermediate predictions. To elucidate how LLMs perform computations and progressively refine their outputs towards the correct answer, we extract model predictions at each layer from the residual stream. Let L denote the number of layers. Using the Logit Lens method [3] , instead of generating predictions by computing logits W U h (L) , predictions are derived through W U h (ℓ) where ℓ ∈ [L]. We compute the accuracy of the prediction using each intermediate state h (ℓ) . If the models merely retrieve and recombine pieces of information learned during training, certain layers will directly map this information to predictions. For instance, [25] demonstrates that there is a specific MLP module directly maps a country to its capital.LLMs progressively compute the final answers. Figure 1a instead shows that the model progressively approaches the correct answer, layer by layer. The model is capable of making predictions that fall within the range of ±2 and ±10 relative to the correct answer in the earlier layers, compared to the exact-match accuracy. This observation implies that the Transformer's layer-wise processing structure is beneficial for gradually refining predictions through a series of transformations and updates applied to the token representations.Logits for MLP and attention have periodic structures. We now analyze how each MLP and attention module contributes to the final prediction. We transform the output of the attention and MLP output at layer ℓ into the token space using W U Attn (ℓ) and W U MLP (ℓ) at each layer, thereby obtaining the logits L for each MLP and attention module. We use the running example "Put together 15 and 93. Answer: 108" to demonstrate how the fine-tuned GPT-2-XL performs the computation. As illustrated in Figure 1b and Figure 1c , both the MLP and attention modules exhibit a periodic pattern in their logits across the output number space, e.g., the MLP in layer 33, outlined in green, promotes all numbers that are congruent to 108 mod 2 (in Figure 20 in the appendix, we zoom into such layers to make this clearer). Overall, we observe two distinct types of computation within these components. Some components predominantly assign a high weight to numbers around the correct answer, which we term approximation. Meanwhile, other components predominantly assign a high weight to all numbers congruent to a + b mod c for some constant c, which we term classification.Logits for MLP and attention are approximately sparse in the Fourier space. It is natural to transform the logits into Fourier space to gain a better understanding of their properties such as the periodic pattern. We apply the discrete Fourier transform to represent the logits as the sum of MLP . The component with period 2 has the largest magnitude, aligning with the observations in Figures 1b and 20a . (b) The logits in Fourier space for the attention output of the 40-th layer, i.e., LAttn . The components with periods 5 and 10 have the largest magnitude, aligning with the observations in Figures 1c and 20b .sine and cosine waves of different periods: the k-th component in Fourier space has period 520/k and frequency k/520 (see Appendix A for more details). Let L denote the logits in Fourier space. Figure 2 shows the Fourier space logits for two layers from Figure 1b and Figure 1c that have a clear periodic pattern. We find that the high-frequency components in Fourier space, which we define as components with index greater or equal to 50, are approximately sparse as depicted in Figure 2 . This observation aligns with [28] , which found that a one-layer Transformer utilizes particular Fourier components within the Fourier space to solve the modular addition task.In Figure 3 , we show that similar sparsity patterns in Fourier space hold across the entire dataset. We compute the logits in Fourier space for the last 15 layers, i.e., L We select the top-5 Fourier components of L (L) based on their magnitudes and transfer them back to logits in number space via the inverse discrete Fourier transform (Figure 4a ). The large-period (low-frequency) components approximate the magnitude while the small-period (high-frequency) components are crucial for modular addition. Figure 4b shows that aggregating these 5 waves is sufficient to predict the correct answer.Why is high-frequency classification helpful? The Fourier basis comprises both cos and sin waves (see Definition A.3). By adjusting the coefficients of cos and sin, the trained model can manipulate the phase of the logits in Fourier space (number shift in number space), aligning the peak of the wave more closely with the correct answer. As shown in Figure 4a , consider a wave with a period of 2. Here, the peak occurs at every even number in the number space, corresponding to the mod 2 task. In contrast, for components with a large period such as 520, the model struggles to accurately position the peak at 108 (also see Figure 14 in the appendix for the plot of this component with period 520 in the full number space). This scenario can be interpreted as solving a "mod 520" task-a classification task among 520 classes-which is challenging for the model to learn accurately. Nevertheless, even though the component with a period of 520 does not solve the "mod 520" task precisely, it does succeed in assigning more weight to numbers near 108. The classification results from the high-frequency components can then provide finer-grained resolution to distinguish between all the numbers around 108 assigned a large weight by the lower frequencies. Due to this, the lowfrequency components need not be perfectly aligned with the answer to make accurate predictions.In the previous section, we demonstrated that there are outlier Fourier components in the logits generated by both the MLP and attention modules, as shown in Figure 3 . We also illustrated that, in one example, the low-frequency components primarily approximate the magnitude, while the high-frequency components are crucial for modular addition tasks, as depicted in Figure 4 . In this section, through an ablation study conducted across the entire test dataset, we show that both types of components are essential for correctly computing sums. Moreover, we reveal that the MLP layers primarily approximate the magnitude of the answer using low-frequency features, whereas the attention layers are responsible for modular addition using high-frequency features.Filtering out Fourier components. To understand the role various frequency components play for the addition task, we introduce low-pass and high-pass filters F. For an intermediate state h, and a set of frequencies Γ = {γ 1 , . . . , γ k }, the filter F(h; Γ) returns the vector h that is closest in L 2 distance to h subject to the constraint that the Fourier decomposition of W U h at every frequency γ i is 0. We show in Appendix A that this has a simple closed-form solution involving a linear projection. We then apply either a low-pass filter by taking Γ to be all the components whose frequencies are greater than the frequency of the τ -th component for some threshold τ (i.e., removing high-frequency components), and a high-pass filter by taking Γ to be all the components whose frequencies are less Different roles of frequency components in approximation and classification tasks. We evaluated the fine-tuned GPT-2-XL model on the test dataset with different frequency filters applied to all of the output of MLP and attention modules. The results, presented in Table 1 , indicate that removing lowfrequency components from attention modules or high-frequency components from MLP modules does not impact performance. This observation suggests that attention modules are not crucial for approximation tasks, and MLP modules are less significant for classification tasks.Eliminating high-frequency components from attention results in a noticeable decrease in accuracy. Furthermore, removing high-frequency components from both the attention and MLP modules simultaneously leads to an even greater reduction in accuracy. This finding corresponds with observations from Figure 1b ,c and Figure 3 , which indicate that both MLP and attention modules are involved in classification tasks due to the presence of high-frequency components in the logits.As shown in Table 1 , the approximation tasks are primarily performed by the MLP modules, with contributions from the attention modules as well.The errors induced by these ablations align with our mechanistic understanding. Ablating lowfrequency parts of MLPs leads to off-by 10, 50, and 100 errors: the model fails to perform the approximation subtask, though it still accurately predicts the unit digit. Conversely, ablating highfrequency parts of attention leads to small errors less than 6 in magnitude: the model struggles to accurately predict the units digit, but it can still estimate the overall magnitude of the answer. See Figure 21 in the Appendix for more details. These observations validate our hypothesis that low-frequency components are crucial for approximation, while high-frequency components are vital for classification. The primary function of MLP modules is to approximate the magnitude of outcomes using low-frequency components, while the primary role of attention modules is to ensure accurate classification by determining the correct unit digit.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: The goal of this paper is to understand how LLMs compute addition. We believe the code is not central to our contribution. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
effects of pre-training: 
 The previous section shows that pre-trained LLMs leverage Fourier features to solve the addition problem. Now, we study where the models' reliance on Fourier features comes from. In this section, we demonstrate that LLMs learn Fourier features in the token embeddings for numbers during pre-training. These token embeddings are important for achieving high accuracy on the addition task: models trained from scratch achieve lower accuracy, but adding just the pre-trained token embeddings fixes this problem. We also show that pre-trained models leverage Fourier features not only when fine-tuned, but also when prompted.Number embedding exhibits approximate sparsity in the Fourier space. Let W E ∈ R p×D , where p = 521 and D is the size of the token embeddings, denote the token embedding for numbers. We apply the discrete Fourier transform to each column of W E to obtain a matrix V ∈ R p×D , where each row represents a different Fourier component. Then we take the L 2 norm of each row to yield a p-dimensional vector. Each component j in this vector measures the overall magnitude of the j-th Fourier component across all the token embedding dimensions. Figure 5a shows the magnitude of different Fourier components in the token embedding of GPT-2-XL. We see that the token embedding has outlier components whose periods are 2, 2.5, 5, and 10. Therefore, similar to how the model uses different Fourier components to represent its prediction (as shown in Section 3.2), the token embeddings represent numbers with different Fourier components. Figure 15 in the Appendix shows that the token embeddings of other pre-trained models have similar patterns the Fourier space. This suggests that Fourier features are a common attribute in the token embedding of pre-trained LLMs.In Figure 5b , we use t-SNE and k-means to visualize the token embedding clustering. We can see that numbers cluster not only by magnitude but also by their multiples of 10. To understand the necessity of Fourier features for the addition problem, we trained the GPT-2-XL model from scratch on the addition task with random initialization. After convergence, it achieved only 94.44% test accuracy (recall that the fine-tuned GPT-2-XL model achieved 99.74% accuracy).Fourier features are learned during pre-training. Figure 6 shows that there are no Fourier features in the intermediate logits of the GPT-2-XL model trained from scratch on the addition task. Furthermore, Figure 7a shows that the token embeddings also have no Fourier features. Without leveraging Fourier features, the model merely approximates the correct answer without performing modular addition, resulting in frequent off-by-one errors between the prediction and the correct answer (see details in Figure 23 ). The number embedding in Fourier space for GPT-2-XL trained from scratch. There are no high-frequency outlier components, in contrast with the pre-trained embeddings (Figure 5a ). (b) Validation accuracy of GPT-2-small trained from scratch either with or without pre-trained token embeddings. We show the mean and the standard deviation of the validation accuracy across 5 random seeds. GPT-2-small with pre-trained token embedding consistently achieves 100% accuracy, while GPT-2-small without pre-trained token embedding only achieves less than 60% accuracy.Pre-trained token embeddings improve model training. We also trained GPT-2-small, with 124 million parameters and 12 layers, from scratch on the addition task. GPT-2-small often struggles with mathematical tasks [26] . This model achieved a test accuracy of only 53.95% after convergence. However, when we freeze the token embedding layer and randomly initialize the weights for all other layers before training on the addition task, the test accuracy increases to 100%, with a significantly faster convergence rate. This outcome was consistently observed across five different random seeds, as illustrated in Figure 7b . Following Section 3.2, to validate that the model learn to leverage the Fourier feature to solve addition, we analyze the logits in Fourier space for all the test data across all 12 layers In Figure 8 , we can clearly observe that, with solely the pre-trained number embedding, the Fourier features appear in the MLP and attention modules' output for most layers. This demonstrates that given the number embeddings with Fourier features, the model can effectively learn to leverage these features to solve the addition task.Finally, we ask whether larger language models use similar Fourier features during prompting.Pre-trained LLMs use Fourier features to compute addition during in-context learning. We first test on the open-source models GPT-J [43] with 6B parameters, and Phi-2 [21] with 2.7B parameters on the test dataset. Without in-context learning, the model cannot perform addition tasks. Therefore, we use 4-shot in-context learning to test its performance. Their absolute errors are predominantly multiples of 10: 93% of the time for GPT-J, and 73% for Phi-2 . Using the Fourier analysis framework proposed in Section 3.2, we demonstrate that for Phi-2 and GPT-J, the outputs of MLP and attention modules exhibit approximate sparsity in Fourier space across the last 15 layers (Figure 9 and Figure 19 ). This evidence strongly suggests that these models leverage Fourier features to compute additions.Closed-source models exhibit similar behavior. We study the closed-source models GPT-3.5 [33] , GPT-4 [34] , and PaLM-2 [16] . While we cannot analyze their internal representations, we can study whether their behavior on addition problems is consistent with reliance on Fourier features. Since closed-source LLMs are instruction tuned and perform well without in-context learning, we conduct error analysis with 0-shot. Most absolute errors by these models are also multiples of 10: 100% of the time for GPT-3.5 and GPT-4, and 87% for PaLM-2. The similarity in error distribution to that of open-source models leads us to hypothesize that Fourier features play a critical role in their computational mechanism. 
related work: 
 Learning mathematical tasks. Previous studies primarily explore what pre-trained LMs can achieve on arithmetic tasks, with less emphasis on the underlying mechanisms [29, 37] . For instance, [22] demonstrates that small Transformer models can effectively learn arithmetic by altering the question format and utilizing a scratchpad method [30] . [19] identifies activation patterns for the "greater-than" operation in GPT-2, and [5] focuses on the enumeration and selection processes in GCD computation.In this paper, we dive into the specific roles of MLP and attention layers in solving mathematical tasks. Our research analyzes these components' distinct contributions to integer addition tasks.Mechanisms of pre-trained LMs. Recent studies have significantly advanced our understanding of the underlying mechanisms of pre-trained Transformer models. For instance, research on "skill neurons" by [44] and "knowledge neurons" by [8] underscores the development of specialized neural components that encode task-specific capabilities or hold explicit factual information in the pre-trained LMs, enhancing model performance on related tasks. [25] and [15] discuss how MLPs and FFNs transform and update token representations for general language tasks. In contrast, we show that the pre-trained LMs use multiple layers to compute addition by combining the results of approximation and classification. Additionally, [46] demonstrated the capacity of GPT-2 to consolidate similar information through pre-training in the model weights, which aligns with our observations on the importance of pre-training in developing effective number embedding and arithmetic computation strategies in LMs.Fourier features in Neural Networks. Fourier features are commonly observed in image models, particularly in the early layers of vision models [32, 31, 10] . These features enable the model to detect edges, textures, and other spatial patterns effectively. Recently, Fourier features have been noted in networks trained for tasks that allow cyclic wraparound, such as modular addition [28, 27] , general group compositions [6] , or invariance to cyclic translations [38] . [28] demonstrates that learning Fourier features can induce 'grokking' [35] . Furthermore, [24] provides a mathematical framework explaining the emergence of Fourier features when the network exhibits invariance to a finite group. We extend these insights by observing Fourier features in tasks that do not involve cyclic wraparound. [39] found that by selecting problem-specific Fourier features, the performance of MLPs can be improved on a computer vision-related task. 
conclusion: 
 In this paper, we provide a comprehensive analysis of how pre-trained LLMs compute numerical sums, revealing a nuanced interplay of Fourier features within their architecture. Our findings demonstrate that LLMs do not simply memorize answers from training data but actively compute solutions through a combination of approximation and classification processes encoded in the frequency domain of their hidden states. Specifically, MLP layers contribute to approximating the magnitude of sums, while attention layers contribute to modular operations.Our work also shows that pre-training plays a critical role in equipping LLMs with the Fourier features necessary for executing arithmetic operations. Models trained from scratch lack these crucial features and achieve lower accuracy; introducing pre-trained token embeddings greatly improves their convergence rate and accuracy. This insight into the arithmetic problem-solving capabilities of LLMs through Fourier features sets the stage for potential modifications to training approaches. By imposing specific constraints on model training, we could further enhance the ability of LLMs to learn and leverage these Fourier features, thereby improving their performance in mathematical tasks. 
limitations: 
 We note that our contributions are limited by the size of the dataset. As the maximum number that can be represented by one token for GPT-2-XL is 520, we analyze on the dataset whose operands are less than 260. However, as the Fourier features commonly exist in many different pre-trained models as shown in Section 4, we believe different models still use Fourier features, possibly with a more complicated strategy.Throughout the model, h undergoes only additive updates (Eq. ( 1)), creating a continuous residual stream [9] , meaning that the token representation h accumulates all additive updates within the residual stream up to layer t.To analyze the logits in Fourier space, we give the formal definition of the Fourier basis as follows: Definition A.3 (Fourier Basis). Let p denote the size of the number space. Let -→ x := (0, 1, . . . , (p -1)). Let ω k := 2πk p-1 . We denote the normalized Fourier basis F as the p × p matrix: Remark B.2. The result of the optimization problem defined in Definition B.1 is the projection of x to the null space of BF W U . Let N (BF W U ) denote the null space of BF W U . We haveF :=               1 p-1 • - → 1 2 p-1 • sin ω 1 - → x 2 p-1 • cos ω 1 - → x 2 p-1 • sin ω 2 - → x . . . 2 p-1 • cos ω (p-1)/2 - → x               ∈ R p×p The first component F [0] is defined as a constant component. For i ∈ [0, p -1], F [i] isF γ (x) = N (BF W U ) • N (BF W U ) ⊤ • x ⊤For the single-pass filter, we only retrain one Fourier component and analyze how this component affects the model's prediction. The residual stream is then updated as follows:h (ℓ) = h (ℓ-1) + F γ (Attn (ℓ-1) ) + F γ (MLP (ℓ-1) )We evaluated the fine-tuned GPT-2-XL model on the addition dataset with the Fourier components period 520 and 2. Given that T k := V -1 k (Definition A.3), we retained only the Fourier components with γ = 1 and 260, respectively.As shown in Figure 11a , with only one frequency component, whose period is 2, the model accurately predicts the parity with 99.59% accuracy. As depicted in Figure 11b , with a single frequency component of period 520, the model fails to accurately predict with 96.51% accuracy. We consider the frequency component with a period of 2 as the model's prediction for the mod 2 task, and the frequency component with a period of 520 as its prediction for the mod 520 task. Figures 11 and 12 suggest that the model effectively learns the mod 2 task, as it involves a two-class classification, but struggles with the mod 520 task, which requires classifying among 520 classes. As the model does not need to be trained to converge to the optimal for these low-frequency components as explained at the end of Section 3.2, predicting with the period-520 component leads to predictions that normally distributed around the correct answers. The Fourier components with larger periods present greater difficulty in solving the corresponding modular addition task compared to those with smaller periods. As demonstrated in Figure 12 , components with large periods serve primarily as approximations of the correct answer. Consequently, we categorize the Fourier components into low-frequency and high-frequency groups. The low-frequency components approximate the magnitude of the answer, whereas the high-frequency components are employed to enhance the precision of the predictions.In reference to Figure 4 , to elucidate the contribution of these distinct Fourier components to our final prediction and the rationale behind their separation, consider the example: "Put together 15 and 93. Answer: 108". We selected the top-10 Fourier components of L (L) based on their magnitudes and converted them back to logits in the numerical space by multiplying with F ⊤ . We plotted the components with components index less than 50 in Figure 13a and those with components index greater than 50 in Figure 13b . Leveraging the constructive and destructive inference for different waves, the components with low periods assign more weight to the correct answer, 108, and less weight to numbers close to 108. These high-frequency (low-period) components ensure the prediction's accuracy at the unit place. For the low-frequency (large-period) components, the model fails to precisely learn the magnitude of the factor between the cos and sin components, which results in failing to peak at the correct answer. Thus, the low-frequency (large-period) components are used to approximate the magnitude of the addition results. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We run the experiments with 5 random seeds. In Figure 7 , we show the plot the error bar with the mean and the standard deviation of the validation accuracy. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The detail of the computing resourse is provided at the end of Section F.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The potential societal impact is discussed in Section G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Our paper works on simple addition task and dataset. We belive there is no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The details are listed in Section F Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We did not introduce any new assets in this paper.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We first show that other pre-trained LMs also have Fourier features in their token embedding for the numbers [0, 520]. A key question is whether pre-trained models utilize Fourier Features solely for solving addition tasks or if they generalize to other arithmetic tasks. We hypothesize the latter, knowing that numbers are represented by their Fourier features in the token embeddings after pre-training. Consequently, this Fourier representation should be leveraged in a variety of number-related tasks. To validate this hypothesis, we perform a Fourier analysis on the GPT-2-XL model fine-tuned for the multiplication task.Considering a maximum number of 520 for multiplication would result in an insufficient dataset size. Therefore, we set the maximum allowable product to 10000.For each pair of numbers where the product does not exceed this limit, we used a distinct phrasing for each pair of numbers, selecting one template from five available templates. This ensures that every unique pair of numbers between 0 and 260 is presented with a consistent phrasing from these templates. We have fixed that typo in the revised version. The different phrasings used include: "What is the product of num1 and num2?", "Find the product of num1 multiplied by num2.", "Calculate num1 times num2.", "num1 multiplied by num2 equals what?", and "Multiplication of num1 with num2." The dataset is then shuffled to ensure randomness and split into training (80%), validation (10%), and test (10%) sets. We finetune the model for 25 epochs with a learning rate of 1e -4. Upon convergence, the validation accuracy reaches 74.58%.As the primary objective is to determine whether the Fourier features are utilized in tasks other than addition, Figure 16 displays the logits in Fourier space for each layer, as in Figure 3 . It is evident that the logits are sparse in Fourier space. To demonstrate that our observations are not confined to a specific description of the mathematical problem, we conducted experiments on another format of addition problem and obtained consistent results. From Figure 17 , we can see that there are also periodic structures in the intermediate logits. From Figure 18 , we can also see the Fourier features for the MLP and attention output. These two experiments validate that our observations are not confined to a specific format of the addition problems. Using the Fourier analysis framework proposed in Section 3.2, we demonstrate that for GPT-J, the outputs of MLP and attention modules exhibit approximate sparsity in Fourier space across the last 15 layers (Figure 19 ) We selected the layers that clearly show the periodic pattern in Figure 1b and Figure 1c and plot their logits in Figure 20 . Figure 21 illustrates that the errors resulting from the ablation study (Section 3.3) correspond with our theoretical insights. Removing low-frequency parts from the MLP results in errors such as off-by 10, 50, and 100. Without these low-frequency components, the MLP is unable to accurately approximate, although it still correctly predicts the unit digit. In contrast, removing high-frequency components from the attention modules results in smaller errors, all less than 6 in magnitude. These findings support our statement that low-frequency components are essential for accurate approximation, whereas high-frequency components are key for precise classification tasks. Consequently, the primary function of MLP modules is to approximate numerical magnitudes using low-frequency components, and the essential function of attention modules is to facilitate precise classification by identifying the correct unit digit. Following the methodology proposed in Section 3, we plotted the logits of the MLP and attention modules for each layer, as shown in Figure 22 . The prediction is solely determined by the 40-th layer MLP. Unlike Figure 3 , there is no observable periodic structure across all layers. For the model trained from scratch on the created addition dataset, all of the predictions on the test dataset deviate from the correct answer within 2 as shown in Figure 23 . Fine-tuned GPT-2-XL We finetune GPT-2-XL on the "language-math-dataset" with 50 epochs and a batch size of 16. The dataset consists of 27, 400 training samples, 3, 420 validation samples, and 3, 420 test samples. We use the AdamW optimizer, scheduling the learning rate linearly from 1 × 10 -5 to 0 without warmup.Train GPT-2-XL from scratch We train GPT-2-XL on the "language-math-dataset" from scratch with 500 epochs and a batch size of 16. The dataset consists of 27, 400 training samples, 3, 420 validation samples, and 3, 420 test samples. We use the AdamW optimizer, scheduling the learning rate linearly from 1 × 10 -4 to 0 without warmup.Train GPT-2 from scratch For both with pre-trained token embedding and without token embedding, we train GPT-2 on the "language-math-dataset" with 700 epochs and a batch size of 16. The dataset consists of 27, 400 training samples, 3, 420 validation samples, and 3, 420 test samples. We use the AdamW optimizer, scheduling the learning rate linearly from 5 × 10 -5 to 0 without warmup. In Figure 7b , we train the model with five different seeds and plot the mean and deviation for them.We consider numbers in base 10 up to a maximum value of 260. For each pair of numbers between 0 and 260, we used a distinct phrasing for each pair of numbers, selecting one template from five available templates. This ensures that every unique pair of numbers between 0 and 260 is presented with a consistent phrasing from these templates. We have fixed that typo in the revised version. The different phrasings used are: "Total of num1 and num2.", "Add together num1 and num2.", "Calculate num1 + num2.", "What is the sum of num1 and num2?", and "Put together num1 and num2.". The dataset is shuffled to ensure randomness and then split into training (80%), validation (10%), and test (10%) sets.Create the addition dataset in Appendix C.3 with different format We consider numbers in base 10 up to a maximum value of 260. We generate all possible pairs of numbers within this range using combinations with replacement. For each pair, we convert the numbers to the specified base and create questions formatted as "num1,num2+" with their corresponding answers. The dataset is then split into training (80%), validation (10%), and test (10%) sets.All experiments involving fine-tuning and training from scratch in this paper were conducted on one NVIDIA A6000 GPU with 48GB of video memory. The fine-tuning process required less than 10 hours, while training from scratch took less than 3 days.Other experiments, such as those involving Logit Lens, were completed in less than 1 hour.Licenses for Existing Assets & Open Access to Data and Code. For the following models, we use the checkpoints provided by Huggingface. For all the trained models, we use default hyperparameters during all the training but with different random seeds.• GPT-2-XL: https://huggingface.co/openai-community/gpt2-xl, Modified MIT License• GPT-2: https://huggingface.co/openai-community/gpt2, Modified MIT License• GPT-J: https://huggingface.co/EleutherAI/gpt-j-6b, Apache-2.0 License • Phi2: https://huggingface.co/microsoft/phi-2, MIT License• GPT-3.5 and GPT-4: https://chatgpt.com/ or https://openai.com/index/openai-api/ • PaLM-2 https://ai.google/discover/palm2/Our work aims to understand the potential of large language models in solving arithmetic tasks. Our paper is an interpretability paper and thus we foresee no immediate negative ethical impact. We believe improved understanding and enhancement of LLMs can lead to more robust AI systems that are capable of performing complex tasks more reliably. This can benefit areas such as automated data analysis, financial forecasting, and more.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: In the introduction (Section 1), we explicitly state the observations and their implications.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: The paper analyzes how LMs perform addition using Fourier features and with logit lens as well. The work makes a solid niche contribution in our understanding of LMs mechanism of computing addition. While the study is conducted on a very small dataset and mainly the addition property of LMs is analyzed, the work is solid and it contributes to own understanding of LMs.

Reviews: review: paper_summary: paper_summary: This paper analyzes how language models conduct the task of mathematical addition. Specifically, the paper shows that when performing the mathematical addition task, the internal states of language models exhibit periodical patterns, referred to as "Fourier Features" in the paper. The paper then conducts Fourier transformations to analyze such features and shows that (1) high-frequency features are more helpful in reaching the correct prediction (2) filtering out Fourier components causes performance degradation, hence these features are causally important for model predictions (3) such feature arises during pre-training, in the token embedding components. Finally, the paper shows that their findings generalize to other pre-trained LLMs (GPT-J, Phi-2, GPT-4, etc.) since they exhibit similar behaviors.
strength_weakness: strength_weakness: 1. While the study is thorough, I find the scope to be quite narrow -- it's only focused on one specific arithmetic task. I'm skeptical about how much this will help downstream model development.
2. To make the study feasible, the paper focused on the single-token case, which in turn limits the range of addition to <520. This makes the setup very artificial. That said, this is unfortunately an inherent weakness of many similar analysis papers, and I realize it's tricky to reach a good solution.
questions: questions: ### Presentation Suggestions

- Section 3: I know this is mentioned in the existing literature but it would be nice to define "modular addition" before you use it here for the first time.
- L91: I'm confused by $[L]$. Do you mean a range $[0, L)$ or something similar?
- L114: "approximately" is a weird word in this context. I'm not sure what you are trying to say, but to me, it makes perfect sense even if you remove it (similar occurrence in L121).

### Questions

1. How much do you think the choice of positional embedding has played in the creation of Fourier features? To be more specific, if we use learned positional embedding rather than sinusoidal/rotary positional embedding, do you think you'll still observe these features?
2. I might have missed it but I don't think there's ever an explanation for why attention only learns high-frequency features, while MLP has both. Do you have any insights?
3. More of a suggestion: Did you look at stronger open-source models such as the Mixtral ones? It's a pity that you had to do in-context learning for the model to be able to conduct arithmetic addition.
limitations: limitations: The authors should address the limitations their experiment configurations have created, including but not limited to:

- single token case only
- small number addition only
- findings are constrained to a single arithmetic operation, and it's not clear whether they'll generalize to others

(To clarify, not asking for extra experiments, but just acknowledging the limitations)
paper_summary: paper_summary: This paper analyzes how language models perform addition, showing that they use Fourier features. Most of the analyses are done on fine-tuned models. First, periodicity is shown with a logit-lens technique on different layers. Fourier analysis shows this nicely in the frequency space, and shows differences between MLP and attention components, the latter exhibiting only high-freq components. This is connected to "approximation" of the answer vs "classification" per small modulo. Ablation by removing low/high frequency supports a difference in behavior of the two components. The next section shows that models trained from scratch do not have such Fourier features, but plugging in trained number embeddings recovers this behavior. Finally some analysis shows a similar pattern in prompted models.
strength_weakness: strength_weakness: 1. Missing quantitative analysis of the superposition pattern in section 3.2. See below in questions. 
2. While the main experiments in section 3 (and to a lesser extent also 4.1 and 4.2) are quite thorough and convincing, the ones in 4.3 and in some places in the appendix are less detailed and not sufficiently contrasted with the main results. See below in questions. 
3. Potential problem with datasplit having leakage of expressions from training to test. See below in questions.
questions: questions: 1. Data split seems to allow cases where the same expression (say 12 + 17) appears in both training and test sets, just with different templates. When looking at generalization, especially when fine-tuning a pretrained model, it could be beneficial to make sure no such overlap happens. 

2. Section 3.1 assumes that "If the models merely retrieve and recombine pieces of information learned during training, certain layers will directly map this information to predictions". I am not convinced by this. I don't see why 'memorization and recombination' cannot be implemented by the full model, with this gradual refinement process as an implementation of memorization. 

3. The periodicity analyses in section 3.2 are striking and insightful. That said, one concern is quantitative analysis - I found the example in figure 4 helpful, but would appreciate a quantitative evaluation of the superposition phenomenon. Also, given the residual stream view, one could say that this is trivial. That is, the final logits are always the sum of earlier ones. So, it would be useful to explain the significance of this finding and whether there is anything special in this case of addition compared to the general behavior of the model. Even better, some control test in a quantitative evaluation could make this point more compelling. 

4. Section 3.3 is excellent - it shows which components are causally important for which roles. The experiments are well planned and convincing. One comment I have is on line 183, which says "the approximation tasks are primarily performed by the MLP modules alone". If I read Table 1 correctly, then, given that removing low-freq from both Attn and MLP leads to an even greater reduction in accuracy, we can conclude that both MLP and Attn are involved in approximation. This is essentially the same argument made in the paper just a few lines above. 

5. Since the analysis in section 3 is done on fine-tuned model, I don't think we can conclude that "The previous section shows that pre-trained LLMs leverage Fourier features to solve the addition problem" (line 195). The rest of the section in fact gives evidence to this, but the phrasing at this point should be revised. 

6. Figure 3 is very nice but other figures aren't as clean, specifically figure 8 (4-shot) and figure 15 (multiplication), figure 17 (other format), and figure 18 (GPT-J). I think the discussion of each of these cases was a bit too optimistic and naive, and the comparison to the main analyzed case should be made more carefully, and differences highlighted. 

7. The related work section is well done. Clarifying the difference from [28] early on in the paper could be useful for the reader. And, one can also mention this paper on arithmetic mechanisms in the related work: Stolfo et al., A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis.
limitations: limitations: Sufficient discussion.
paper_summary: paper_summary: This paper focuses on understanding the mechanisms the LLMs employ to carry out mathematical operations, in particular sum of two numbers.  It demonstrates: a) Models utilize Fourier features, with different parts of the model utilizing different frequency ranges — attention mechanism uses high frequency components while MLP primarily uses low frequencies.  This is further corroborated by conducting filtering studies.  b) Model embeddings and the pre-training process play a critical role in learning these Fourier features.  Models that are not pre-trained do not seem to be able to induce Fourier features and have lower accuracy.  However, such models when seeded with only embeddings from pre-trained models do learn the Fourier features.
strength_weakness: strength_weakness: * Study is limited to addition and that too of numbers where the final sum can be represented by a single token, leaving a gap in understanding of how addition of larger numbers spanning multiple tokens is achieved.
* With use of tools / functions in conjunction with the LLMs, mathematical operations are often carried out using a calculator or similar tool.   Thus attempting to improve LLMs' arithmetic abilities may not be very fruitful.  However, do the discoveries made in the paper apply only to number processing by LLMs or can they have broader implications, say on logical reasoning?
questions: questions: Lines 154-156 : “We also illustrated that in one example, the high-frequency components primarily approximate the magnitude, while the low-frequency components are crucial for modular addition tasks, as depicted in Figure 4.“  Is it the other way around?
limitations: limitations: Limitations sufficiently addressed
paper_summary: paper_summary: The authors utilize discrete Fourier transform to determine which Fourier components play the most important role in computing the addition of relatively small numbers (< 520) in Large Language Models, such as the GPT-2 family, and, _likely_, Phi-2, GPT-J, and others. They found out that MLP modules of GPT-2 use mostly low-frequency Fourier components to approximate the magnitude of the answer. In contrast, MHA modules utilize high-frequency components to approximate the last digit of the answer. Moreover, the model converges to the correct answer layer by layer.

The paper shows that this mechanism appears after pretraining the model; the weights of the embedding layer play a special role in it.
strength_weakness: strength_weakness: - Most of the experiments were devoted to the models from the GPT-2 family; very few experiments have been carried out on other models, so any arguments about other models are weaker;
- Only small (< 520) numbers were considered;
- The experimental setup is questionable (see "Questions" part of the review);
- The paper is not easy to follow. Some figures are confusing, and there are many typos (see "Questions" part of the review).
questions: questions: Questions:

- If I understood correctly, Table 1 contains the ablation experiments only for GPT-2-XL. Why didn't you repeat the experiments from Table 1 for other models? E.g. for Phi2, GPT-2-base, etc.
- How does the mechanism change when you add the bigger numbers, encoded in several tokens instead of one token?
- You wrote in 91 lines:
> ℓ ∈ [L].

Does it mean, that you use the outputs of all layers, preceding layer number L, for prediction?
- You wrote at 594 lines:
> We consider numbers in base 10 up to a maximum value 260. For each pair of numbers between 0 and 260, we generate various phrasings of addition questions and their corresponding answers. The different phrasings used are: “Total of num1 and num2.”, “Add together num1 and num2.”, “Calculate num1 + num2.”, “What is the sum of num1 and num2?”, and “Put together num1 and num2.”. The dataset is shuffled to ensure randomness and then split into training (80%), validation (10%), and test (10%) sets

Does it mean that train, valid and test could contain the addition questions for the same numbers, but in different phrasing? I.e., could the train contain the phrase “Total of 11 and 12.”, while the test contains “Put together 11 and 12.”?

---

Suggestions:

- GPT-2 is a model with absolute position embeddings. What about other types of position embeddings? (see also my question/suggestion about ablation experiments on Phi2 above). It would be interesting to research the connection of the addition mechanism with the position embedding type. See the paper "**Positional Description Matters for Transformers Arithmetic**" by Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang.
- It would also be interesting to research the connection between the Fourier addition mechanism and outlier dimensions (see, for example, the paper "**Outlier Dimensions Encode Task-Specific Knowledge**" by William Rudman, Catherine Chen, Carsten Eickhoff, and many other papers about Outlier Dimensions in transformers).

--- 

Typos and presentation issues:

- Figures 1 (b) and (c) are very confusing because the word "Number" is closer to the color bar, than the word "Logits". So, at first, I didn't understand, that the color means the **Logit value**. As a result, I was confused about these diagrams, and it took time for me to sort things out.
- You wrote in line 165:
> We show in Appendix A that this has a simple closed-form solution involving a linear projection.

I didn't find this in Appendix A, can you please, elaborate, on what you meant?
- You wrote in the description of Figure 13: 
> Visualization of the Fourier component whose period is 520 analysis for the final logits.

Can you please, elaborate, on what did you mean?
- You wrote in line 204:
> D is the size of the token embeddings, denote the token embedding for numbers

Can you please, elaborate, on what did you mean?

- What does Fig.5(a) show? Are these some average number embeddings?
- Why does Fig.6 show the legend with the different colors of cos/sin, while only red bars are visible? Maybe the bars are too narrow, so their colors mix together?
limitations: limitations: There are two small "Limitation" and "Impact" sections: Appendix G Limitations, and Appendix H Impact Statement.

The authors addressed the limitations correctly overall. However, I would add that most of the experiments were devoted to the models from the GPT-2 family; very few experiments have been carried out on other models, so any arguments about other models are weaker. It is also a big limitation of this work.

Comments: title: Thanks for the response
Thanks for clarifying!
Clarify Q2 about data overlap
Thanks for the response; missing one important question
Detailed Elaborations: Part 2
Detailed Elaborations: Part 1
More details about Q5 and Q6
comment: We are glad that we address your concern about ‘other figures besides 3’ and ‘difference from reference [1,2]’ and we will make sure to add these discussions to the revised version. We will also incorporate the statement you mentioned into the main body of the paper to highlight our contribution, which we believe will provide valuable insights for future work.

We have addressed the ‘concern about data overlap’ in our response to all reviewers. In summary, there is no overlap between the training data and the test data.

Thank you once again for your insightful questions. Below is our response to the rest of your questions:

***Q1: Concerning memorization vs computation/recombination***

We acknowledge that it is partly philosophical and difficult to prove that "direct output" is a requirement for memorization. However, as noted in other memorization tasks mentioned in our previous response, where the model uses only a few layers to make the prediction, we believe it is reasonable to hypothesize that the model is performing computation rather than memorization. In later sections of our paper, we correctly split the dataset to fine-tune the model. Through ablation studies and Fourier analysis, we provide additional evidence to support the claim that the model is indeed computing rather than memorizing.

***Q2: superposition phenomenon***

Thank you for your comments. We would like to clarify that the purpose of Figure 4 is to help readers understand how the results in Fourier space correspond to number space. Our experiments are primarily conducted in Fourier space, which may cause some confusion regarding how the results in Fourier space enable the model to identify the correct answer in number space. As explained in lines 137-150, we aim to help readers grasp, for example, why we assert that the Fourier component with a period of 2 is performing a ‘mod 2’ task. Although the superposition phenomenon is not the primary focus of our contribution, we believe that studying this phenomenon (using the top-k Fourier components to express the final logits) is an interesting direction. Since with only a few components, the model can accurately express the final prediction, a potential direction for future research could be making the model to process only with these outlier Fourier components to enhance LLMs’ speed and performance on arithmetic tasks.

***Q3: "primarily performed by the MLP modules alone," is inaccurate***

Thanks for your feedback! We acknowledge that the word ‘alone’ is misleading. We have changed the sentence to ‘As shown in Table 1, the approximation tasks are primarily performed by the MLP modules, with contributions from the attention modules as well.’  

Thanks for reading our responses! Please let us know if the above addresses the questions and concerns.


Sincerely,

Authors
thanks!
Thank you for your question. To clarify, no, the same pair of numbers $x,y$ that appears in the training data does not appear in the test data, even with a different phrasing.

Best,

Authors
Can you clarify the answer about Q2 above? If there is a pair of numbers x, y, between 0 and 260, which is in the training data, could the same pair x, y appear also in the test data (even with a different phrasing) or not?
Thank you for your response. 

I'd love to see a response about my concern regarding the data split in my question 1. 

1. Concerning memorization vs computation/recombination, I guess it's partly philosophical and hard to prove, but I don't see "direct output" as a requirement for memorization. I think memorization can be implemented by a gradual computation process (even if in other, retrieval-oriented tasks, it's more direct). 

2. Thanks for the clarification about figure 4. Still, figures 2+3 don't quantify the superposition phenomenon, do they? Would there be a way to quantify this specifically?

3. I still think that the statement "primarily performed by the MLP modules alone," is inaccurate, since attention modules are involved, as you also state. There may be some redundancy, but the above statement seems misleading. 

4. Thanks for a more careful discussion of other figures besides 3 and look forward to seeing it in the paper's next revision. This statement is excellent:
" we believe that without fine-tuning, the model does not fully leverage Fourier features to solve the addition problem. Even though the pre-trained model has learned to represent the numbers in Fourier space, it fails to fully understand the question phrasing and apply the Fourier method to solve it. We believe that better leveraging the Fourier features to improve the models’ performance on arithmetic tasks can be an interesting direction."

I believe something along these lines should appear in the main body of the paper, to accurately represent the contribution of the present work. 

5. Your clarifications about differences from refs [1], [2] are helpful; thank you.
As the discussion period approaches the deadline, we would like to follow up to see if our response and elaboration above has addressed your questions. Again, thanks for your time and effort. Looking forward to hear from you.
***Q8: You wrote in line 204: D is the size of the token embeddings, denote the token embedding for numbers Can you please, elaborate, on what did you mean?***

In line 204, we wrote ‘Let $W_E \in \mathbb{R}^{p \times D}$, where $p = 521$ and $D$ is the size of the token embeddings, denote the token embedding for numbers.’ By this, we mean that $W_E$ denotes the token embedding matrix for numbers. Normally the token embedding has a shape (vocabulary size, token embedding size). However, since we only consider the integer in [0,520], the vocabulary size becomes the size of the number space, and we have $p=521$.

***Q9: What does Fig.5(a) show? Are these some average number embeddings?***

Fig. 5(a) is not an average number embedding. Lines 203-216 explain Figure 5(a). Let us elaborate on the process of creating Figure 5(a) step by step to help you gain a better understanding.
1. For each pre-trained LLM, there is a pre-trained token embedding matrix. Since we only consider the embeddings for operands in the range [0, 520], we select the 521 rows corresponding to these 521 numbers from the token embedding matrix. We define this selected token embedding as the number embedding $E$ for simplicity. This number embedding matrix’s shape is (size of embedding dimension, 521).
2. Next, we apply the Discrete Fourier Transform to this number embedding matrix. We multiply this number embedding matrix by a Fourier basis, whose shape is (521, 521), resulting in the number embedding matrix in Fourier space. The shape of the number embedding matrix in Fourier space is (size of embedding dimension, 521). Each entry represents the magnitude of each Fourier component distributed to each embedding dimension.
3. Then, we take the L2 norm along the embedding dimension and obtain a vector ‘v’ with size 521. Each entry reflects the magnitude of each Fourier component distributed over the embedding dimension. Intuitively, each entry in this vector shows how important each Fourier component is in constituting the number embedding.
4. We ignore the constant term in that vector and plot the remaining values in Figure 5(a). According to the construction of the Fourier basis (Definition A.3), v[i] represents the sine wave when ‘i’ is even, and the cosine wave when ‘i’ is odd. Hence, we obtain the plot shown in Figure 5(a).
5. Figure 5(a) demonstrates that the number embedding has outlier components with periods of 2, 2.5, 5, and 10. The insight here is that these outlier Fourier components represent the numbers in the pre-trained GPT2-XL model. Compared with Figure 7(a), we show that using Fourier features to embed numbers is a learned behavior during pre-training. Additionally, we provide further evidence in Figure 14 to show that pre-trained LLMs tend to use Fourier features to embed numbers.

***If you need any further elaboration, please let us know.***
***Q6: In line 165: We show in Appendix A that this has a simple closed-form solution involving a linear projection. I didn't find this in Appendix A, can you please, elaborate, on what you meant?***

Thanks for your comments. In Definition A.6 (line 476 in Appendix A), we formally define the low/high-pass filter used in Section 3.3 as an optimization problem. We provide the closed-form solution to this problem in Remark A.5 (line 483 in Appendix A). To clarify this closed-form solution, let's walk through the derivation behind it step by step.

**Problem Statement**

We aim to filter the vector $x \in \mathbb{R}^D$ using a high-pass or low-pass filter, defined by the following optimization problem:

$
\min_y \| x - y \|_2^2 \quad \text{subject to} \quad B F W_U y = 0
$

Here, $B$ is a diagonal binary matrix that selects which frequency components to retain, $F$ denotes the Fourier basis, and $W_U$ is the output embedding matrix.

**Intuition Behind the Solution**

The objective is to find the vector $y$ that is as close as possible to $x$ while satisfying the constraint $B F W_U y = 0$. This constraint ensures that the filtered output $y$ lies in the null space of $B F W_U$.

**Null Space Projection**

To solve this optimization problem, we project $x$ onto the null space of $B F W_U$. The null space $N(B F W_U)$ consists of all vectors $v$ such that $B F W_U v = 0$. The projection of any vector $x$ onto this null space gives us the closest vector in the null space to $x$.

**Deriving the Projection**

The objective function $\| x - y \|_2^2$ measures the Euclidean distance between $x$ and $y$. Minimizing this function ensures that $y$ is as close to $x$ as possible. The constraint $B F W_U y = 0$ ensures that $y$ lies in the null space of $B F W_U$. To satisfy the constraint while minimizing the distance, we need to project $x$ onto the null space of $B F W_U$. The projection operator onto the null space $N(B F W_U)$ can be represented by the matrix $N(B F W_U) N(B F W_U)^\top$.

**Closed-form Solution**

The closed-form solution to the optimization problem is obtained by projecting $x$ onto the null space of $B F W_U$:

$
y = P_{N(B F W_U)} x = N(B F W_U) N(B F W_U)^\top x
$

Here, $N(B F W_U)$ is the basis for the null space of $B F W_U$, and the matrix $N(B F W_U) N(B F W_U)^\top$ represents the projection operator.





***Q7: You wrote in the description of Figure 13: Visualization of the Fourier component whose period is 520 analysis for the final logits. Can you please, elaborate, on what did you mean?***

Thank you for your comments. Let us emphasize the significance of Figure 13 and its implications. In our paper, we separate the frequency components into low-frequency and high-frequency components. We state that MLP layers primarily approximate the magnitude of the answer using low-frequency features, while attention layers primarily perform modular addition. One may ask: why can't we just use low-frequency components for classification? For example, ‘a+b mod 520’ will give the correct answers in our dataset. Figures 12, 13, and lines 515-526 in the appendix address this question.

When the model computes ‘a+b mod 2’, it is equivalent to a binary classification problem, which is simpler than ‘a+b mod 520’ (a 520-class classification). Figure 12b shows that the model accurately places the peak of the wave at the answer for ‘a+b mod 2’, ‘mod 5’, and ‘mod 10’. However, for large-period components, Figure 12a shows that the model only approximates the answer for ‘a+b mod p’, where p is the period of the components.

As we need to show all the large-period components in Figure 12a, we cannot display the entire number range. Therefore, the reader cannot tell whether the wave with a period of 520 correctly places its peak at the correct answer. Hence, we include Figure 13, which shows the period-520 wave over the entire number range. Figure 13 clearly demonstrates that the period-520 wave fails to place its peak on the correct answer and only approximates it.

As discussed in lines 129-136, the final logit is the accumulation result from the output of all the layers and is sparse in Fourier space. Figures 12 and 13 imply that the accumulation of low-frequency (large-period) components across all layers cannot make the correct prediction for ‘a+b mod p’ for large periods ‘p’. This provides evidence for why high-frequency (small-period) components are necessary for the model to make accurate predictions.
***Q5: Figure 8 (4-shot) and figure 15 (multiplication), figure 17 (other format), and figure 18 (GPT-J) are not as clean as Figure 3***

**Figure 17 (alternative format)**: Compared to Figure 3, Figure 17 also clearly shows the outlier frequency components. However, the outlier components in Figure 17 are not distributed across as many layers as in Figure 3, especially for the attention output. From the color bars, we can see that in Figure 17, the magnitude for the 10-period component in the last few layers is larger, particularly for the attention output. This suggests that the last few layers in Figure 17 learn the ‘mod 10’ task well and contribute more to the final logits compared to other layers. Hence, if we clip the values above 600 to 600, we can achieve results similar to those in Figure 3.

**Figures 8 (4-shot) and 18 (GPT-J)**: Compared to Figure 3, we can see the same outlier components in Figures 8 and 18, but they are not as clean as in Figure 3. Our experiments show that the accuracy for the 4-shot Phi2 (Figure 8) is 55.44%, and the accuracy for 4-shot GPT-J (Figure 18) is 72.21%. Hence, we believe that without fine-tuning, the model does not fully leverage Fourier features to solve the addition problem. Even though the pre-trained model has learned to represent the numbers in Fourier space, it fails to fully understand the question phrasing and apply the Fourier method to solve it. We believe that better leveraging the Fourier features to improve the models’ performance on arithmetic tasks can be an interesting direction.

**Figure 15 (multiplication)**: We can clearly see the outlier components in Figure 15, but they are not as clean as Figure 3. Note that for multiplication tasks, even if we expand the dataset size for multiplication, the accuracy only reaches 74.58% after fine-tuning. Multiplication is more difficult for LLMs, as shown in [1,2]. Hence, we believe that since the model does not fully learn to leverage Fourier features to solve the problem, Figure 15 appears less clean compared to Figure 3.

[1] McLeish, Sean, et al. "Transformers Can Do Arithmetic with the Right Embeddings." arXiv preprint arXiv:2405.17399 (2024).

[2] Dziri, Nouha, et al. "Faith and fate: Limits of transformers on compositionality." Advances in Neural Information Processing Systems 36 (2024).

***Q6: Clarify the difference from Nanda et al. (2023) and add Stolfo et al. (2023) to related work.***

Mechanisms of pre-trained LMs …Furthermore, [1] helps to understand the mechanism of pre-trained LMs when solving arithmetic tasks, by illustrating how LMs transmit and process query-relevant information using attention mechanisms and MLP modules.

Fourier features in Neural Networks. …[2] notes that Fourier features occur in shallow transformer models trained on modular addition. In contrast, our work focuses on the addition task for pre-trained LLMs. We specifically explore the necessity of various Fourier components and their contributions to the predictions...

[1] Stolfo, Alessandro, Yonatan Belinkov, and Mrinmaya Sachan. "A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis." arXiv preprint arXiv:2305.15054 (2023).

[2]Nanda, Neel, et al. "Progress measures for grokking via mechanistic interpretability." arXiv preprint arXiv:2301.05217 (2023).