FoNE: Precise Single-Token Number Embeddings via Fourier Features
Authors: Tianyi Zhou
Deqing Fu
Mahdi Soltanolkotabi
Robin Jia
Vatsal Sharan
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs
Arithmetic
Embedding
Numbers
Primary area: foundation or frontier models, including LLMs

Abstract: Language models treat numbers in the same way as ordinary word tokens, which introduces two major issues: (1) embeddings of numerical tokens primarily reflect their frequency in text corpora rather than their inherent numerical properties, leading to frequency bias, and (2) numbers are often split into multiple tokens, forcing the model to aggregate these pieces to recover their values.  Inspired by the observation that pre-trained Large Language Models (LLMs) internally learn Fourier-like features for number tokens, we propose **Fo**urier **N**umber **E**mbedding **(FoNE)**, a novel method that directly maps numbers into the embedding space with their Fourier features. FoNE encodes each number as a single token with only two embedding dimensions per digit, effectively capturing numerical values without fragmentation. 
Compared to traditional subword and digit-wise embeddings, FoNE achieves higher accuracy on arithmetic tasks, requires significantly less training data, and offers more efficient training and inference. 
A $38$M-parameter Transformer trained from scratch with FoNE outperforms a fine-tuned Llama-3.2-1B model on addition, subtraction, and multiplication. FoNE is also the only method that achieves $100\\%$ accuracy on over 100,000 test examples across these tasks. On 6-digit decimal addition, FoNE needs 64$\times$ less data than subword and digit-wise embeddings to reach $\ge 99\\%$ accuracy, while using 3$\times$ and 6$\times$ fewer tokens per number, respectively.

Published as a conference paper at ICLR 2026
FONE: PRECISE SINGLE-TOKEN NUMBER EMBEDDINGS VIA FOURIER FEATURES
Tianyi Zhou', Deqing Fu', Mahdi Soltanolkotabi Robin Jia', Vatsal Sharan'
'Department of Computer Science, University of Southern California [tzhou029, degingfu, fu, sol coltanll,, rapoolorrrree robinjia, ] @usc edu
ABSTRACT
Language models treat numbers the same way as ordinary word tokens, which introduces two major issues: (I) embeddings of numerical tokens primarily reflect their frequency text corpora rather than their inherent numerical properties, leading to frequency bias, and (2) numbers often split into multiple tokens, forcing the model to aggregate these pieces to recover their values. Inspired by the observation that pre-trained Large Language Models (LLMs) internally learn Fourier-like features for number tokens, we propose Fourier Number Emhedding (FoNE), a novel method that directly maps numbers into the embedding space with their Fourier features. FoNE encodes each number as single token with only two embedding dimensions per digit, effectively capturing numerical values without fragmentation. Compared to traditional subword and digit -igtt-wise embeddings, FoNE achieves higher accuracy on arithmetic tasks, requires significantly less training data, and offers more efficient training and inference. A 38M-parameter Trans- former trained from scratch with FoNE outperforms fine-tuned Llama-3.2-11 model addition, subtraction, and multiplication. FoNE is also the only method that achieves 100% accuracy on over 100,000 test examples across these tasks. On 6-digit decimal addition, FoNE needs 64x less data than subword and digit-wise embeddings to reach 2 99% accuracy, while using 3x and 6x fewer tokens per number, respectively.
INTRODUCTION
LLMs require precise representations of numerical data to perform number-related tasks effectively. However, since LLMs treat numbers just like any other token, embeddings of numerical tokens do not systematically capture important numerical features. As a result, challenging for even billion-parameter models to achieve perfect accuracy in solving simple arithmetic tasks (Saxton et al., 2019; Dziri et al., 2024; Lee et al., 2023; Shen et al., 2023; Zhou et al., 2023a). While generating code can be useful workaround, relying solely this capability highlights fundamental limitation: without proper understanding of numbers, the model cannot fully grasp concepts critical to domains like mathematical theorems, physics laws, quantitative reasoning. Even with approaches like Chain-of- Thouugt (CoT) prompting (Wei et al., 2022), it is important to have a perfect accuracy in solving basic arithmetic tasks build strong foundation for more complex reasoning.
Standard tokenization approaches, such as subword tokenization (e.g., GPT-4o Achiam et al., 2023, Llama-3 (Dubey et al., 2024). Phi-2 (Abdin al., 2024)) or digit-wise tokenization (e.g.. Llama-2 (Touvron et al., 2023). Mistral (Jiang et al., 2023)), require the model to aggregate multiple tokens to understand numbers and introduces inefficiencies by tokenizing one number into multiple tokens. However, this inefficiency in tokenizing numbers leads to larger challenges when it comes to their representation. Numbers, unlike words, require systematic, frequency-agnostic representations, yet LLMs often exhibit frequency bias (Razeghi et al., 2022; Shrestha et al., 2025; Shao et al., 2025), predicting numbers based on training data prevalence rather than their mathematical properties.
Our evaluation (See Appendix E. of recently released L. on arithmetic confirms this limitation: they still struggle with multi-digit addition and multiplication.

---

Published as a conference paper at ICLR 2026
Besuts Error Rata [Lo Scale)
Results Error Rate (Log Scale]
18
Error Rate (Log State)
30-1
(Log Scale)
20-1
10
20-3
10
to
digtt
subword
Ivai
10-9
10.42
FoNE(LB) - [cos]
10
10
100
2m
1100
2
10
0.0
Trinnig Size (Log Scale)
Model Site (Millinn Parameters)
20
40
Figure 1: Overview of Fourier Number Embedding (FoNE). Left: FoNE encoder illustrated with the token '1 '1'', directly mapped into its FONE. Middle: Test error on 6-digit decimal addition as the size of the training set increases. Right: Test error on the same task as model size increases. both plots, train transformers from scratch with different embedding or tokenization methods until convergence and report the final error. We compares FoNE (ours) against digit-wise tokenization, subword tokenization, XVAL (Golkar et al., 2023), and a fine-tuned Llama-3.2- IB model. FoNE achieves higher accuracy with less data and model size, even surpassing the finetuned Llama baseline
We draw inspiration from interpretability analyses of LLMs, which reveal that models internally develop Fourier-like features. Specifically, pre-trained models embed number tokens using a sparse set of features the Fourier domain (Zhou et al., 2024). These features enable the representation of numbers capturing both the magnitude and exact values of numbers, which are critical for solving arithmetic tasks (Zhou et al., 2024). However, because numbers are split into subwords and their embeddings are learned from co-occurrence statisties in text during pre-training, current LLMs fail to learn precise numerical representations and struggle to extend these mechanisms to larger numbers, underscoring the need for more systematic approaches numerical representation.
In this paper, we propose a novel approach called Fourier Number Embedding (FoNE), which directly maps numbers to their Fourier representations, bypassing the tokenization step entirely. By representing each digit using cosine and sine functions with different periods, as shown in the left panel of Figure 1, FoNE ensures precise representation of numbers. FoNE represents each digit using only two dimensions in the embedding vector. This compact design not only reduces computational overhead but also creates opportunities for future extensions by incorporating additional features to better capture numeric properties. By embedding and predicting numbers directly as single tokens, our method eliminates the need for multiple forward passes and token aggregation, significantly enhancing computational efficiency. Furthermore, we provide a theoretical justification for why FoNE can represent numbers accurately as single tokens, leveraging the modular encoding properties of trigonometric functions to ensure exact recovery of each digit through periodic embeddings.
Beyond theoretical justification, we demonstrate the effectiveness FoNE through extensive ex- periments arithmetic tasks, including addition, subtraction, and multiplication. Our results show that FoNE is the only approach which-when used to train a Transformer from scratch achieves perfect accuracy multiple arithmetic tasks while requiring significantly less training data and fewer model parameters compared to existing methods. Moreover, FoNE offers faster training and inference times by encoding each number into a single token. On 6-digit decimal addition it achieves 99% accuracy using 64x less data than subword or digit-wise embeddings, while cutting token usage per number by 3x and 6x, respectively. These findings underscore FoNE's capacity represent and manipulate numerical data both efficiently and precisely within large language models.
RELATED WORK
Arithmetic and Number-Related Tasks in L.Mss. Using language models for number-related tasks, including solving math problems (Saxton et al., 2019; Yu et al., 2023; Meidani et al., 2023), time-series prediction (Tan et al., 2024; Ma et al., 2024; Zhou et al., 2023b; Liu et al., 2024a; Jin et al., 2023; Cao et al., 2023; Li et al., 2025), quantitative reasoning (McLeish et al., 2024b; Liu et al., 2024b; Chen et al., 2023; Jin et al., 2024; Cobbe et al., 2021), and handling tabular data (Gao et al., 2024; Fang et al., 2024; Sahakyan et al., 2021), remains significant challenge. Despite advancements in transformer- based models, LLMs such as Qwen3-235B and GPT-5, with billions of parameters, struggle to solve simple arithmetic problems involving multi-digit addition and multiplication across
2

---

Published as a conference paper at ICLR 2026
multiple forward passes (Dziri al., 2024; Feng al., 2024), even when using scratchpads (Nye et al., 2021).
Golkar et al. (2023); Sundararaman et al. (2020); Jiang et al. (2019); Sivakumar & Moosavi (2024), introduce number embedding methods to enhance model performance on sumber-relater tasks. However, the range of numbers these methods can accurately represent is typically limited to fewer than five digits and fail to achieve perfect accuracy. Additionally, a line of research (McLeish et al., 2024a; Shen al., 2023) incorporates the positional information of digits into embeddings or adds it as extra tokens (Nogueira et al., 2021). Lee et al. (2023) demonstrate that smaller transformer models can successfully handle multiplication when equipped with carefully designed seratchpads. However, these approaches are tailored specifically for arithmetic tasks and are difficult to integrate seamlessly into ceneral-purpont I.I.M training. Thawani et (2021) explores encoding strategies like digit-by-digit, scientific notation, and b0se-10 formats, while Jiang et al. (2019) maps numbers to finite "prototype numerals". These methods help the model align digits of equal significance but often require digit-wise tokenization and introduce additional tokens, reducing training and prediction efficiency. In contrast, the method proposed this paper precisely encodes all numbers as a single token, eliminating range limitations and avoiding the efficiency drawbacks associated with previous approaches (see Section 5 for further details).
3 METHODS
Building on insights from prior studies (Zhou et al., 2024) that highlight the importance of Fourier features in numerical embeddings, propose Fourier Number Embedding. Unlike existing methods that often require digit-wise tokenization or pre-training to handle numeric tasks, FoNE directly maps numbers into compact Fourier representations, Sections 3.1, 3.3, and 3.4 describe our embedding, decoding, and integration methods, respectively. The complete process is shown in Figure 2.
3.1 FOURIER NUMBER EMBEDDING (FONE)
We first introduce the following function that maps each x E to point on the unit circle. Definition 3.1 (Circular embedding). Let T be a given period. We define the function & R IR2 as
d(x, T) (cos 2 sin (27 (2)))
Next, we formally define FONE, which directly maps any floating point number J to embedding. We predefine and n as the maximum number of digits before and after the decimal point, respec- tively. Note that m and are global hyperparameters fixed for the entire dataset model, rather than chosen per number.
Definition 3.2 (Fourier Number Embedding). Let m be the integer digit length, and n be the decimal digit length. We define the Fourier Number Embedding function functioonn FoNE: (B2((++))) for an input number x as follows:
FoNE(æ, m, n) == [o(=,
(z,
where T; = 10s for each integer i in the range -n + 1 to
To align the embedding dimensions of FoNE with the model's 's input emhedding dimension d, we map the Fourier Number Embedding, which lies in 2((m+n), to Rd This mapping can be achieved in two ways: (1) by applying a learnable linear transformation w E Rdx2(m+tx) or (2) by appending zeros to the embedding vector to match the dimensionality d. As demonstrated in Section 4.3, both approaches achieve comparable results.
3.2 FONE's REPRESENTATIONAL PROPERTIES
Then, introduce elementary lemma and demonstrate why FoNE can preserve the numeracy on numbers.
Lemma 3.3 (Informal version of Lemma D.1). Given the pair (cos ( (2 .sin ( (27 x)) recover & mod T.
we can
3

---

digit, e.g. the first two Published as a conference paper at ICLR 2026
Hence, by applying Lemma 3.3 to each frequency component in FoNE, immediately obtain the following result.
Lemma 3.4 (FoNE preserves numeracy). Given number's Fourier Number Embedding (oNE(x)), its integer digit length m, and the decimal digit length n, by using Lemma 3.3, we can recover x mod 10y for each integer in the range -n n to m.
natural question that arises here is the need for mod 10, we already know mod 100. The reason is that even though knowing r mod 100 exactly suffices to recover I mod 10, this estimation noisy in practice. When T becomes very large in a circular embedding (Definition 3.1), the 27 difference T (z to become arbitrarily close on the unit circle. Consequently, single large T cannot sufficiently distinguish adjacent values in the embedding. Hence, one must choose T across broad range of scales to ensure that the embedding remains adequately distinguishable for all values of z. this paper, we choose T as 10', Vi, that each T effectively captures digit of x.
2r r approaches zero, causing the embedded representations of x and x +
To provide a clear illustration of our method, we present a detailed example demonstrating how we map number 4.17 to its embedding.
Example 3.5. Consider 4.17. Its Fourier Number Embedding is given by
(6(4.17, 0.1); (4.177 6(4.17, 10)],
where & is defined in Definition 3.1. From these components, by using Lemma 3.3, we can recover
[4.17 mod 0.1, 4.17 mod 1, 4.17 mod 10).2
which simplifies to [0.07, 0.17, 4.17]. If we used only T 10, then (64.17, 10) would be nearly indistinguishable from Φ(4.18, 10), causing the embedding to lose fine- fine-grained information about
less significant digits. However, with these chosen periods T, we can capture all the digits.
3.3 DECODING
FoNE Integrated with Transformer
Fourier Number Decoding
predicted number
82
tolput
[NUMM]
(a, b)
Unambitt)
lnstiedin abute
in
Unit digt =2
hs
Transformer
add togeherr
Jts
FoME(18)
FoME(64)
Imput embeictig
(Postimb)
(Postimb)
(Postimb)
Eimbi (MUMM]
10mb(++))
Eimbi [NUUM]
Eada' 'm))
Ieputtokon
Figure 2: Left: Each number in the input sequence replaced by a special token [NUM] and embed- ded the sum of [NUM] token embedding, its FoNE (see Figure (a)), and the standard position embedding (if used by the architecture; I.lama-3.2 does not use an explicit position embedding). Right: At decoding time, every pair of hidden-state entries corresponds entries ho and h2 correspond to the unit digit. The model identifies the digit whose circular Fourier representation best matches those two entries. Digits are then combined by their positional weights.
As each number has its own FONE, calculating the logits for all possible numbers becomes compu- tationally infeasible. Therefore, we introduce novel decoding head that maps hidden states from
For real and positive real m, mod is defined
yiclding value in the range [0, m)

---

Published as a conference paper at ICLR 2026
Fourier space to number space shown in Figure 2. Below, we explicitly define the loss function and prediction function for each digit and then show how to combine these to obtain the final loss and prediction.
Definition 3.6 (Fourier Number Loss Function). Let h Rd denote the last-layer hidden state of the model. Let yi denote the i-th digit of the label number Let Lcg denote the cross entropy loss. For digit i, we define the Fourier Number Loss Function LFONE as:
TFONE y, i) = I.C. Yi, (1 2
(6(0, 10):
10)]
2x10
Note that the comparison set {0( 10), ................................ (9, 10)} } is the same for all digit positions i. This is because each digit, regardless of its positional significance, must be classified as one of the 10 possible values {0, 1....,.9) 9}. This design enables parallel decoding where each digit position is treated as separate classification task over the same 10-class space. The final training loss is obtained by averaging LFONE( (h, i) over all digit positions i.
Definition 3.7 (Fourier Number Prediction for the i-th digit). Let h E IRd denote the last-layer hidden state of the model. For digit i, we define the Fourier Number Prediction as:
D arg
max jE(O....99)
[h[2i], h|2i 1]] [6 10]])
Here, 7. is determined by the similarity between the hidden states and the circular embedding of number in (0..... 9) as illustrated in Figure 6(d). Once we have computed ÿ; for each digit the final prediction for the entire number can formed by concatenating these digit-wise predictions.
We defer the detailed algorithms to Appendix A.
3.4 INCORPORATING FONE INTO INPUT SEQUENCES
To incorporate FONE, we create one special token [NUM] and add to the vocabulary. This token must be generated by the model in order generate any number. We can then remove any tokens corresponding to numbers from the vocabulary. In practice, numbers appear in text with varying formats (c.g., "1,234.56", "$99.99", "3.14e-2"). FoNE handles this diversity by first applying standard numeric-string parser that detects numbers, ensuring consistent representation regardless of the original textual format.
Integration Procedure. The integration FoNE into input sequences proceeds as follows, as illustrated in Figure and Figure 6:
Extract all numbers from the input sequence using the numeric parser to create a number list. Each detected number replaced with [NUM] and its parsed canonical value. Tokenize the modified sequence to obtain token list.
2. Embed the token list using standard word embedding methods.
3. Map each canonical number value from the number list to its FoNE representation using Algorithm 1 (Section 3.1).
4. Add the FoNE to the word embedding of the corresponding [NUM] token.
5. Feed the combined embeddings into the model.
6. Use the model's output embeddings to predict the next token in the sequence.
7. If the predicted token is [NUM], decode the numerical value using the method described in Section 3.3, or compute the loss during training.
This procedure ensures that FoNE embeddings seamlessly integrated into the input sequence. Transformer architectures can effectively aggregate information across tokens; with FONE, each numeric token carries precise numerical representation, enabling the model to aggregate and interpret numerical information more reliably than with trequency-biase standard embeddings.
EMPIRICAL EVALUATION
4.1 EXPERIMENTAL. SETTING
We evaluate the performance of our proposed FoNE method on arithmetic tasks designed to bench- mark different number embedding methods. The dataset includes tasks such 6-digit integer
5

---

Published as a conference paper at ICLR 2026
addition, 6-digit decimal addition (with 3 digits after the decimal), 5-digit integer subtraction, 3-digit integer multiplication, and 4-digit integer multiplication. These tasks are curated to measure model capabilities in accurate numeric computation, while remaining within ranges where baseline embed- ding methods are still competitive- sinpe their performance degrades rapidly for larger numbers. To further probe the scalability of our approach, we additionally evaluate FoNE on 60-digit addition in Section 4.4, which highlights its ability to handle much larger operands where other embeddings fail.
Dataset. Each example in the dataset is formatted as [operand a [operator] [operand b]=, where the operands a and b are sampled based on the operation type. For addition and multiplication, we ensure I to avoid duplication (e.g., a + b and b + a are treated as identical and included only once). For subtraction, we enforce 2 b to ensure non-negative results. For an z-digit operands dataset, each operand can have up to x digits. The dataset divided into training. validation, and test subsets as shown in Table 10 in Appendix H.
Baselines. We compare our proposed FoNE method against several baseline methods for numeric embeddings. First, we consider digit-wise tokenization, where each digit in number is treated as an individual token. Second, we evaluate subword tokenization, where numeric values are tokenized into subword units based on the Llama3.2-11 tokenizer's 's default vocabulary. Third, we include the XVAL method Golkar et al. (2023), which leverages explicit value-based representations for numeric computation. As VAL predict floating point numbers, predictions are rounded to calculate accuracy. Finally, we fine-tune pre- trained LLMs on the same dataset for comparison.
Setup. Our experiments involve multiple configurations of randomly initialized wansformer-based models. Models were evaluated across varying sizes, ranging from small to large architectures as defined in Table 12. For the accuracy training data size experiments, similar to Llama-3.2 but with 38M parameters. In Appendix I, we conduct more experiments on transformer model with a diffferent configuration and observe consistent results.
configuration
Learning rates were determined through an extensive search, with the best rates selected separately for each method based on the validation performance. Model evaluation used exact match accuracy to assess numeric prediction correctness. All models were trained from random initialization, except the fine-tuned Llama-3.2-1 IB baseline model. We varied the training data size by uniformly sampling subsets and adjusted model sizes to compare accuracy across methods.
4.2 EXPERIMENT RESULTS
10"
Hoadlo bro Noti Sudk
10"
Roodt Hote bord
Emer Dase Logg Stakes
10-2
30
Erer Rare Lbg
32-1
Instune
2000
Texture
sum
ours
dgt
dgt
32-1
10-12
10**c
10
122
Daia Stod lug Suno
Mudel oi (MSiiio anddors)
Tunnn Data Sua lon Stako
Modoo 5:st (M on Poramntter))
(a) 6-digit integer addition
(b) 5-digit integer subtraction
1.2
Apcults- Ertr Aate 3.n Scalo
1.0
Resulto Error Tate Scale!
10"
Resutt Eror Ratic Scare
D00L 14.g Staret
Emtrr Sate Bag Scdaa
So NOOU Haa Siard
Erer tat Dupg
10
1074
20-3
Stature
1274
Freeune
2375
dgt
adbwwod
well
12'9
10
10-22
10
307
102
Trasiig Daca Nin (1og Sciae
Modil Siw
(Millio Becamettrn)
Taainn bank Siw
Wodel 5
04 Pacomettnn
(c) 3-digit integer multiplication
(d) 4-digit integer multiplication
Figure 3: Comparison of accuracy for various arithmetic tasks with respect to model and data size.

---

Published as a conference paper at ICLR 2026
Data Efficiency. The middle panel of Figure illustrates the accuracy trends of different embedding methods as the data size increases for the 6-digit decimal addition task. Remarkably, our model achieves 99% accuracy with just 400 training samples and 37.55 million parameters, requiring x4x less training data than traditional embedding methods (409, 600/6, 66, 400 = 64). Even with only 200 training samples, method outperforms the fine-tuned Llama-3.2 model. Additionally, achieves perfect accuracy with 51, 200 training samples.
Beyond synthetic tasks, our approach also improves compute efficiency in real-world scenarios. For instance, FoNE requires only 149.25 tokens on average to represent numerical values from table in the WilitableQuestionn dataset (Pasupat & Liang, 2015), compared to 329.7 tokens used by digit-wise tokenizer. This sigmificant reduction in token usage highlights the efficiency of our method encoding numerical data, making more scalable for number-heavy tasks.
Parameter Efficiency. The right panel of Figure shows the accuracy trends of different embedding methods as the model size increases for the 6-digit decimal addition task. Our method achieves 97% accuracy with just layer and 8.31 million parameters using 200k examples for training. Furthermore, with 26.62 million parameters, it surpasses the fine-tuned Llama-3.2 model and achieves 100% accuracy.
Different Tasks. We conducted the same experiments across all different datasets. As shown in Figure 3, our method consistently demonstrates superior data and parameter efficiency compared to other approaches. Notably, it is the only method that achieves perfect accuracy on 6-digit decimal addition, 6-digit integer addition, 5-digit subtraction, and 3-digit multiplication. We also show that method performs better in binary classification task that involves numerical values in Figure 7 and Figure 8. Specifically, the task requires predicting label based on linear equation applied three integers. In addition, we evaluate FoNE modular addition task. In Table 8, we show that it outperforms standard tokenization methods, especially under large moduli where conventional approaches fail. Due to space limitations, we defer the details to Appendix B.
Training and Inference Efficiency. Table 1 compares the training and test times used for one epoch across different embedding methods. Our method consistently faster than digit-wise and subword embedding methods, uses one token embed each number. Compared with x XVAL, our method consistently achieves higher accuracy. Additionally, we show the number of tokens required tokenize the maximum number for different methods, highlighting the efficiency of our approach.
Table 1: Training and inference efficiency comparison across three arithmetic tasks. The times are reported in minutes (') and seconds (").
Decimal Addition
Subtraction
Multiplication
Method
Train Time Test. Tokens Accuracy Train.
Test.
Toks.
Acc.
Train.
Test.
Toks.
Acc.
Ours
3'18"
29
100
2'42"
29
100
2'56"
33"
98.56
Digit-wise
11'48"
1'25"
99.85
9'41"
1'15"
5
99.71
10'11"
1'18"
81.21
Subword
6'46"
58"
3
97.94
5'47"
54"
2
91.66
6'20"
58"
8.05
XVAL
3'17"
27"
0.44
2'54"
27"
3.41
2'56"
26"
4.3 ABLATION STUDIES
Linear Layer after FONE. As discussed in Section 3.1, we evaluate the use of linear layer applied after FoNE and compare it with the approach of appending zeros to align the embedding dimensions with the model's input requirements. As shown Table 2, both configurations achieve almost the same accuracy. Hence, either technique can be used to align FoNE with the embedding dimension.
Table 2: Accuracy Comparison Across Datasets
Table Accuracy Comparison Across Periods
Task
Linear
Zero Padding
Dataset
2,5,10
10
5
Decimal Addition
100%
100%
Decimal Addition
100
100
1.52
3.64
Integer Addition
100%
100%
Integer Addition
100
100
1.55
0.02
Multiplication
99.95%
99.91%
Multiplication
99.99
99.95
3.67
1.91
Subtraction
100%
100%
Subtraction
100
100
4.64
0.24

---

Published as a conference paper at ICLR 2026
Effeet of Different Periods. As discussed in Section 3.1, the modular group captures the necessary information for each digit, ensuring the effectiveness of our approach. We test the model with base periods of [2, 5, 10), [5], and [7], shown in Table 3. The [2, 5, 10] configuration achieves accuracy comparable to that of the 10-period setup across different datasets. In this paper, we choose single 10 to make more parameter efficient. However, configurations using only mod 5 or mod 7 exhibit significantly lower accuracy. This is because neither mod5 nor mod7 can fully represent the required information for all digits.
The mispredictions are attributed to the absence of critical modular information. As illustrated in Table 13 in Appendix H. the decimal addition task, using only a mod5 representation prevents the model from distinguishing between certain digits, such as 2 and 7, which results in errors.
Necessity of Sine and Cosine Encoding. A natural question arises: are sinusoidal encodings truly necessary for arithmetic tasks? One could directly encode each digit into a separate dimension of the embedding, representing a number like 567 as [5, 6, 7]. However, this approach fails to achieve perfect accuracy. For instance, numbers such as 999 and 888 become nearly indistinguishable after layer normalization, which reduces their differences and can lead to confusion during training. We evaluate this direct encoding method on 6-digit decimal addition and, after performing a learning rate search, find that the best accuracy is 99.3% with learning rate of 0.01 and training for 100 epochs. In contrast, FoNE achieves better accuracy in just 6 epochs with the same dataset and model size. This suggests that naive direct encoding does not adequately preserve numerical distinctions for reliable arithmetic operations. As illustrated in Table 14 in the appendix, the model frequently mispredicts 8 as 9, further demonstrating the limitations of direct encoding in preserving numerical structure.
4.4 APPLICATIONS AND COMPLEMENTARITE OF FONE
Average Accuracy Difference (mean diff: 16.04%)
Accuracy Percentage (Average: 97.42%)
1.00
D
0.75
First Number Length
0.50
First Number Length
20
0.25
20
60
30
40
40
-1.00
10
20
30
40
10
20
30
40
Second Number Length
Second Number Length
(a) Impact of combining FoNE with Abacus embedding
(h) Test accuracy of 60-digit addition with FoNE
Figure 4: (a) Performance improvements achieved by combining FoNE with the Abacus embedding method across various random seeds. The transformer is trained on addition tasks with up to 10-digits numbers (represented by the smaller square) and tested up to 50-digit numbers. (b) Average accuracy of an 8-layer transformer model on 60-digit addition tasks using FoNE for chunked input.
Combining FoNE with Abacus. FoNE can be combined with other positional embedding methods even with different tokenization methods, requiring only minor modifications. For instance, we integrated FoNE with the Abacus embedding method (McLeish et al., 2024a), which operates on digit-wise tokenization. In this setup, the embeddings for each digit (0-9) are replaced with their corresponding FoNE. We trained an 8-layer transformer model integer addition tasks with up to 10 digits and tested it on addition tasks involving up to 50-digit numbers. The results, as illustrated in Figure 4(a) and Figure 12 in Appendix G, show that incorporating FoNE consistently improves the performance of the Abacus method across various random seeds. This highlights the complementary benefits of combining FoNE with other positional embedding strategies.
8

---

Published as a conference paper at ICLR 2026
Validation Perplexity
Training Perplexity
10
FoNE: 46.8
FoNE: 20.6
48.8
XVal:
52.6
BPE:
SingleDigit: 40.7
10*
Single 32.1
Perplexity
Lest 18 Tokera
Lam Takam
10¹
20
107
10a
10: Tokens (Billions)
1.t
Tokens (Billonns
Figure 5: Pretraining 117M GPT-2 model on 10 Billion tokens with various number tokenization methods. Our method (FoNE) achieves the lowest validation perplexity. This experiment shows that FoNE does not harm the language models' semantic abilities.
How does FoNE Handle numbers with longer digit sequences The maximum digit length that float 64 data type can represent is 15 digits. When r exceeds 15 digits in length, applying FoNE(z) directly may result in loss of precision. To address this, can be divided into smaller chunks, and FoNE can be applied to each chunk independently. For example, x can be split into groups of five digits. The FoNE can then be calculated for each chunk, resulting in representation of length 10 per chunk, as each digit is encoded in two dimensions. These embeddings are subsequently concatenated obtain the final number embedding for æ. Note that we are still using one token for each number. By using this method, as shown in Figure 4(b), an 8-layer transformer trained on 60-digit addition achieved an average accuracy of 97.42% across different operand length with just one forward pass. This demonstrates the effectiveness of FoNE in handling long sequences.
5 DISCUSSION
Q1: Why not use regression loss instead of classification loss, which minimizes RMSE and can yield smaller prediction errors (Zausinger et al., 2024). The key limitation is that regression produces continuous values, making it impossible to integrate number-related tasks with general lan- guage modeling. For example, when predicting the year "1997", regression may output "1996.9999 which is acceptable under regression metrics but unusable in sequence generation or token- reasoning. In contrast, FoNE retains a classification-base identifying which numbers are used for arithmetic. This ensures seamless compatibility with standard LLM training while still delivering accurate numerical representations.
loss, so it does not require explicitly
Q2: Why does FONE outperform other number embedding methods? Note that since FoNE uses the ratio between entries to represent numbers as shown in Lemma 3.4, it is unaffected by layer normalization and RMS normalization (Lemma D.2), in contrast to x Val (Golkar et al., 2023), which uses the magnitudes of entries. Other approaches, such as DICE (Sundararaman et al., 2020) and SALSA (Stevens et al., 2024), map numbers onto single unit circle, which can limit their capacity to distinguish between different magnitudes effectively. FONE, by leveraging multiple sinusoidal components, captures both the magnitude and periodicity of numbers more precisely. This comprehensive representation enables FoNE to achieve higher accuracy and generalization in number-related tasks.
03: Why do we choose components with periods that are multiples of 10, ie., 10, 100, 1000...? As shown in Zhou et (2024) and Figure 10, pre-trained LLMS trained on diverse datasets and strategies consistently learn nearly identical key frequency components. These components have periods of and its divisors, such as 2 and Since mod10 can already represent a single digit, we believe that mod 2 and mod 5 contribute to enhancing robustness. Models trained on real-world text data-where numbers are almost always expressed in decimal- comalllly learn frequency components that correspond to mod 10. In principle, we could choose alternative bases (such as 5, 16, etc.) to help the model better learn arithmetic in those bases, as demonstrated in Table 7. However, since most large language models primarily encounter numbers in base 10, and our results show that
9

---

Published as a conference paper at ICLR 2026
base-10 10 FoNE already performs well arithmetic tasks in other bases (Table 6), we adopt base 10 as the default. Additional experiments validating this choice are provided in Appendix B.
Q4: Can FONE be integrated into pretrained LLMs without harming their semantic abilities?
We study this question in two complementary settings. First, we perform continual pretraining of an existing LLM with simplified FoNE adaptation. Second, we train smaller GPT-2-11 18-1170 17M model from scratch on 1OB Web tokens with different number tokenization schemes. In both cases, FoNE preserve, and in the latter case even improving, language modeling performance.
Continual pretraining on Llama-3.1-IB. We demonstrate that our FoNE token embedding can be merged with any existing LLM with slight continual pretraining. We made simplification of FoNE where instead of overriding all the number embeddings which could make continual pretraining harder, we build the simplified FoNE on top of existing BPE tokenization. We compute FoNE embedding of each subword, and continual pretrain a linear projection layer from the FoNE embedding space to the original token embedding space align the embeddings similar to vision language models' alignment phase. We continual pretrain both the original Llama-3. 1- model and our FoNE adapted version on 15B tokens from MegaMath- Weehh-WWe--Prrr (Zhou et al., 2025) and evaluated on arithmetic tasks from Brown et al. (2020) and on MMLU (Hendrycks et al., 2021). As shown in Table 4, this simplified version of FoNE improves the model's zero-shot arithmetic with larger number of digits without affecting language abilities evaluated by MMLU.
Table 4: A simplified version of FoNE improves model's 's zero-shot arithmetic abilities without sac- rificing language abilities.
Table 5: Perplexity of text tokens only on sequences that contain both text and numbers. (evaluated on ~ million tokens in total).
TASK
REGULAR
FONE (Simplifted)
Method
Perplexity
4 digit addition
51.35%
59.00%
d-gigt subtraction
29.60%
39.90%
BPE
67.390
5-digit addition
29.40%
35.75%
Single Digit
$5.922
5-digit subtraction
24.95%
33.95%
x XVal
49.160
MMLU
38.10%
38.21 %
FONE
46.856
Pretraining from scratch on Fine Web. To make the model learn text and numbers together, we also pretrain a GPT-2-117M model from scratch on 10 billion tokens from FineWeb (Penedo et al., 2024) under four number tokenization schemes: BPE, Single Digit, x XVal, and FoNE. As shown in Figure 5, the FoNE model achieves the lowest validation perplexity on the Web validation set. Furthermore, we evaluate the perplexity of text tokens only sequences that contain both text and numbers. As shown in Table 5, FoNE remains the lowest perplexity while other methods show even worse perplexity compared to their performance on all sequences. Since numbers in Fine appear naturally interleaved with text, this experiment shows that using FoNE throughout pretraining not only preserves the model's semantic abilities but can in fact improve its language modeling quality by giving better number representation.
Semantic vs. numerical roles. These findings are consistent with prior interpretability results. As discussed by Meng et al. (2022), semantic and factual associations of cokens-such as the historical or cultural meaning of year- are often stored in the MLP layers of transformer models rather than in the token embeddings themselves. FoNE therefore plays complementary role: it provides precise, input embeddings for numbers, while the model's 's MLP layers continue to store and retrieve semantic knowledge. Together, this separation of roles allows FoNE enhance numerical reasoning without sacrificing semantic competence.
CONCLUSION
In this paper, we introduced FONE, novel method for representing numbers in the embedding space of L.LMs. By leveraging Fourier features, FoNE directly maps numbers into compact and precise representation, bypassing tokenization inefficiencies and preserving essential numerical properties. FoNE has significant implications for pre-training LLMs. By incorporating FoNE, models can develop a robust understanding of numerical concepts, addressing a fundamental limitation in current architectures. We believe our work establishes a solid foundation for future research wide range of number-related tasks, including time-series analysis, quantitative reasoning, and complex operations in fields like physics and mathematies.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We are grateful to Wang Zhu for providing helpful feedback on our paper. This work was supported by AWS credits from a gift from the USC-Amazon Center on Secure and Trusted Machine Learning. DE, MS, and RJ were supported in part by gift from the USC-Capital One Center for Responsible Al and Decision Making Finance (CREDIF). RJ was also supported in part by the National Science Foundation under Grant No. IIS-240343.. The work of MS was also supported in part by AWS credits through an Amazon Faculty Research Award, NAIRR Pilot Award, and generous funding by Coefficient Giving. MS is also supported by the Packard Fellowship in Science and Engineering, Sloan Research Fellowship in Mathematics, NSF CAREER Award # #1846369, DARPA FastNICS program, NSF CIF Awards #181 13877 and #2008443, and NIH Award DP2LM014564- opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Any
REPRODUCIBILIIY STATEMENT
We have taken several steps ensure the reproducibility of our results. The FoNE method is defined step by step in Section 3.1 and Section A. Our experimental setup. including datasets, sampling rules, model configurations, and training details, is described in Section 4.1 and Appendix H.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone, arXiv preprint arXiv:2404. /42 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar 23xx:23303008774 2023.
Jawang Bai, Li Yuan, Shu-Tar Xia. Shuicheng Yan, Zhifeng Li, and Wei Liu. Improving vision transformers by revisiting high-frequency components. In European Conference on Computer Vision, pp. 1-18. Springer, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.
Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:23 10.04448 2023.
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma. Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A sooorem-drvve question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7889-7901, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemyy Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint a 10. 14/68, 2021.
Abhimanyu Dubey. Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The laama herd of models. arXiv preprint arXiv:2407.2 2 1783, 2024.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers compositionality. Advances in Neural Information Processing Systems, 36, 2024.
11

---

Published as a conference paper at ICLR 2026
Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large language models on tabular data-a survey. arXiv e-prints, pp. arXiv 24022 2024.
Guhao Feng, Kai Yang, Yuntian Gu, Xinyue Ai, Shengjie Luo, Jiacheng Sun, Di He, Zhenguo Li, and Liwei Wang. How numerical precision affects mathematical reasoning capabilities of Ilms. arXiv preprint urXiv:2410. 13857, 2024.
Pierre- Fiquet and Eero Simoncelli. A polar prediction model for learning to represent visual transformations. Advances in Neural Information Processing Systems, 36, 2024.
Yanjun Gao, Skatje Myers, Shan Chen, Dmitriy Dligach, Timothy A Miller, Danielle Bitterman, Matthew Churpek, and Majid Afshar. When raw data prevails: Are large language model embed- dings effective in numerical data representation for medical machine learning applications? arXiv preprint arXiv:2408. / 1854, 2024.
Jonas Geiping and Tom Goldstein. Cramming: Training a language model on a single gpu in one day. In International Conference on Machine Learning, pp. 11117-11143 PMLR, 2023.
Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, et al. xval: A continuous number encoding for large language models. arXiv preprint arXiv: 10.0:22300.02999 2023.
Jiuxiang Gu, Chenyang Li, Yingyu Liang, Zhenmei Shi, Zhao Song, and Tianyi Zhou. Fourier circuits in neural networks: Unlocking the potential of large language models in mathematical reasoning and modular arithmetic. arXiv preprini MXXi::4402009460 2024.
Keji He, Chenyang Si, Zhihe Lu, Yan Huang, Liang Wang, and Xinchao Wang. Frequency-enhanced data augmentation for vision-and- lannguagn navigation. Advances in Neural Information Processing Systems, 36, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https: orq/abs// 2009. 03300.
Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Ning Ding, Youbang Sun, Biqing Qi, Yuchen Fan, Xuekai Zhu, and Bowen Zhou. Fourier position embedding: Enhancing attention's 's periodic extension for length generalization. arXiv preprint arXiv:2412.17739 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
Chengyue Jiang, Zhonglin Nian, Kaihao Guo, Shanbo Chu, Yinggong Zhao, Libin Shen, and Kewei Tu. Learning numeral embeddings. arXiv preprint arXiv:2011.00003 2019.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yo Chen, Yux- uan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming 2023.
large language models. arXiv preprint arXiv:2310.01
Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max -leiman-Weiner, Mrinmaya Sachan, et al. Cladder: A benchmark to assess causal reasoning capabilities of language models. Advances in Neural Information Processing Systems, 36, 2024.
Subhash Kantamneni and Max Tegmark. Language models trigonometry do addition. arXiv preprint arXiv:2502.00883. 2025.
Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381, 2023.
Amit Arnold Levy and Mor Geva. Language models encode numbers using digit representations in base 10. arXiv preprint arXiv:2410. 1781, 2024.
12

---

Published as a conference paper at ICLR 2026
Shixuan Li, Wei Yang, Peiyu Zhang, Xiongye Xiao, Defu Cao, Yuehan Qin, Xiaole Zhang, Yue Zhao, and Paul Bogdan. Climatellm: Efficient weather forecasting via frequency-awer aware large language models. arXiv preprint arXhv:2502.11054 2025.
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL https: /tts:/traae taass forannforrmm rmmo-meeee ei reuits. pub/ hub/20255/ twtattttruuuiooo graphs biology. html #dives addition.
Peiyuan Liu, Hang Guo, Tao Dai, Naiqi Li, Jigang Bao, Xudong Ren, Yong Jiang, and Shu-Tao Xia. Taming pre-trained Ilms for generalised time series forecasting via cross-modal knowledge distillation. arXiv preprint 2xXix:403.073000 2024a.
Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, and Yansong Feng. Are ilms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. arXiv preprint arXiv: 240. 2024b.
Qianli Ma, Zhen Liu, Zhenjing Zheng, Ziyang Huang, Siying Zhu, Zhongzhong Yu, and James T Kwok. A survey on time-series pre-trained models. IEEE Transactions on Knowledge and Data Engineering, 2024.
Scan McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, et al. Transformers can do arithmetic with the right embeddings. arXiv preprint 2rXiv:2405.77399 2024a.
Sean McLeish, Avi Schwarzschild, and Tom Goldstein. Benchmarking chatgpt algorithmic reasoning. arXiv preprint wwXiv:2404.03441 2024b.
Kazem Meidani, Parshin Shojaee, Chandan K Reddy, and Amir Barati Farimani. Snip: Bridging math- ematical symbolic and numeric realms with unified pre-training, arXiv preprint arXiv:2310.02227, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-1737. 2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.052/7, 2023.
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations of transformers with simple arithmetic tasks. arXiv preprint arXiv:2102.1. 13019, 2021.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez. Maarten Bosma, David Luan, al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.001 14, 2021.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. An overview of early vision in inceptionvl. Distill, 5(4):e00024-002 2020.
Bruno A Olshausen and David Field, Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311-33 1997.
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 1480, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.31 1/155/111P11.1 142. httpst
13

---

Published as a conference paper at ICLR 2026
Guilherme Penedo, Hynek Kydlicek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https:/ /oppnreview.w.. net. forum? idd=nSckn22aG.
Yasaman Razeghi, Robert Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206, 2022.
Maria Sahakyan, Zeyar Aung. and Talal Rahwan. Explainable artificial intelligence for tabular data: A survey. IEEE access, (11133992-154222 2021.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. arXiv preprint arXiv: 2019.
Jiandong Shao, Yao Lu, and Jianfei Yang. Benford's curse: Tracing digit bias to numerical hallucina- tion in llms. arXiv preprint arXiv:2506.01734. 2025.
Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang. Positional description matters for transformers arithmetic. arXiv preprint arXi::2311.14737 2023.
Safal Shrestha, Minwu Kim, and Keith Ross. Mathematical reasoning in large language mod- els: Assessing logical and arithmetic errors across wide numerical ranges. arXiv preprint arXiv:2502.08660 2025.
Jasivan Alex Sivakumar and Nafise Sadat Moosavi. How to leverage digit embeddings to represent numbers? arXiv preprint arXiv:2407.0082) 2024.
Samuel Stevens, Emily Wenger, Cathy Li, Niklas Nolte, Eshika Saxena, François Charton, and Kristin Lauter. Salsa fresca: angular embeddings and pre-training for ml attacks on learning with errors. arXiv preprint arXiv:2-42.010082 2024.
Dhanasekar Sundararaman, Shijing Si, Vivek Subramanian, Guoyin Wang, Devamanyu Hazarika, and Lawrence Carin. Methods for numeracy prrerrccvii word embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4753-4753, 2020.
Mingtian Tan, Mike A Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen. Are language models actually useful for time series forecasting? arXiv preprint arXiv:2406. NXKv:2400616664. 2024.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal. Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Avijit Thawani, Jay Pujara, Pedro Szekely, and Filip Ilievski. Representing numbers in nlp: survey and a vision. arXiv preprint arXiv:2 13136, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.002288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thouuht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2433 2022.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok. Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 000009//28/4 /2284, 2023.
14

---

Published as a conference paper at ICLR 2026
Jonas Zausinger, Lars Pennig, Anamarija Kozina, Sean Sdahl, Julian Sikora, Adrian Dendorfer, Timo- fey Kuznetsov, Mohamad Hagog, Nina Wiedemann, Kacper Chlodny, al. Regress, don't I guess a regression-like loss number tokens for language models. arXiv preprint arXtv:2411.0208. 2024.
Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, and Eric Xing. Megamath: Pushing the limits of open math corpora, 2025. URL https: //arxiv. org/ abs/ 2504. 02807.
Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? study in length generalization. arXiv preprint arXiv:2310.1602 2023a.
Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36:43322-43355, 2023b.
Tianyi Zhou, Deqing Fu, Vatsal Sharan, and Robin Jia. Pre-trained large language models use fourier features to compute addition. arXiv preprint arXiv:2406.03445. 2024.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A Detailed Algorithms for Computing FoNE, and Making Predictions
17
FoNE on Binary Classification and Modular Arithmetic Task
18
C More Related Work
20
D Preliminaries and Missing Proofs
21
D.I Preliminaries
D.2 Missing Proofs
21
More Evidence
23
E.1 LLMs Struggle with Multi-digit Arithmetic
23
E.2 Emergence of Fourier Features during Pre-training
FoNE for 60-digit Integer Addition in One Forward Pass
25
G Combining FoNE with Abacus
25
More Details on Experimental Setup
26
H.l Ablation Study
27
I Replicating Results on GPT2-Large Based Model
28
J R2 Comparison for Different Arithmetic Tasks
29
K Comparison with Number Token Loss (NTL)
29
K.1 Experimental Setup
29
K.2 Results.
30
16

---

Published as a conference paper at ICLR 2026
A DETAILED ALGORITHMS FOR COMPUTING FONE, AND MAKING PREDICTIONS
In this section, provide the detail pipeline and algorithms of how compute FONE, get the final loss and final prediction as defined in Section 3.1.
We first show the how are FoNE and Fourier number decoding integrated with regular Transformer pipeline.
(e)
Input
(d)
Fourier Number Decoding
Imput Seqpanoe
Nambers: (10.649
Tokens
18 y 64
[Num]] + (Num)
18
[b)
Fourier Number Emneeddig
lx
10
LBB
Prediction (h, by, hg) -
Last hidden
by
(c)
Transformer
FoNE(EB)
2n
2m
110
100
22
100
22
00.0-1
Fourier Number
sood
Embedding:; [Num]]
Figure (a) We extract all the numbers from the input sequence. (b) For each number, we use FoNE to directly map the number to its embedding. The first two entries in the embedding represent 18 mod 10, while the next two entries represent 18 mod 100. (c) We pad the FoNE with zeros, add it to the word embeddings, and then feed the combined embeddings into the model. (d) For each digit, we take every two entries from last hidden state and find the number whose representation is closest to these two entries.
Next we show the exact algorithms we use to compute FoNE, compute loss and make the final prediction.
Algorithm 1 Fourier Number Embedding (FoNE) Algorithm
I., m Zzo n Zzu, d E Z,0) 2: Inputs: Number z, integer digit length m, decimal digit length n, embedding dimension d
procedure FOURIERNUMBBREEBEDDDSS
3: Initialize empty embedding vector FoNE +
4: for i = -7
do
b Loop over all scales from 10-*+1 to 10th Set the period for the current seale Compute the circular embedding for scale Ti
5:
T. < 10
6:
d(x,Ti) (cos( ž7 x), sin ( x))
7:
Append d(x, T,) to FoNE end for
Add the embedding for this scale the result
8:
9:
while (FoNE) dl do
Ensure embedding dimension matches the target
10:
Append 0 to FoNE end while
4 Zero-pad
12: return FoNE
end procedure
17

---

Published as a conference paper at ICLR 2026
Algorithm 2 Fourier Number Loss & Prediction
function FOURIERNUMBERT SOREERRRNNNNELLLSOOCCOCOOOOO y,
2: Yi - the i-th digit of
3: < [ h(2 + 1]
4: + [o(0, 10), (
5: logits a a
6: loss < (y: logits)
Cross-entropy loss for digit i
7: return loss
8: end function
9: function FOURIERNIMBERRREDCCTOON i)
Prediction for digit
10: logits + [+ 422 + 1] [6(j. 10) j=0.....
D/ f arg m logits[j]
12: return 1/
end function
Algorithm 3 Fourier Number Final Loss & Prediction
1: function FOURIERN y, m, n.)
b Compute average loss
2: totalLoss <-
3: I < [m + nl]
4: for i do
5:
digitLoss < FOURIERNUNBEELOOSFUUTTOIO y,
6: totalLoss + totalLoss + digitLoss
7: end for
8: finalLoss < LosalLoss
Average over all digit positions
[2
9: return finalLoss
10:
end function
11: function HOTREERNIMMRRRFIPPPPPDDTTTOO m, m)
D Compute final prediction
12:
13: Itre + IR..... 1]
b Fractional digit indices D Integer digit indices
Iimt n,...,
15:
for i E Time do
logits; + [h(2i], h(2i 1]] [o(j. 1 jo.....
17:
U, - arg max jetoxeEeo...... (0.....) logits,
18:
7 - yj + Di 10 (n-f)
Scale fractional part by 10-(n-t)
19:
end for
for j E Iim do
21:
logits + [h[2j], + +1] (o(i. j=0....9
22:
Tj - arg max logits,
23:
ワ + Ūj 10
Scale integer part by 10
24:
end for
25:
return 7
26:
end function
B FONE ON BINARY CLASSIFICATION AND MODULAR ARITHMETIC TASK
In this section, we demonstrate that FoNE outperforms other methods on binary classification tasks and modular arithmetic tasks, benefiting from precise representation.
Binary Classification Task Each example in the dataset is formatted as [numl, num2, num3] ).
where the integers num1, num2, and num3 sorted in ascending order (numl < neun2 S num33 to ensure uniqueness and eliminate duplicate representations of the same combination. The integers are uniformly sampled from the range [0, 1000). The label for each example is determined by evaluating the linear equation
numl bb num2 + num3 d,
18

---

Published as a conference paper at ICLR 2026
using predefined coefficients and = - 190. If the result is greater than zero, the label is assigned as 1; otherwise, it is assigned as 0. The dataset is divided into training, validation, and test subsets, outlined in Table 10. Figure 7 and Figure 8 show that FONE outperforms the regular embedding method, XVAL, and even a fine-tuned Llama-3.2-BB model by requiring less data and achieving higher accuracy.
= b-,, 0.5, and d = 10 and 1 = c 0.5, c=
Bosi Accuracy (8)
Results Accuracy
100
100
Ours [Best Afc=20.8055.
Accunicy (%)
60
Anstured Ilama- -b-inctruct [Best Acc-98.1958)
Reguaa thitt wiss) [Best Acc- 165%%]
40
Raggdar (oubwardd [Best Acc-07.20%)
Kval (Best Acc- 54.30%
40
Durs [Aest Acc=900.81%
Regular (diglt- se) [Best 9ct 175%)
20
Regudarlsuwoord (subword! [Bett 725%)
Fingtined on Liama 3.2 18 Instruct [Best [Best Acc-97.75)
0
Xhal [Best Acc-55.885K]
Thaining Data (Lag Scate)
(killoon Parameters)
(a) Accuracy vs. Training Data Size
(b) Accuracy vs. Model Size
Figure 7: We train Llama-3.22 B. from scratch with random initialization using different number embedding methods on number classification where d = 10. The test accuracy is compared across varying data sizes and model sizes.
Rosi Its Accuracy (80)
Besults Accuracy (%)
100
100
80
Auturny (%)
Azcumer
Ours |Best Acc-50.825)
Ours Acc-08. 76556)
Rinetned on lama-3 2:1b-instruct [Best Acc-98.465%]
Regular (digit-wie) [tlest Acci99 435%)
Rogular (dist))
kcc-97.975%4
20
finetuned Lame- 3.2-1B- (Dest acc-98.0755)
Regular (subward) |Best Acc-07.8000 (7.8%%
Regular (subword)) Best A6.-97.6659
Kyal (Best Acce 20%)
Xuak Rcc.)2
0
10
103
10
10
20
40 Model Size (Million Perameters)
50
Training Data Size (Log Scale)
(a) Accuracy Training Data Size
(b) Accuracy vs. Model Size
Figure 8: We train Llama-3.2-4 BB from scratch with random initialization using different number embedding methods on number classification where 190. The test accuracy is compared across varying data sizes and model sizes.
Modular Arithmetic and Base Selection We conduct experiments varying the FoNE base from 7 to 13 and observed clear "sweet spot" at base 10. For example, at base 7 the addition, subtraction, and multiplication accuracies are only 17.66 %, 1.93 %, and 0.54 %. respectively, while base 10 we achieve 100 % accuracy on addition and subtraction after just 4 and 26 epochs, and 99,25 % on multiplication after 50 epochs. Accuracy then fall off again at larger bases (e.g., at base 13: 1.06 %. 5.71 %. and 1.91 %). This confirms that bases that are too small or too large degrade digit-level distinguishability.
We also aligned FoNE's moduli to the numbers' representation base by preprocessing inputs into base 5. In that setting, only the FoNE embedding with period 5 reached perfect accuracy (100 % in epochs), whereas other periods (2-4 and 7-8) yielded near-zero to sub-20 % accuracy (e.g.. period 2: 0.25 %; period 3: 0.12 %; period 7: 16.38 %). This again demonstrates that FoNE performs best when its moduli match the underlying digit base.
19

---

Published as a conference paper at ICLR 2026
Table 6: Accuracy (%) arithmetic tasks under different bases.
Base
Epochs (Add/Sub/Mul)
Accuracy (Add/Sub/Mul)
/ 50 50
17.66 1.93 / 0.54
50 50 50
40,91 6,07 / 0.33
50 50 50
65.49 / 65.63 65.78
10
26 50
100.00/ 100.00/ 99.25
50/ 50
99.99 999.99 78.76
12
50 50 50
99.78 99.99 54.14
13
50/ 50 50
1.06 / 5.71 / 1.91
Table 7: Accuracy (%) with base-5 number inputs, under varying FoNE periods.
FoNE Period
Epochs
Accuracy (%)
50
0,25
0.12
50
0.17
5
6
100,00
50
16.38
99.97
We additionally benchmarked FoNE on modular addition tasks with varying moduli. The goal is to predict + y mod m, where æ, y E [0, m). FoNE consistently outperforms standard tokenization. especially at higher moduli.
Table 8: Accuracy (%) modular addition tasks across various moduli.
Modulus m
FoNE Accuracy
Standard Tokenization Accuracy
100.00
99.98
99.27
99.99
97
100.00
0.89
100
100.00
18.17
113
100.00
99.99
121
100.00
5.48
225
100.00
65.10 10
256
100.00
70.20
257
100.00
11.36
These experiments validate our claim that FoNE is most effective when its moduli align with the target numeral base, and it remains robust even in high-modulus arithmetic settings where standard tokenization breaks down.
C MORE RELATED WORK
Fourier Features. Fourier features are commonly observed in image models, particularly in the early layers of vision models Olshausen Field (1997); Olah al. (2020); Fiquet & Simoncelli (2024). These features enable the model to detect edges, textures, and other spatial patterns effectively. However, Transformers struggle capture high-frequency components (Bai et al., 2022; Tancik et al., 2020). Augmenting data with figh-frequency components or explicitly encoding coordinates using Fourier features has been demonstrated to improve model performance (Tancik al., 2020; He et al., 2024; Hua et al., 2024). In fact, the original Transformers paper (Vaswani et al., 2017) uses Fourier features to encode the position information of tokens: however, it does not apply this idea to number tokens to aid with numerical tasks. In modular addition tasks, studies have revealed that after "grokking," a one-layer Transformer can learn to solve the task perfectly by leveraging Fourier features (Nanda et al., 2023; Gu et al., 2024). Furthermore, Zhou et al. (2024) demonstrate that LLMs naturally encode numbers using Fourier features during pretraining, leveraging these representations for arithmetic tasks (Levy & Geva, 2024; Kantamneni & Tegmark, 2025; Lindsey
20

---

Published as a conference paper at ICLR 2026
al., 2025). Building this insight, propose using modular Fourier components to explicitly represent digits, enabling models to perform precise numerical computation. This allows algebraic operations to be carried out in component-we parallel manner and overcomes the limitations of token-based number representations.
D PRELIMINARIES AND MISSING PROOFS
D.1 PRELIMINARIES
In this section, provide the necessary mathematical definitions and concepts used throughout the paper.
Period and Frequency. A function (() is periodic with period T 0 Of [(r+ T)) f(x) for all I. The period 7 represents the smallest positive value for which the function repeats. The frequency of periodic function the reciprocal of its period, f 7.1 and describes the number of cycles completed one unit interval. For the sine and cosine functions cos (7x and sin ()) the period is T.
Unit Circle. The unit circle is the set of points in the plane a distance of 1 from the origin, given by x² + y² 1. The coordinates of points the unit circle can be parameterized as (cos 0 sin 0), where 0 is the angle measured counterclockwise from the positive r-axis. For any angle 0 cos represents the z-coordinate, and sin 0 represents the y-coordinate.
Aro-mettment Inverse Tangent. The two-argument inverse tangent function, alan2 x), deter- mines the angle 0 (modulo 2r) given the coordinates (x, y) (cos 0, sin Specifically,
0 atan2(y, x)),
which resolves the angle 0 uniquely based on the signs of z and
Modular Arithmetic. Modular arithmetic considers equivalence classes of numbers under modulus T O. For integers a and b, a = b (mod T) if T (a b), meaning and b differ by an integer multiple Tf
Fourier Representation. Periodic functions with period T can be represented using the fundamen- 2x tal frequencies T For example, the embeddings (cos 2m xxx). 2), sin (2x z)) capture the periodicity modulo T by mapping it to unique point on the unit circle.
D.2 MISSING PROOFS
In this section, we provide some missing proofs. Lemma D.1 (Formal version of Lemma 3.3). Given the pair (cos( 27 x)) sin( 25 x)). we can recover mod! T.
Proof. Let = 2x T x. Then the given pair becomes
(cos(0), (in)())))nn(()).
From this pair, one can recover / uniquely modulo 2n. Concretely, 0 can be obtained (modulo 2r) using the two-argument inverse tangent function:
f = asinn()in(0),, cos( (mod 2T))
Since =
2x T I, we have
T
T
2m
Hence is determined up integer multiples of T. i.e., x mod T. In other words, if
(cos( 27.1) sin( 27 x1))
(cos( 27 ao2)) ( sin 12)),
21

---

Published as a conference paper at ICLR 2026
then 7 (cos( 7 x),
22 (mod 2TT), which implies 21 22 (mod T). Therefore, from the pair sin( 25 r)). indeed recover mod T.
Lemma D.2 Laaerr-ormmalized FONE Preserves Numeracy). Given number's Layer-Normalized Fourier Number Embedding LN(FoNE(z) + p), where FoNE(z) is the Fourier Number Embedding of and p is an orthogonal positional encoding vector, assume the mean of FoNE(x) + p is O. Let m be the integer digit length of x and n be the decimal digit length of z. Then, using Lemma 3.3, we can recover z mod 10s for each integer the range n + 1 m.
Proof. Assume the mean of x FoNE( + is O, i... μ 0. Under this assumption, LayerNorm simplifies to:
LN(x)
where a is the standard deviation of x.
Let FoNE(x) encode the scalar and let p be orthogonal positional encoding vector such that:
||l|| |||| = and u p = 0.
Then, the input to LayerNorm is:
x= u P.
The standard deviation of is given by:
(x: 12)²,
where d is the dimensionality of x. Since
O, this simplifies to:
x
Substitute x = u1
|||| = ||u + pll = ||u|| + ||p|| + 2u P.
By orthogonality and unit norm, u p 0, |u|| = 1, and ||p| 1. Thus:
||*|| +++++=-
Therefore:
V 2 = d
The LayerNorm operation simplifies to:
x
u u p
d u (u+p).
LN(x)
This rescales u and p by factor of
The key observation is that LayerNorm applics a uniform scaling to all components x. Since u and p are orthogonal and their relative directions preserved, the numerical relationships encoded in u (which represent .t) are preserved up to a scaling factor.
By Lemma 3.3, the numeracy of is preserved. This means recover mod 10* for all in the range -n + 1 < i S m, as the normalized embedding retains the necessary information about x.
22

---

Published as a conference paper at ICLR 2026
The same result holds for RMSNorm because it also applies uniform scaling (based on the root mean square of the input) while preserving the relative directions of the embedding components, thus maintaining the numeracy of æ.
Notet that standard sinusoidal positional encodings (PEs) (Vaswani et al., 2017) cannot serve as a substitute for numerical embeddings. PEs explicitly constructed to distinguish token positions in a sequence, not to represent the magnitude or digit structure of real numbers. They use a fixed set of exponentially spaced frequencies ensure each position has a unique yet non-invertible signature; as a result, there is no straightforward way to recover the original numeric value (or its individual digits) from a PE embedding. Moreover, the frequencies in PEs are chosen for positional uniqueness, not for digit-aligned modular arithmetic, so small changes in numeric value can produce large, non- con-monotonic changes in the embedding ppace-precisefe the opposite of the smooth, digit-wise variation required for accurate number encoding. Unlike FNE's digit-aligned sinusoidal components (e.g., mod 10, mod ..), PEs do not guarantee invertibility with respect to numeric operations.
E
MORE EVIDENCE
E.1 LLMs STRUGGLE WITH MULTI-DIGIT ARITHMETIC
We evaluate five production LLMs (Claude 3.7 Sonnet, DeepSeek V3.1, Gemini 2.5 Flash, GPT-5, and Qwen3-235B) on direct arithmetic without chain-of tf-thughhh code execution, or tools. For each setting, we sample 100 IID problem instances with uniformly distributed operands over the full d-digit numbers. Tasks include d E {3, 4, 5, 6} for multiplication and d E {7, 8, 5,50 for addition. Models receive two-shot, operation-matched prompt and must return single integer only. Decoding uses temperature O. We parse the first integer token from the response and score exact-match accuracy by numeric equality with the reference.
Multiplication
Addition
Model
4
6
8
50
Claude 3.7 Sonnet
1.00
0.53
0.02
0.00
1.00
0.97
0.82
DeepSeck V3.1
0.94
0.25
0.01
0.00
0.95
0.95
0.34
Gemini 2.5 Flash
0.58
0.11
0.00
0.00
0.99
0.96
0.33
GPT-5 Qwen3-235B
0.74
0.09
0.00
0.00
0.98
0.92
0.66
0.94
0.62
0.05
0.01
0.99
0.98
0.28
Table 9: Exact-match accuracy on direct multi-digit arithmetic; columns indicate digits per operand. Each entry averages 100 IID problems per setting with uniformly sampled d-digit operands. Two-shot prompting, single-integer output only.
Results. Even the most recently released L.Mss still struggle with multi-digit multiplication, while fail to achieve perfect accuracy on addition. For multiplication, accuracy drops sharply as operand length increases: models perform well 3-digit multiplication (2 0.58), but fall below 0.10 for 5-digit cases and nearly 0 at digits. Qwen3-235B is the most robust, reaching 0.62 on 4-digit and 0.05 on 5-digit multiplication, yet still fails 6 digits. For addition, all models achieve > 0.95 7-8 digit tasks, but accuracy declines on long-sequence addition (50 digits), ranging from 0.28 (Qwen3-235B) to 0.82 (Claude 3.7 Sonnet). In summary, LLMs excel at addition with short to medium operands but remain brittle for both long-sequence addition and especially large-digit multiplication.
E.2 EMERGENCE OF FOURIER FEATURES DURING PRE-TRAINING
We follow Zhou et al. (2024) and conduct the same Fourier analysis on Pythia model. Figure 9, we show how Pythia gradually learns the Fourier features during pre-training. With different model size, the model gradually learn the same frequency components.
We extend the work of Zhou et al. (2024) to other are-trained LLMs and observe similar findings: pre-trained LLMs, regardless of the dataset used, tend to learn the same outlier frequency components.
23

---

Published as a conference paper at ICLR 2026
Analysis of Number Embbodd nioo Pythia Differont Checkpoints
(atta 14m
875
203 Chocpper pesciic
20
Chack pori digy 200o
Cheis Digiene
comporer
roure uampeer:
coder comporee
rCali octpaner
rouer LangON
199-
203
100
203
Figure 9: Fourier analysis the Pythia model's number embeddings across pre-training checkpoints. The figure illustrates how the Fourier features are progressively learned during pre-training, showing the emergence of specific frequency components. Models of varying sizes exhibit similar trend, gradually learning the same frequency components over time.
Number Embedding Fourier Spece pythis-6.99
Number Embeskling Fourreer Space: Lleme-3.2--3
3.5
sin
3.5
Hagnitute
T-30 03
7-2.00
2.5
1.5
7::10.00
T=5.00
T-5.20
[w4.00
50
1.5
10.00
10.00
135.83
T-3.5
Tw2 50
1-3,35 35
0.5
50
100
150
200
250
100
150
200
250
Fourier Component
Fourier Component
(a) pre-trained Pythia
(b) fine-tuned Llama3.
.2
Number Embediding Spare: opt-1.3h
Number Frhheddin Fotarier Space: gin2-llarge
5
sin
Magnitus
T- c01
T-110.00
1-5.00
T-3,00
1=2.00
Magnitude
T-10.025
T1..00
T-2.00
1-10.03
5..000
Tw2 RP
50
100
150
200
250
50
100
150
200
250
Fourier Component
Fourler Component
(c) pre-trained OPT
pre-trained GPT2
Figure 10: Number embedding in Fourier space for different pre- pre-trained models.
24

---

Published as a conference paper at ICLR 2026
F FONE FOR 60-DIGIT INTEGER ADDITION IN ONE FORWARD PASS
Accuracy Percentage (Aetrage 97.42%)
10
0.6
62
30
Second Numberu Lenghh
Figure 11: Accuracy of an 8-layer transformer on 60-digit addition tasks, illustrating the effectiveness of FoNE embeddings in handling long sequences. The model achieves an average accuracy of 97.42% across different operand lengths, showcasing its capability in numerical precision and sequence representation.
As discussed in Section 5, the maximum digit length that a float 64 data type can precisely represent 15 digits. Consequently, even we convert numbers float 64 and then back to float 1 6 to match the model weight precisionm it still introduce numerical inaccuracies when the input x exceeds 15 digits. To mitigate this issue, we process x by dividing into smaller chunks, allowing the FoNE to operate effectively without precision loss.
Specifically, æ is split into groups five digits, and FoNE is applied independently to each chunk. Each digit within chunk is encoded into two dimensions, resulting in an embedding of length 10 per chunk. These chunk embeddings are then concatenated to form the final representation of æ. This method ensures that even for long inputs, the FoNE still preserve the numeracy of the numbers.
We adopt the data generation approach from McLeish et al. (2024a), which includes all combinations of operand lengths (i,j) up to maximum length k, generating 20 million stratified samples to ensure balanced representation across all length pairs. Training is conducted using a language model cramming approach (Geiping & Goldstein, 2023), constrained to exaFLOP (equivalent to 24 hours of training on a single Nvidia RTX A6000 GPU). Using this strategy, as depicted in Figure 4(a), an 8-layer transformer trained 60-digit addition achieves average accuracy of 97.44 97.42% across various operand lengths in just one forward pass. This result underscores the effectiveness of the FoNE in processing long numbers with high precision and computational efficiency in just one forward pass.
G COMBINING FONE WITH ABACUS
We train decoder-only causal language models to solve arithmetic problems, following the setup described in Mcl.eish et al. (2024a). Inputs are formatted in a least-significan-diiii 98282 3859172 2787472), without padding between digits operands. The training dataset includes all combinations of operand lengths (i.j) up to a maximum length k with 20 million stratified samples ensuring balanced representation across all length pairs.
order (e.g..
For input representation, we combine Fourier Number Embeddings (FoNE) with the Abacus method McLeish al. (2024a). That each digit is embedded with FONE. Training is conducted using a language model cramming approach (Geiping & Goldstein, 2023), constrained to 8 exaFLOP (equivalent to 24 hours of training single Nvidia RTX A6000 GPU).
25

---

Published as a conference paper at ICLR 2026
Accuracy
Accuracy
3.0
83
64
RiZz
Second Number
Second Number Length
FNE Abbacus Accuracy (Seed 21
Abacus Accuracy (Seed 21
30-
68
10-
1ith
20
Firtt
52
00
Gecond Number Length
Second Number Length
FNE+Abacus Accuracy (Seed 31
Abecus Accuracy (Seed 31
1.0
14
Number
32
Length
Length
Figure 12: Heatmaps of accuracy percentages for " (left column) and "Abacus" (right column) across three different random seeds. Each heatmap represents accuracy as a function of the first and second number lengths, with lighter blue shades indicating higher accuracy. The color scale ranges from white (low accuracy) blue (high accuracy). These visualizations highlight FoNE can combine with Abacus to improve performance.
We train and evaluate the models across three runs, each with different random seed, as shown in Figure 12. Results indicate that incorporating FoNE enables the Abacus method to achieve better generalization and higher accuracy.
H MORE DETAILS ON EXPERIMENTAL SETUP
In this section, we provide the experiments settings that we used in the Section 4.1.
Learning rates were determined through extensive search, with the best rates selected separately for each method based on validation performance. Final training hyperparameters include learning rate of 0.005 for regular and FoNE methods, and 0.0001 for the x Val method, a batch size of 512, and 100 epochs. The fine-tuning process required less than 10 hours, while training from scratch took less than 3 days.
26

---

Published as a conference paper at ICLR 2026
Dataset
Train Size
Validation Size
Test Size
6-digit decimal addition
720,000
80,000
200,000
6-digit integer addition
720,000
80,000
200,000
5-digit integer subtract
720,000
80,000
200,000
3-digit integer multiplication
360,000
40,000
100,000
4-digit integer multiplication
720,000
80,000
200,000
classification
720,00
80,00
200,00
Table 10: Dataset Sizes for Training, Testing, and Validation
Dataset
Model Size Varying Data Size
Data Size Varying Model Size
d-iig decimal addition
37,55M
200,,000
6-digit integer addition
37,55M
200,000
5-digit integer subtract
37.55M
200,000
d-digit integer multipicatiion
37.55M
360,000
digit integer multiplication
37.55M
360,000
d-digit integer multiplication
37.55M
360,000
classification
37.55M
50,000
Table 11:
Dataset and Configuration Sizes for Model and Data Variation Experiments
H.1 ABLATION STUDY
In this section, we present the mispredictions the model trained with an FoNE, where the periods multiples of instead of 10. Table 13 demonstrates that, for each digit, the mispredictions consistently deviate from the true labels by 5.
We also present the model's mispredictions in Table 14, where each digit is encoded into separate dimension of the embedding. For example, the number 567 is represented as [5, 6, 7]. During training. compute the RMSE loss between the last hidden states and the labels. During prediction, interpret each entry in the last hidden state as a single digit.
27

---

Published as a conference paper at ICLR 2026
Model
Hidden Size
Intermediate Size
# Hidden Layers
Attention Heads
# Key-
Heads
64
256
2
128
512
2
192
768
256
1024
5
320
1280
5
8
384
1536
Table 12: Model Configuration Table
Table 13: Mispredictions in the Final Evaluation with when embed each digit with only mod5.
Index
Predicted Value
Actual Value
934.03
934.585
2
3.009
558.509
3
912.311
917.366
6201.000
1756.008
5
1240.34
1290.84
I REPLICATING RESULTS ON GPT2-LARGE BASED MODEL
We conduct the same experiments on decimal addition using a GPT-2 Large-based model. The results indicate that changing the model architecture does not affect the outcomes. For instance, GPT-2 Large employs LayerNorm, while Llama 3.2 uses RMSNorm.
Results Accuraty (S)
100
300
80
(5)
Reguiar (d wdig))wise) (est Acc. 99 0055%5]
Accumery (92)
60
Reguar (duwword) (Best Accu 9u 081%)
60
Rinetuned get2 aarge [Best Scck84,805390
[Best 0.0005%)
Acc-100.0s)
kccuacy
Regulár (cigltsise) Acc- 0025%)
Finetined on gpt? large [itest Accw87 999599
Reqular (suswordl [Dest Acc= 25.2515)
A.cc=0.000 0.0%)
Porfect Accuracy
10a
10a
20
10*
Tainnng Data Siow Scale)
Model Size (Million Perameters)
(a) 6-digit decimal addition: Accuracy Train- ing Data Size
(b) 6-digit decimal addition: Accuracy Model Size
Figure 13: We train GPT2-Large from scratch with random initialization using different number embedding methods on 6-digit decimal addition. The test accuracy is compared across varying data sizes and model sizes.
28

---

Published as a conference paper at ICLR 2026
Table 14: Mispredictions in the Final Evaluation when directly encoding numbers into their embed- dings.
Index
Predicted Value
Actual Value
883.888
993.999
2
787.878
898.989
888.7 758
989.759
4
748.785
849.895
677.677
688.788
10
1179.488
1189.499
J Rº COMPARISON FOR DIFFERENT ARITHMETIC TASKS
x XVal Golkar et al. (2023) performs well on the R² metric
R² =1 E De 0:)
E 7)2
because it uses RMSE as its loss function. However, demonstrate that FoNE outperforms x Val on R most tasks. We show the final 1t2 on test dataset our experiments(Section 4.2).
Sesults
11
c.000100
Rogudar (ciglowtte)) [Bust (1w0.994148)
-0,00002
Regider (subwaord) [Beet Ri-O 0.711117)
0.000025
Perfect
-0. 20.00004
C.000000
-0.000075
-0.00006
Jegguar (wdbward) [0-01/99955]
Fineluned Jamm 2-21Binnttttt (Best Rhw6 9975731
0.000075
Ree-Jar (diat-wise) [Best R2-227 6368547
Pertect 87
-0.000100
10
-0.00010
Taining Data Size lLog Scale!
Model Size (Mi lon Parametent)
(a) Data size vs. Accuracy
(b) Model size vs. Accuracy
Figure 14: Comparison of R2 trends for 6-digit decimal addition with respect to model size and data size.
K COMPARISON WITH NUMBER TOKEN Loss (NTL)
To validate the effectiveness of FoNE, we conduct comprehensive experiments comparing against Number Token Loss (NTL) (Zausinger al., 2024), method designed to improve numerical reasoning by incorporating new losses between number tokens. We evaluate two NTL variants: NTL with diget-wise tokenization (denoted NIL-D),, where numbers decomposed into individual digit tokens, and NTL with subword tokenization (NTL-S), using the standard tokenizer with an extended number vocabulary covering tokens (-999.. For each variant, test three lambda values (A E 10.15,00 0.3, 0.83) that control the relative weighting of the NTL loss component.
K. EXPERIMENTAL SETUP
We conduct experiments on two arithmetic reasoning benchmarks: 4-digit integer addition and 3-digit integer multiplication. To assess data efficiency, we train models with varying amounts of data: 5K, 25K, and 50K samples for addition; 10K, 50K, and 75K samples for multiplication. All experiments use 1lama-3.22-1 1.2-IBB configuration, training from scratch with consistent architectural parameters (512 hidden dimensions, layers, attention heads) to ensure fair comparison. We train all models for 50 epochs with batch size 512, using learning rates of 5 10 for FoNE and x 10-3 3 for NTL methods based on preliminary tuning. Models are evaluated on 1000 held-out test samples using whole number accuracy, R²2 and RMSE.
29

---

Published as a conference paper at ICLR 2026
0.0030
Taxdb -
820011
10000
0.09912
03
120012
Sgt
-000934
C.0600
C0000
-3000122
-021300
09964
-3002112
0.-
(a) 6-digit integer addition: Model& Data size vs. Ac-
Soeld
MokiSiee Bin hateetesl
33'
Thaning Sug
122
307
D0F
20
Farameters)
(b) S-digit integer addition: Model&Data size vs. Ac- curacy
curacy
Senu 27
1000
100000
-E.00902
2.66
-0.0000
-E:0005
-0.0000
09564
-0.0000
0.000
0.000
Thaning Dala Suee Lon Scakd
Model Miilin Tarmetens
100
Thannng Deta Size luog Suelch
100
32*
-0.00100
Mode: Siee (Milion Tarmmetnii
(c) 5-digit integer subtraction: Model&Data size (S.(d) 3-digit integer multiplication: Model&Data size vs. Accuracy
Accuracy
Figure 15: Comparison of R² trends for various arithmetic tasks with respect to model size and data size.
K.2 RESULTS
Figure 16 presents comprehensive performance comparisons across both tasks and all metries. The results demonstrate FoNE's substantial and consistent superiority over all NTL variants.
Addition Task FoNE reach 98.6% accuracy with only 5,000 training samples and perfect 100%
accuracy with 25,000 or more samples (Table 15). The R scores consistently exceed 0.999, and RMSE drops to effectively zero at larger training scales. NTL methods exhibit significantly de- graded performance. The best-performing variant, NTL(D,0.15), achieves 97.4% accuracy with 50K samples-still still below FoNE's performance at 5K samples. Other NTL configurations struggle considerably: NTL(D,0.33 peaks at only 51.4% accuracy, while higher lambda values (A = 0.8) yield near-zero performance (3.3-6.1%) 16). Most critically, NTL with subword tokenization fails catas- trophically across all configurations, achieving near zero accuracy with negative R2 values as low as -13, 170, indicating performance substantially worse than naive baseline. RMSE values for NTL(S) exceed 4.7 x 10S, compared to 0.0 for FONE.
Multiplication Task The multiplication task proves more challenging, yet FONE maintains strong performance. With 75,000 training samples, FoNE achieves 89.3% accuracy with Re=0.9996 and RMSE=4,286, demonstrating consistent and reliable predictions (Table 16). Performance scales smoothly with data: from 35.1% 10K samples to 84.5% 50K and 89.3% 75K. NTL methods again significantly underperform. The best NTL configuration, NTL((,0015) with 75K samples, achieves only 16.3% accuracy- 73 percentage points below FONE. Most other NTL variants achieve less than 5% accuracy even at the largest training scale. Subword tokenization continues to fail, with negative R²2 scores (e.g.. -0.534 for NTL(S,0.8)) and RMSE values exceeding 2 x 10 three orders of magnitude worse than FONE.
Analysis Several key observations emerge from our experiments. First, FoNE demonstrates superior data efficiency, achieving strong performance with minimal training data while NTL requires substantially more samples to reach moderate accuracy levels. Second, NTL's performance is highly sensitive to the lambda hyperparameter. increasing À from 0.15 to 0.8 consistently degrades accuracy across both tasks, suggesting that heavily weighting the distance- based loss interferes with
30

---

Published as a conference paper at ICLR 2026
Comparison: Methods
Addition Accurscy
Addition
Addiion RMSE
140
Numteen (50)
E.C
SOun
103
Sumpler
Sumplea
Sumpler
Xultpleation-Accee Accuracy
Xulliphcatan WV
Kaltglication HOSSE
108
156)
22.0
X.C
Samples
Figure 16: Performance comparison of FoNE vs, NTL methods on addition (top) and multiplicatiion (bottom) tasks across three metrics: accuracy, R², and RMSE. FoNE (solid blue squares) consistently outperforms all NTL variants, achieving 100% accuracy on addition and 89.3% on multiplication vs. NTL's best of 16.3%.
Table Addition task results comparing FoNE with NTL methods across different training data sizes. FoNE achieves perfect accuracy with 225K samples while all NTL variants struggle.
Method
5K Samples
25K Samples
50K Samples
Acc. (%)
R²
Acc. (%)
R²
Acc. (%)
R²
FoNE
98.6
0.9993
100.0
1.0000
100.0
1,0000
NTL(D.0.15)
0.9
-3.57
96.3
0.9998
97.4
0.623
NTL(D.0.3
41.2
0.107
51.4
0.506
49.1
0.887
NTL(D.0.8)
3.3
-0,304
6.1
0.774
5.3
0.773
NTL(S,0.15)
0.0
-2.41
4.6
0.957
8.6
-39.3
(S,003)
0.0
-7763
0.1
-13030
5.7
-8884
NTL.(S,0.8
0.1
-8418
3.6
-13170
0.7
-13730
learning. Finally, FoNE also exhibits better training efficiency, requiring 2.5 11.7x less training time than NTL methods while achieving far superior accuracy.
Table 16: Multiplication task results. FoNE maintains strong performance (89.3% at 75K samples) while the best NTL variant achieves only 16.3%.
Method
10K Samples
50K Samples
75K Samples
Acc. (%)
n?²
Acc. (%)
RR²
Acc. (%)
I2a
FoNE
35.1
0.9932
84.5
0.9994
89.3
0.9996
NTL(D.0.15)
0.3
0.955
5.1
0.993
0.9991
NTL(D.0.3)
0.2
0.958
3.1
0,978
8.2
0,990
NTL(D.0.8)
0.0
0.928
0.4
0.969
0.5
0.983
(S.
0.2
0.957
2.2
0.944
4.7
0.983
NT..3S.033)
0.2
-0.462
0.7
-0.032
3.9
-0.107
NTL.(S.0.))
0.3
-0.132
1.9
-0.525
1.6
-0,534
31

---

Published as a conference paper at ICLR 2026
THE USE OF LLMS
LLMs were used only to polish language, such as grammar and wording. These models did not contribute to idea creation writing, and the authors take full responsibility for this paper's content.
32


Reviews:
[{"review_id": "7flAOUsbWX", "reviewer": "Reviewer_Cez7", "summary": "This paper proposes a new embedding scheme for numerical tokens, based on past empirical work showing that number embeddings often have fourier-like features. By directly encoding numbers in a pre-determined fourier-like scheme, they are able to dramatically improve an LLM's ability to do mathematical tasks.", "questions": "See weaknesses!", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The empirical results are super strong! They had the baselines I was looking for too (e.g., encoding each number as its own token).", "weaknesses": "I am curious about how this encoding scheme would impact training perplexity and stability on normal LLM training settings, where we have number prediction mixed in with regular language modeling loss. The paper talks about how this could work easily by using a special number token. I think for this work to really get adopted by a next-gen frontier or open-source model, it would require someone exploring sweeps of this at least at the small or medium pre-training scale.\n\nI’m also interested how the authors think we should be handling cases of how messy tokenization can get in practice, e.g., real numbers look like everything from $1.000 to $1,000 to one thousand and 99/100 to 5263–36181 etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bybkAXN99Q", "reviewer": "Reviewer_gsTU", "summary": "This paper proposes a special method called Fourier number embedding to tokenize numbers in natural language, which is defined as a list of sinusoids. A special token [NUM] is used to handle its integration with other word embeddings in both the encoding and decoding stages. Experiments show the proposed methods are effective in doing arithmetic tasks.", "questions": "Q1&2: see weaknesses.\nQ3: In the evaluation (Figure 3), is there a strict control that samples in the test set are not in the training set?\nQ4: How would the author explain the result in Table 8 (authors name it Table H in the appendices), where the arithmetic ability, though higher than that of the pre-trained model, is still lower than the ones in Figure 3? How to falsify the assumption that large models that understand languages to some extent (like GPT/LLama) cannot understand numbers well, regardless of the tokenization method?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper identifies a well-known limitation of LLMs, their weak representations and reasoning over numbers.\n2. The proposed approach is clearly explained and easy to follow, and the experiments demonstrate meaningful gains.", "weaknesses": "1. There are probably not sufficient discussions and experiments on the performance of tasks involving floating point and other numbers that do not necessarily represent a value (e.g., year, flight number, music pitches, etc.).\n2. The contribution of the paper is quite limited on the arithmetic task, which can be done in fact by a calculator, not an LLM. A further exploration can be related to how improved number representation enhances natural language inference, in some tasks related partially to numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pnSutrPWPO", "reviewer": "Reviewer_KuCM", "summary": "The paper introduces FoNE (Fourier Number Embedding), a novel method for representing numerical values in language models. FoNE maps each number directly into an embedding using sine and cosine functions with multiple periods (powers of 10). This creates a single-token representation that encodes both the magnitude and structure of numbers.\n\nEmpirical evaluations on synthetic arithmetic tasks (addition, subtraction, multiplication) show that a small Transformer (38M parameters) trained with FoNE achieves perfect or near-perfect accuracy while being more data- and parameter-efficient than subword, digit-wise, or XVAL embeddings. The paper also discusses integration with existing models, such as continual pretraining on LLMs, and provides theoretical justifications for why FoNE preserves numeracy through modular encoding.", "questions": "Interestingly, if one compare FoNE with the classic sinusoidal positional encoding (from Vaswani et al., 2017, Attention Is All You Need), they share very similar expressions (compared to Definition 3.2 in this paper, when switching $pos$ with the encoded number):\n\n$$PE_{(pos, 2i)} = \\sin\\left(\\frac{pos}{10000^{2i/d_{\\text{model}}}}\\right), PE_{(pos, 2i+1)} = \\cos\\left(\\frac{pos}{10000^{2i/d_{\\text{model}}}}\\right)$$\n\nIn its essence, every PE is a way to encode a number (position index). Would it be possible to modify other forms of PE into number embedding methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[S1] Clear motivation and valuable conceptual connection. I think it's a valuable connection between the previous finding that LLMs internally learn Fourier-like features and the clear need for accurate number embeddings (fragmented multi-token representations and frequency-based biases).\n\n[S2] The empricial results are strong. Across multiple arithmetic tasks, FoNE consistently outperforms established baselines such as digit-wise, subword, and XVAL embeddings. The method achieves up to 64x higher data efficiency and strong parameter efficiency, reaching perfect or near-perfect accuracy even with a 38 M-parameter model, surpassing fine-tuned large models like Llama-3.2-1B on the same tasks. The analysis also highlights faster training and inference, fewer tokens per number, and robustness across different architectures, which together underscore the practicality and scalability of the approach.\n\n[S3] Theoretical grounding is solid. The paper provides clear and mathematically sound explanations for why FoNE preserves numeric precision (via modular recovery and periodic embeddings).", "weaknesses": "[W1] Benchmarks are still synthetic. All experiments are conducted on synthetic arithmetic datasets, which may not fully reflect the complexities of real-world numeric reasoning (e.g., financial QA, table reasoning), where additional challenges might present. E.g., numbers can take different forms (e.g., \"eighteen\", \"XVIII\", $18), or with units associated (18g vs 0.018kg), and contextual grounding might be essential. \n\n[W2] Model scale. The core experiments use very small Transformer model (38M) where text capabilities are not tested. It remains unclear whether FoNE can coordiate well with LLMs' semantic embedding space of pretrained text tokens without introducing instability. For example, there might be a misalignment between the embedidng spaces of texts and numbers; Another concern is that FoNE removed number semantics entirely: information such as \"integers around 2000 is more likely to be a year number\". The paper briefly argues MLP layers preserve semantics but it remains to be empirically validated. A demonstration of how FoNE interacts with pretrained embeddings or retains contextual numeric meaning would strengthen the paper’s impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uCZ8ex6uTz", "reviewer": "Reviewer_pYSK", "summary": "The paper presents a new number encoding for LLMs based on Fourier features. The embedding is the sum of a number token and the separately constructed Fourier number embedding (FoNE). Experiments on synthetic datasets with addition and multiplication tasks show the superiority of the method over other number representations such as a simple single-digit-encoding or the xval approach.", "questions": "1. The main question is how the method compares to a simple transformer model on a standard numerical reasoning dataset.\n2. m and n are the number of integer and decimal digits, but m and n seem to be set for each number, and not by the maximum m and n of the dataset. Doesn’t this lead to issues because the resulting FoNE vectors are not aligned, I.e. the third cell for 1000 would represent the tens digit but the third cell for 0.12 would be one  hundredths? \n3. In definition 3.6, why is it \\phi(0, 10) to \\phi(9,10)? I expected it to be dependent on i and not fixed to 10.\n4. How efficient is the end to end training? The manuscript claims that this method is even more efficient than the baselines since fewer tokens are used, but I imagine that computing the FoNE representation and adding it to the embedding vector creates an overhead. How long does training one epoch take in comparison to the baselines?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The approach is novel and makes sense. Using Fourier features is intuitively more suitable for numerical reasoning than standard number tokenization, and the encoding-decoding approach is elegant.\n2. The paper is well written and the methodology is easy to follow.", "weaknesses": "1. The experiment is not sufficient to show the superiority of the approach. Why is the method not tested on standard benchmarks for mathematical reasoning, such as GSM8K or MATH? The synthetic datasets is very artificial since it only contains equations of fixed-digit numbers, no other text. However, number representations in LLMs are only relevant for data that consists of both text and numerical tasks. \n2. The method seems incompatible with existing models since it requires specific modifications to the tokenizer and an edit after the embedding, so it is not applicable for finetuning, only for training models from scratch. This makes it difficult to evaluate the effectiveness, and limits the applicability of the method.\n3. The method is only compared to xval as a baseline approach, although there have been other papers on this topic, such as the Number Token Loss (NTL) [1]. In the discussion section, it is argued that NTL is not relevant since \"regression produces continuous value\", but NTL is only a loss function that does not impose any constraints on what the number looks like. It should be comparable. In general, the paper only reports accuracy as a metric, although the numeric value clearly matters and regression metrics such as RMSE would be suitable. \n\n[1] Zausinger, Jonas, et al. \"Regress, Don't Guess--A Regression-like Loss on Number Tokens for Language Models.\" arXiv preprint arXiv:2411.02083 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)