Number Cookbook: Number Understanding of Language Models and How to Improve It
Authors: Haotong Yang
Yi Hu
Shijia Kang
Zhouchen Lin
Muhan Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: number understanding
large language model
reasoning
Primary area: datasets and benchmarks
TL;DR: A benchmark, test and investigation about number understanding and processing ability of large language models

Abstract: Large language models (LLMs) can solve an increasing number of complex reasoning tasks while making surprising mistakes in basic numerical understanding and processing (such as $9.11 > 9.9$). The latter ability is essential for tackling complex arithmetic and mathematical problems and serves as a foundation for most reasoning tasks, but previous work paid little attention to it or only discussed several restricted tasks (like integer addition). In this paper, we comprehensively investigate the numerical understanding and processing ability (NUPA) of LLMs. Firstly, we introduce a benchmark covering four common numerical representations and 17 distinct numerical tasks in four major categories, resulting in 41 meaningful combinations in total. These tasks are derived from primary and secondary education curricula, encompassing nearly all everyday numerical understanding and processing scenarios, and the rules of these tasks are very simple and clear.
Through the benchmark, we find that current LLMs fail frequently in many of the tasks. To study the problem, we train small models with existing and potential techniques for enhancing NUPA (such as tokenizers, PEs, and number formats), comprehensively evaluating their effectiveness using our testbed. We also finetune practical-scale LLMs on our proposed NUPA tasks and find that 1) naive finetuning can improve NUPA a lot on many but not all tasks, and 2) surprisingly, techniques designed to enhance NUPA prove ineffective for finetuning pretrained models. We further explore the impact of chain-of-thought techniques on NUPA. Our work provides a more detailed and comprehensive understanding of NUPA in LLMs.

Published as a conference paper at ICLR 2025
NUMBER COOKBOOK: NUMBER UNDERSTANDING OF LANGUAGE MODELS AND How TO IMPROVE IT
Haotong Yang! Yi Hu Shijia Kang!? Zhouchen Lin 1234+
Muhan Zhang 23+
1 School of Intelligence Science and Technology, Peking University
2 Institution for Artificial Intelligence, Peking University
3 State Key Lab of General Artificial Intelligence,, Peking University
4 Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China
ha saotoogyangg@pkk. edu. en [huyi2002, kangshi j jia @stu. pku. edu, on 22 i2lii felin, muban (pku. edu. cn
ABSTRACT
Large language models (LLMs) can solve an increasing number of complex rea- soning tasks while making surprising mistakes in basic numerical understanding and processing (such as 9.11 > 9.9). The latter ability is essential for tackling complex arithmetic and mathematical problems and serves as a foundation for most reasoning tasks, but previous work paid little attention to it or only discussed several restricted tasks (like integer addition). In this paper, comprehensively investigate the numerical understanding and processing ability (NUPA) LLMs.. Firstly, we introduce benchmark covering four common numerical rep- resentations and 17 distinct numerical tasks in four major categories, resulting in 41 meaningful combinations in total. These tasks are derived from primary and secondary education curricula, encompassing nearly all everyday numerical understanding and processing scenarios, and the rules of these tasks are very simple and clear. Through the benchmark, we find that current LLMs fail frequently in many of the tasks. To study the problem, we train small models with existing and potential techniques for enhancing NUPA (such as tokenizers, positional encoding, and number formats), comprehensively evaluating their effectiveness using our testbed. We also finetune practical-scale LLMs on our proposed NUPA tasks and find that 1) naive finetuning can significantly improve NUPA on many but not all tasks, and 2) surprisingly, techniques designed to enhance NUPA prove ineffective for finetuning pretrained models. We further explore the impact chain-of-though techniques NUPA. Our work provides more detailed and comprehensive understanding of NUPA in LLMs. Our benchmark and codes are released https:://githu.. https///gthubb.oo//rrpppp Woom/GrapPPKUUU com/GraphPKKnnmbbr b
INTRODUCTION
The mathematical and reasoning abilities of large language models (LLMs) are currently quite impressive (OpenAl, 2023; Meta 2024a; OpenAl 2024a Yang et al., 2024a), capable solving problems the level of high-school student or even more difficult ones like GAOKAO (a nation- wide examination of high school students applying to universities China) (Zhang et al. 2024b), Olympiad-lewe level problems (He al. 2024), and college mathematics (Tang et al. 2024). However, upon closer examination of the models' outputs, we find that although the models demonstrate remarkable proficiency in problem-solvie solving approaches, they often struggle with basic numerical understanding and processing - like a careless student who claims, "/ know how to do it, but I didn't 't get it right." Some of these errors are quite surprising, such believing that 9.11 9.9 or making mistakes in simple addition 8/7 + 3/5. These errors are a major cause of hallucinations when dealing with math, reasoning, and data analysis tasks, as the model presents seemingly correct mroblem-solving approaches, but ultimately produces incorrect results (Huang et al.) 2024] [i et al.) 2024b] Diang et al. 2024). Therefore, investigating and improving the fundamental "numerical understanding and processing abilities" (NUPA) of models is crucial.
"Corresponding Authors

---

Published as a conference paper at ICLR 2025
Table 1: Task overview of NUPA Test. The four rows represent four numerical representations, and the 17 columns correspond to different tasks. v: 41 tasks included in our test. X: Not included, too complex. O: Not directly included but can be easily adapted from an included task. -: Not applicable. The detailed explanation for these non-included tasks is provided in Appendix A.1.5
Elementary arithmetic
Comparison
Digit Understanding
Conversion
Digit Digit Digit Get
To
To Sig.
Add Sub Multiply Truediv Floordiv Mod Max
Min
Length Count Float Scientific Fig.
Max Min Add Digit
Inleger Float
OCO
Fraction
/
Scientific
/
However, in current research, reasoning ability and NUPA are often tested together, both on classic datasets such as GSM8k (Cobbe al., (021), MATH (Hendrycks et al. 2021b), MMLU (Hendrycks) eet al. 202Ta), and in more challenging tests mentioned above. For example, problem in GSM8K is: " "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?" Solving this problem requires two aspects: on the one hand, mathematical reasoning including understanding the text, extracting relevant information, formulating mathematical equations (or finding other solution methods), solving the equations or executing an algorithm, and obtaining the result; on the other hand, it also requires understanding and processing the numbers provided in the problem or produced as intermediate results at each step. like 48/2- = 24 and 48 + 24 = 72. While these two abilities are both essential to correctly solving the problems, tests such datasets do not distinguish between them.
A more severe issue is that the numerical content is often deliberately simplified in these datasets. In various exam questions (like in the American Invitational Mathematics Examination (Li et al. 2) to focus on assessing students' understanding of mathematical concepts such as how to set up the correct equations and apply the right theorems the numbers in both the questions and answers are often specially chosen to be integers. However, this is not the case in real-world scenarios (Chen et al. 2021).
Despite the importance of NUPA, there is still a lack of accurate, detailed, and comprehensive formalization, measurement, and analysis of this fundamental capability. In this paper. we take the preliminary step towards formalizing the NUPA of LLMs. We categorize the numerical concepts and operations from primary and secondary education into four representations: flaatig point numbers (finite decimals), fractions, and scientific notation, along with four ability categories comprising 17 tasks. Pairing these representations results in 41 meaningful tasks, forming our NUPA benchmark (Table [T). These representations and tasks cover the most common scenarios involving number understanding and processing, which are typically not challenging for humans, as we read, use, or process such numbers nearly every day.
On this benchmark, we rigorously test several state-of-the-nn art LLMs containing GPT-4o (OpenAl 2024 Llama-3.1 (Meta] 2024a) and Qwen2 (Qwen Team 2024). We ask the models to directly output the answers without calling external tools. Although the latest LLMs perform well on some of the easiest tasks, their performance declines significantly tasks become slightly more complex (such as multiplication, modulus operations, or digit-based calculations), or as the representation of numbers extends beyond basic integers. See Figure 2 performance highlights pronounced mismatch between the claimed strong mathematical reasoning abilities and the poor practical, everyday numerical understanding and processing abilities of today's LLMs.
of Section 2.4 The overall unsatisfactory
To address this issue, we explore three categories of approaches to enhance the NUPA of models. The first category of techniques aims at improving models' NUPA during the pretraining stage, including alternative tokenization, specially designed positional encoding (PE) (Haviv al. 2022) Kazemnejad et al. 2023 Zhou et al. (024b), changing number formats (like zero vero-padding. index- hint (Zhou et al. (024a) and reverse representation (Lee al. 2024 /hhou et al. (024b)). We evaluate and analyze them our newly introduced benchmark, verifying their ffffctiveness/ieffffctieeesss on respective tusks/representations. which extends over previous evaluation mainly on the integer additiion/muliillcaatinn
tasks. Further, we summarize these techniques into three mechanisms: simplifying the reasoning process, aiding digit alignment, and providing regularization, and discuss the potential of these mechanisms to be applied across broader range of numerical representations.
The second category of approaches aim to improve NUPA for an already trained model. We find that while simple direct finetuning can significantly enhance NUPA performance, applying the
2

---

Published as a conference paper at ICLR 2025
aforementioned techniques (PEs, data formats and tokenizers) this stage may have adverse effects. We test various settings and finetuming configurations, but none are able to achieve performance equal to or better than the original model. Our results suggest that these modifications can significantly disrupt the models' established behavior or conflict with its pre-existing knowledge, leading to a decrease in performance.
Finally, we discuss the potential of using thain-of-thought (CoT) techniques (Wei et al. 2022) for numerical processing. Although CoT methods can break down complex problems into simpler sub- tasks and significantly increase the likelihood of obtaining correct answers, their drawbacks such as consuming large context window and requiring extended processing time- become particularly apparent in numerical tasks. We test general CoT method known as RFFT (Hu et al. 2024). and find that for more complex tasks (such as multiplication and fraction addition), chain-of-thought methods face scalability challenges, making them difficult to be applied in practical scenarios. It noteworthy that in this paper, we do not discuss tool use methods (Schick et al. 2023) Lu et al.)
2023a) for NUPA as I) want to study the self-contained NUPA of LLMs, 2) calling external tools whenever encountering numbers increases the inference latency (Xu et al. 2024). and 3) we believe NUPA without tools is necessary ability of AGI.
In summary, we propose a more comprehensive benchmark on the basic numerical understanding and processing abilities (NUPA) of LLMs, evaluate several SOTA LLMs' performance it, and further study three categories of approaches to improve NUPA: pretraining. finetuning and COT. Our results reveal that the current research is insufficient to fully address the NUPA problem, despite it being a fundamental capability for solving many more complex tasks. We hope that by introducing a systematic classification and more comprehensive evaluation of NUPA, we can bring greater attention from the community to this important but overlooked fundamental capability.
NUPA TEST: A BENCHMARK FOR NUMBER UNDERSTANDING AND PROCESSING ABILITY
In this section, will introduce our NUPA benchmark from the following four aspects: number representations, tasks, metrics, and result analysis of current LLMs. We will explain the rationale behind the inclusion (or exclusion) of specific representations and tasks in our benchmark, highlighting their distinctive features.
2.1 NUMBER REPRESENTATION
As discussed above, we believe that the educational curricula on the (Chinese) primary and secondary school levels serve as a valuable reference for determining the essential NUPAs that LLMs should master. We identify four number formats in these curricula that are both common and sufficient to cover most practical scenarios.
Integer: The most common number and the foundation of other number representations.
Floating-Point Number (Float): Floats, finite decimals, are useful subset of fractions. Calculations with floats like addition and comparison, work similarly to integers, making them common in daily life.
Fraction: We consider fractions with integer numerators and denominators. In practical situations involving distribution, fractions become unavoidable, especially when the inaccuracy introduced by converting fractions to floats is unacceptable.
Scientific Notation: Scientific notation is characterized by separating number's precise value from its order of magnitude. It is widely used in fields like physics, economics, and computer science because it efficiently handles a wide range of numbers and clearly conveys significant figures and precision. For L.Mss, mastering scientific notation can significantly enhance their ability handle practical tasks, such as interpreting financial reports or reading scientific texts.
Details of these four representations in our benchmark can be found in Appendix A.I.I There are possible representations of numbers that are not included in these four formats, like complex numbers, infinite decimal representation (repeating and non-repeating), radical expression (like v2), These representations either occur infrequently in practical conversations (c.g., complex numbers) present significant challenges for language models to process without the aid of external tools (c.g., radicals). For these reasons, have opted not to include them in our benchmark at this stage.
3

---

Published as a conference paper at ICLR 2025
2.2 TASKS FOUR ABILITY CATEGORIES
Another aspect of NUPA is defining the tasks that models need to handle. The tasks should have clear calculation rules. Furthermore, most practical numerical processing tasks should either fall within these tasks or can be easily transformed into some of them. Extracted from the primary and secondary education curricula, we propose 17 tasks in four ability categories and students who have completed the stage of education are expected to solve them. 'he complete task list is shown in Table and we provide a more detailed discussion Appendix [A.1.2 and an example for each task in Appendix Below we discuss the rationales for including some tasks in detail.
Elementary arithmetic: addition, subtraction, multiplication, and division. The most funda- mental mathematical operations. For division, we consider three types of related operators: True division, floor division and modulus.
Comparison: max and min. Understanding numbers the concept of "order".
When we care about a language model's understanding, processing (and generation) of numbers, digit a crucial concept, as numbers are not read and processed by the language model as whole, but rather
Digit understanding:
sequence of digits. We specially designed some digit-related tasks to test whether LLMs truly handle digits, including:
Get digit: Given a number and an integer i, return the i-th digit. This task is important when certain digits have special meanings in a number (such as a phone number or SSN).
Length: Return the total length (i.e., the number of digits) of a number.
Count: Count the times that particular digit occurs in an integer.
-Digit compare: Compare and return the larger (smaller) digits one by one.
-Diitt add: Perform the normal addition digit by digit but ignore any carrying. For example, (igit_add(122)) 34567) = 46802. It can test a model's understanding of digit alignment and its mastery of single-digit addition.
Conversion between representations: Converting number to two representations: to float and to scientific notation, as they are frequently used to present final results. These two tasks test whether models understand the relationship between various numerical formats. In particular, since many tasks present answers as approximate values, we designed a "significant digit" (sig. fig.) task evaluate a model's ability round long numbers fixed-length significant digits.
The combination of representations and tasks ultimately results total of 41 meaningful pairs. Without confusion, we refer to each combination
task. The tasks receive either one or two numbers as inputs and return a number as result, and the input numbers and results share the same representation for most tasks unless otherwise stated (refer to Appendix 1.1.4.. The remaining combinations are excluded due to being excessively complex, uncommon, inapplicable, or redundant with other tasks. For further details, the discussion in A..1.
The difficulty of each task depends not only on the nature of the task itself but also on the length of the numbers to be processed longer tasks involve longer inputs and outputs well more steps of internal operations. Therefore, we test on different problem lengths. For tasks that are inherently more difficult, we limit the size of the problem 1-20 digits, and for easier tasks to 1-100 digits. (For which tasks are considered difficult or easy, please refer to the Appendix A.1.6))
We generated 1, 1,000 questions for each task and each length. Unlike some previous works that set the lengths of two numbers to be the same, in our tests, the length L of question is determined by the longer of the two numbers, while the length of the shorter number follows a uniform distribution between L/2 and L. We implemented additional handling to ensure that generated problems do not result overly simple, complex, or meaningless results. Some tasks are further split into a hard and an easy version. More details about generating the benchmark are provided in Appendix A.... A.1.7
2.3 METRICS ABOUT NUPA
Generation: 425.925535321
Measuring the performance of NUPA benchmarks on these tasks is not trivial. "Exact match" accuracy is the golden standard of the performance where the answer is considered as correct when it exactly matches the groundtruth. However, smoother and more detailed metric is useful to understand the behavior and capabilities of model. Therefore, also report
Groundtruth: 31415.92653582
Exact match:
Digit match:
8 58885))=0.6
dlength:
3
Figure 1: An example of metrics.
4

---

Published as a conference paper at ICLR 2025
601-
1r0geer
Fla
Fratinn
Veager
2sage
Fast
trage
tragerr
GF743
666
Liare 708
6.5-
64-
6-4
Oweer 2-1
Liarne
Ditagee
Jitager
20ager
Icrager
22age
Fracd 2000
Doott
Figure 2: Parts of performance of state-of-the-an LLMs on NUPA benchmark D
"-ft" denotes Llama
model we finetuned on these tasks. (See Section 3.4
the "digit match" and "dlength" (difference of length) metrics, as metrics of digit accuracy and length accuracy respectively. We first split numbers into parts (e.g.. integer and decimal parts of a float, numerator and denominator of a fraction) and align the generated answer with the groundtruth digit by digit. Integer parts are aligned from the least significant digit; and the decimal parts of float are aligned from the most significant digit. For "digit match", we measure the correctness of each digit, with missing digits considered as errors, and report the overall aceuracy. For "dlength", we report the sum of absolute difference in length between each part of the prediction and the groundtruth. Figure Soaureeillustter Tillustrates these three metrics.
For each task, we divide the digits into four intervals (S, M, L, XL). For tasks with lengths 1-20, the four intervals correspond to 1-4 5-8, 9-14, 15-20 digits respectively. For tasks with lengths 1-100, they correspond to 1-10, 11-20, 21-60, 61-100 digits respectively. We average the results in each interval for each task and metric. More details of our metrics are given in Appendix A.1.8
2.4 PERFORMANCE OF CURRENT LLMS
We test commonly used LLMs our benchmark, including three Llama models: Llama-2. 7b, Llama-3.1-4 and Llama-3. 3-7b (Meta 2024a), one of the most popular open-source model families from Meta; Mixtral-8 -x (Mistra 2024), a strong MoE model; and Qwen2-7B and Qwen2-72B (Qwen Team 2024) which are also open-source models that are believed to have strong math abilities. Finally, we also test otate-of-the-art commercial models GPT-40-2024-08 GPT-40-minn-20224077-
and (OpenAl (024a). We use prompts to control models to directly output result numbers without relying on external tools or COT. The prompts used for each model and task are included in Appendix A.2 We select the results of some typical tasks in each category in Figure 2, while the complete results" and discussion on all metrics are shown in Appendix [A.3] Here, we mainly focus on the zero-shot performance while we discuss few-shot performance in Appendix/A.3.. I We have several observations regarding the results:
The best model performs well on typical tasks, but its performance declines on more specialized tasks. We find that GPT-4o, GPT-4o-mini and Qwen2 handle typical tasks, such as integer addition, float addition, integer max, and integer length, with high accuracy in the S and M ranges. This aligns with their strong performance on various mathematical datasets. However, their accuracy drops sharply when working with less common representations. like fractions and scientific notation, with average accuracy falling below 20%, even for the shortest S-range (1-4 digits). Similarly, for tasks such significant figures, modulus operations, and digit-based calculations, their performance was unsatisfactory. This highlights the current limitations of L.MSs in understanding numerical diversity and complexity. Despite their good performance on narrow set of numerical tasks, they struggle with many others, failing to produce accurate results in these areas.
Length remains a significant challenge for NUPA of LLMs. We observe a noticeable decline in accuracy for even simple tasks like integer addition as the problem length increases. For instance, GPT-4o's accuracy drops from nearly 100% in the S range and 80% in the M range to around 40% in the range and just 15% in the XL. range. In the more complex task float addition, the accuracy
An interactive performance report shown in http::/wwwgiiiigccccccccc/////an
5

---

Published as a conference paper at ICLR 2025
decreases from 90% (S) and 60% (M) to merely 15% (L) and less than 5% (XL). This trend is consistent across other models and tasks. For example, Qwen2's 's performance in the integer-length task declines from almost 100% in the S range to 50% in the M range, and falls below 5% in the L and XI. ranges.
Length impedes learning both individual digits and overall length. To understand why models struggle with longer input numbers, we examine digit match and dlength performance in F and Figure in Appendix [A.3] These metries reveal that length affects both the accuracy of individual digits (digit match) and the answer's overall length (dlength), with variations across tasks. For example, GPT-4o and Llama-3.1 display consistently low dlength in the add add-integer task, with digit match decreasing sharply as length increases, suggesting that length primarily impacts per-digit accuracy on this task. Conversely, in the max-float task, dlength increases significantly with length (about 30-60 in the XI. range), while digit match remains at 60% in the XI. range. Note that since missing digits are treated as errors, this 0.6 digit match is likely due to these missing digits. This suggests that the main challenge here lies in generating answers of the correct length, rather than individual digit accuracy. In other tasks like fraction, both length and digit accuracy issues arise, as reflected in rising dlength and declining digit match.
[A.3]
"Digit" is more challenging than expected. We were surprised to find that LLMs struggle to fully grasp "digits". For instance, in the "get digit" task, where the model is asked to return the i-th digit of a long integer, performance drops significantly as the length of the number increases. This suggests that current LLMs lack a consistent ability to simply find a digit. Note that the performance is good in the shorter S-range, which indicates that the models can at least comprehend the task instruction. In the XI.-range, GPT-4o achieves only 20% accuracy, barely above the random guessing 10% baseline (since the correct answer is always digit between 0 and 9). This fundamental limitation may explain why current LLMs struggle with numerical understanding and processing, especially as task complexity and input length increase. If a model cannot reliably identify specific digit in a given number, it casts doubt on its ability to generalize to more complex arithmetic tasks, such as addition.
We also have some interesting observations: (1) LLMs find the "max- hard" task easier than "max" with integer imputs. The difference between the tasks is that in the max task, the two numbers often differ in length, whereas in max-hard, they are always the same length and share some left-most digits, requiring more digits be compared. While max-hard intuitively seems more difficult, models actually perform better on it. This is likely because they struggle to effectively use sequence length information, as reflected in their weaker performance on the "length" tasks in the longer ranges. It suggests that models might process tasks in different ways from humans. They could have to compare two numbers digit by digit. In this situation, the "harder" subtasks are actually easier because the numbers are already aligned. (2) GPT-4o and GPT-4o-mini show nearly identical performance across most tasks, similar to the comparison between Qwen2-72B and Qwen2-7B. This suggests that once a model reaches a certain size, NUPA performance relies more on factors like architecture, training strategies, data diversity, and post- postt-raining refinements, rather than simply on increasing model size.
3 HOW DO TOKENIZERS, PES AND DATA FORMATS AFFECT NUPA?
We have observed that the NUPA Test poses significant challenges even for the most advanced LLMs. In this section, we aim to investigate the factors that can influence the NUPA of LLMs during their pretraining phase, including tokenization strategies, PEs, and different data formats. We utilize the architecture of decoder-only transformers and alter the size to create models with O.IB, 0.9B and 3B parameters. These models are trained from scratch, incorporating a wide range of techniques that could potentially impact NUPA. In this section, each model is trained on single task The details of the training process and models are included in Appendix A.4.
(a) 31415. 91415.926535897932
3.1 ONE-DIGIT TOKENIZERS ARE GOOD ENOUGH
(b) 31415. 14415.926335899933
LLMs interpret numbers as segmented tokens rather than whole numbers. With the development of language models, various tokenization strategies have emerged, including mixed tokenizers, one-di one-digit tokenizers, and k-digit tokenizers (k 2 2). as shown in Figure 3 In the BPE tokenizer used by GPT-2. the numbers are not specially treated, which resulted in irregular number cutting and is harmful to digit alignment. The cutting
(c) 31415. 011159996635897933
(d) 31415. 91415.926535897922
Figure 3: Different tokenization of long number. (a) GPT2: mixed digit tokenizer, (b) Llama-2: one-
digit tokenizer. (c) GPT-3.5, GPT-4 and Llama-3: three-digit tokenizer. (d) Aligned three-digit tokenizer.

---

Published as a conference paper at ICLR 2025
Exact Match D6
Bract Match D7
Exa ct Match DB
Exact Match De
Exact Match D10
0.25
0ocuracy
0.73
0.73
6.63-
0.15
0.42
0.12-
0.23
0.23
0.25-
0:22
3.00
5.00
0:5M
(a) int add
Ex Match D6
Exact Match D7
Exact Natch DB
Exact De
Exact Match DEO
1.00-
2.00
1.00
2.75-
2.75-
0:33-
0.50
0.50
0.50
3-c1g
2-
0.25-
0.25
2.00
2.00
0.12
1.3M
6.5M
9.7M
0.12
3.30
6.5N 9.7M
0.1
5.3M
6SN
2,79
5.3M
0.39
2.7M
0.3
3.34
5:3M
3.7M
(h) float add
Exactt Match D6
Exact Match D7
Exact Match De
Exact Match D9
Bxact Match D10
Acorssy
2.20
0.10
0.01
2.20
0.03
2.05
0.02
1.4M 2.7M
4MM
7M
2.7M
0.03
4.0M
3.4M
(c) int multiply
Figure 4: Accuracy of 0.9B models trained with 1-3 digit tokenizer on three tasks of integer addition, float addition and integer multiplication. Shadow shows the standard error. Dr means n digits. X-axis the number of seen training samples.
of numbers in modern tokenizers has become more aligned. These tokenizers greedily segment number from left to right into k-digit tokens until a remainder shorter than &c digits is left, which is then segmented into single token. Llama-2 uses a one-digit tokenizer, but all of the latest LLMs use tokenizer with ls = 3, which comes with an extended vocabulary for numbers, Additionally, Singh & Strouse (2024) discovers that just alternating the greedy direction from "left-to-right" to "right-to-left" (for integers) can improve performance of Llama-3 and GPT-4.
There is growing tendency to expand the vocabulary size as the number of parameters in LLMs rapidly increases. Recent work has shown that larger vocabulary is more suitable for larger LLMs (Tao et al. 2025) because longer tokens can encapsulate more complex and precise meanings for text tokens. However, numbers behave differently:
The long-tail phenomenon (Raunak et al. 2020), common in text tokens, is not as pronounced for the number tokens. The distribution number tokens closer to a uniform distribution.
Two smaller number tokens can always be combined into valid new one (e.g., 3 and 7 form 37), which is not true for text tokens (e.g.. "hello" and "hi" cannot form "hellohi"). ). So the number of possible number tokens grows exponentially k increases, much faster than text tokens.
The next token prediction of number tokens is harder than predicting the next text token because number prediction often involves calculation and operations, whereas word mapping tends to be more intuitive.
We trained 0.9B models on 1- to 8-digit length samples including integer addition, float addition, and integer multiplication, using aligned k-digit tokenizers where k = 1, 2,3 (d in Figure 3. Figure 4 shows the in-domain performance of these models in the first three columns and their out-of-doman (OOD) performance in the last two columns, evaluated using the exact match metric.
From the figure, the one-digit tokenizer shows the best in in-domain performance in these three tasks, while three-digit tokenizer exhibits poor performance. In out-of-domain tests, the one-digit tokenizer also exceeds the others by large margins. Tokenizers with an increasing number of digits significantly hinder sub-billion models' NUPA. We also performed experiments on models of 3 different sizes including O.IB, 0.9B, and 3B in Appendix A.4.2 and got similar results. Even as the model size increases, the performance of 2- 3- digit tokenizer improves but remains either similar or worse than that of the one-digit tokenizers. For these experiments, we also report the digit match and dlength results in Appendix AA.42 Figure 10 and [I] where one-digit tokenizer performs better both on digit learning (larger digit match) and length learning (less dlength). On the contrary. larger vocabularies significantly increase the model size requirements. In conclusion, we find no evidence to support the idea that increasing the vocabulary size improves NUPA performance.
Recently, Sathe et al.) (2024) found that the "random tokenizer"' (Kudo 2018 Provilkov et al. 2020) which splits words like "Hello world" into variable tokens such as "He/llo/ world" or "Hell/o/ world" enhances reasoning by introducing variability in generation path. We also test it in number domain and find the random tokenizers consistently outperform their standard counterparts in length generalization, but still fall short of the performance achieved by the one-digit tokenizer. See the details in Appendix A.4.2
7

---

Published as a conference paper at ICLR 2025
3.2 SPECIALIZED PES ACT AS LENGTH REGULARIZERS
Previous work has suggested that PE could be the key factor (Zhou et al. 2024b) of length gener- alization. To further investigate whether the influence is specific on a certain task, we train 100M models with different PEs: RoPE (Su et al., 2024). NoPE (Kazemnejad et al., (023) and Alibi (Press) et at. et 2022) on four tasks: integer addition. float addition, fraction multiplication (easy) and scientific notation addition respectively. Models trained on 1-8 lengths (S and M range), then test them on full range (S to XL, 1-20). ROPE, widely used in Llama and its derivatives, is the most classic relative PE. Then Alibi, another relative PE, is proposed to address RoPE's length overfitting issues. NoPE (transformers without PE, relying solely on the causal mask to encode the position information) offers surprisingly easy way to achieve length generalization. Therefore, compare these three typical PEs to evaluate the performance on NUPA.
Our results, presented in Figure 13 in Appendix A.4.3 align with conclusions from previous works. Alibi and NoPE demonstrate superior length generalization across various representations and tasks, indicating that the influence of PEs is relatively consistent across these common representations, tasks within the number domain.
Moreover, aim to characterize further the mechanism underlying these differences. Specifically, we found that RoPE leads the model to learn a length-related shortcut, while Alibi and NoPE act as form of regularization by avoiding this, thereby preventing length overfitting. For more details, please refer to the appendix A.4.3
3.3 DATA FORMATS HELP DIGIT ALIGNMENT
series of works have proposed specific data formats including reverse formatting, zero padding and index hints. Reverse formatting (Lee et al.] 2024 Shen et al., 2023) presents numbers in reverse order from the least significant digit to the most significant one to align with the models' autoregressive mechanism, simplifying the learning process for addition. Zero padding (Lee et al.) 2024; Shen et al. 2023 Zhou et al., 2024b; Cho et al.) 2024) adds leading zeros to numbers to standardize the lengths of operands, helping models align operands. Index Hints (Zhou et al.) (202aa) explicitly incorporate positional information by prefixing each digit with its corresponding position index in both input and output sequences.
While previous work mainly focuses on integer addition or multiplication, we extend the techniques to various tasks in the NUPA Test of different number domains. To compare the effects of reverse formatting and zero padding, we demonstrate in Table how the combination of reverse formatting and zero padding impacts length generalization. Reverse formatting, zero padding, and their combination all outperform vanilla formats in integer and float addition, while their performance is comparable to each other, suggesting that their functionality largely overlaps. Zero padding helps ensure proper alignment, while reverse formatting also plays a crucial role in maintaining alignment. The previously believed "helping calculation" function of reverse formatting is minor. As for index hint, we find it doesn't 't work for our models. We discuss the details of these experiment results and the reasons in Appendix A.4.4
3.4 DOES FINETUNING IMPROVE NUPA PERFORMANCE OF LLMs?
The existing techniques aimed at enhancing NUPA have rarely been applied to practical LLMs, mostly being tested on toy models and isolated tasks. This raises the question of whether it is possible to enhance the NUPA capabilities of large models through post-training finetuning. To explore this, we generate training sets (10$ samples for each digit and each task) and validation sets for our NUPA tasks, ensuring no overlap with the original test set. We then use them to finctune a pretrained model. Specifically, we finetune a Meta-L_lama33.1 1-8B model with LoRA (Hu et al. 2022) (rank 128, a=32) on mixed training set comprising all of our NUPA tasks. Remarkably, we find only 800 steps training (about 50M training samples, << epoch) leads to significant improvement, as shown in Figure 2 with the finetuned model labeled as "Llama- 8B ft". Though Llama-3. 1-BBB is not strong baseline, this finetuned version achieves much better performance. For example, in max, max-hare hard, add-float and truediv tasks, this model even surpassed matched GPT-4o. confirming hypothesis: for many NUPA tasks, the model's 's base capacity may not be the main limiting factor, but rather the lack of numerical diversity and task variety in the training data.
8

---

Published as a conference paper at ICLR 2025
However, also found that such finetuning does not provide much improvement certain tasks, such as understanding digits. Furthermore, when we tried to incorporate the various tricks, such as modifying the model's original PEs. tokenizers, model, these methods proved ineffective. When we altered the PÉ or adjusted the tokenization and representation of the model, the changes significantly disrupted the model's original behavior, causing a substantial performance drop. This suggests that enhancing a model's 's NUPA capabilities through post-training may require more revolutionary innovations beyond the current tricks. The detailed results of these attempts are presented in Table 18 in Appendix A.4.5
number formats, into an already trained
4 IS COT SUITABLE AND VALID FOR NUPA?
CoT' has been proven to be effective in enhancing the capacity of LLMs both theoretically (Feng) ict al., 2023 Yang et al. 2024b) and experimentally (Wei et al. 2022| OpenAl 2024b). Thus, we are also interested in whether CoT is the ultimate solution for improving NUPA. Due to the task and representation diversity in our benchmark, it is hard to cover all issues with a single form of COT. So we adapt special CoT form called Rule-Following CoT (Hu et al. (024) (RF-CoT), where LLMs are trained to follow a provided code or pseudo-code that outlines the procedure to solve the task. RF-CoT is capable of handling any problem with solving procedure that can be broken down into recurrences and basic unit operations. making it well-suited for our benchmark tasks. The detailed introduction with an example of RF-CoT can be found in Appendix [A.5.1
Table 3: Performance of RF COT. "-" means exceeding context window limitation (2k tokens).
Exact Match # Digit
Add Float
Multiply Fraction 3
Max Scientific
Mod Integer
38
40
8
RF CoT
1.00
1.001.000
0.93 01 0.88..03
1.00
1.00...00
1.00-.00
0.67
0.43
GPT-4o
0.78
0.66
0.49
0.53
0.20
0.00
0.37
0.46
0.36
0.01
0.00
0.00
Qwen2-72B
0.62
0,50
0,70
0,05
0.00
0.00
0.96
0.98
0.95
0.03
0.00
0.00
Llama-8B-ft
0.88 0.79 0.74
0.50
0,20.103
0.01
=000 0.98
0.97 0.98 0.08 0.05 0.05
To evaluate the performance of this CoT method, we finetuned the LLaMA 3.1 3.1 model on subset of the NUPA tasks with RF-CoT. During both training and testing, we set a context window of 2000 tokens, with any data exceeding this limit being ignored. Table 3 shows the performance on selected tasks. Accuracy and standard error for RF-CoT and finetuned Llama-3.1-8B are averaged over three runs. For GPT-4o and Qwen2, which are not finetuned, we report single-run accuracy without standard error. Within the context length limit, the rule-following finetuned LLaMA 3-B significantly outperformed GPT-4o and Qwen2-72B as well as the one finetuned without RF-CoT in most situations.
Table 2: Average inference time.
batchsize
sec sample
RF CoT
128
5.625
Direct
128
0.371
Direct 256
0.336
However, it requires a significantly longer context window and causes much slower inference speed compared directly generating the answer. As shown in Table with the 2000-token limit, CoT can only handle fraction addition involving numbers up to three digits. We provide the maximal digit length within the 2k context window limitation for each task in Appendix A...2 to show the context window limitation for complex tasks. As for inference time, Table 2 demonstrates the average inference time for generating each sample using "RE-CoT" and "direct answer" during the NUPA Test where both experiments are operated on an A800 GPU. In the table, the "direct answer" with batch size 256 uses a similar amount of CUDA memory as RF-CoT with batch size 128. The RF-CoT method is approximately 17 times slower than directly generating the answer, causing an unsustainable burden for such a basic operation that is frequently encountered in solving real-world problems, especially considering that number calculations may only account for small part of complex, real-world reasoning problem (such as analyzing financial report).
RELATED WORK
We have discussed some related work in the corresponding section. This section highlights some other studies related NUPA in language models.
Numerical understanding in natural language comprehension Earlier studies explored numeri- cal reasoning within language comprehension contexts. For example, Dua et al. (2019) introduced a reading comprehension dataset requiring discrete reasoning, such sorting and addition. Similarly,

---

Published as a conference paper at ICLR 2025
Ravichander al. (2019) proposed benchmark for evaluating quantitative understanding in textual entailment. However, these datasets blend numerical reasoning with broader language understanding tasks, making it challenging to isolate numerical propcessinn abilities.
Probing numerical understanding in LMs Several works have probed numerical comprehension in encoder models. Wallace et al. (2019) trained probing models assess numerical understanding embedded in model representations, while Johnson et al. (2020) extend this conclusion to multi- language settings. Naik et al.] (2019) used contrastive tests evaluate models' understanding of number magnitudes. Geva et al. (2020) demonstrated that finetuning on numerical reasoning data enhances the understanding. Unlike these studies, which focus on embeddings, our work emphasizes generating correct answers in autoregressive models. Recent efforts such models include Razeghi cct al. (2022), who studied few-shot learning correlations between term frequency and performance, and Zhang et al. (2024)), who identified key components in LLMs for basic arithmetic tasks. These works focus on some most classic tasks and our benchmark expands on these by incorporating diverse numerical representations, tasks, and digit ranges, offering more comprehensive analysis.
Numerical dataset in specific domains Datasets like those proposed by Spithourakis & Riedel (2018) and Lin et al. (2020) test numerical commonsense reasoning, while others focus on specific contexts, such as financial reasoning (Chen et al. [021 2022) tabular data (Akhtar et al. (2233). These works highlight numerical reasoning within specitic domains rather than general numerical processing tasks. In contrast, our benchmark targets core numerical understanding, emphasizing tasks decoupled from domain-specific constraints.
Mathematical Reasoning Datasets Despite its close relationship with NUPA, matheraticaa reasoning is a broader field involving diverse skills such as task comprehension, equation solving, tool usage, and more (Lu et al. 2023b). While correct numerical processing a critical component of mathematical reasoning, it is not the entirety of it (Stolfo et al. (023). Datasets like MathQA (Amini
al. 2019). GSM8k (Cobbe et al. 202T). MATH (Hendryeks et al., 2021b), and SVAMP (Patel et al., 2021) focus on math word problems requiring multi-step reasoning and problem-solving. Few works isolate numerical processing from mathematical reasoning. [Saxton et al. (2019) introduced a dataset for numerical tasks, such as adding floating-poins point numbers, but lacked task categorization by difficulty or length. Moreover, mixing numerical and algebraic tasks complicated analyses of pure numerical processing. Our benchmark addresses this gap, offering fine- men--grined categorization and evaluation of numerical understanding tasks.
CONCLUSION
We investigate NUPA of LLMs and introduce a comprehensive benchmark, the NUPA Test, to reveal that numerical problems remain challenging for modern LLMs. Our comprehensive test, which includes variety of numerical representations and tasks, has exposed the surprising vulnerability of LLMs in this fundamental area. To explore ways to improve NUPA, we extend and evaluate previous pretraining techniques on the NUPA benchmark. While direct finetuning on the NUPA tasks does improve the performance, utilizing those tricks specifically designed for NUPA in the finetuning tends to harm NUPA, suggesting that these methods are not casily transfcrable to practical LLMs. We also explore the potential of chain-of-then offthought techniques enhance NUPA and discuss their limitations.
7 LIMITATION
As a benchmark that specifically focuses on number understanding and processing abilities, we acknowledge that the range of tasks could still be incomplete and biased toward certain aspects. We will continue updating our benchmark, including but not limited to adding new tasks and refining existing ones to ensure appropriate difficulty. Additionally, the number of models we have tested so far is limited, and we plan to include more promising pretrained models in future evaluations.
On the other hand, although we have identified the limitations of LLMS' NUPA, the existing solutions each have their own drawbacks, We have yet to find a path that fully addresses the problem. Solving this issue may require research across multiple fields, such as enhancing the diversity of pretraining corpora, developing new techniques, or enabling more efficient reasoning paradigms that make more complex CoT approaches feasible. We hope our work can contribute to and be complemented by advancements in these areas.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper are fully reproducible. Detailed descriptions of the number formats, construction and metrics of our NUPA dataset are pro- vided in Section 2 and A.1.5 and examples for each task in A.4.4 To further facilitate reproducibility, we have included the complete dataset and the source code, enabling the generation of the entire dataset and the training and assessment of models, within the supplementary materials and the github page https//gituub.coo/iaaaaaa Unumber cookbook] Researchers wishing to generate NUPA benchmark or replicate our experiments can refer to these resources for all necessary information.
ETHICS STATEMENT
In conducting this research, we have adhered to the highest ethical standards to ensure the integrity and faimess of our work. For source code releases, we have ensured compliance with applicable legal standards. During the construction of the dataset, all data was entirely generated randomly, without including any personal identity information or other private data of individuals.
ACKNOWLEDGEMENTS
This work was supported by National Key R&D Program of China (2022ZD0160300) and the NSF China (No. 62276004).
REFERENCES
Emmanuel Abbe, Samy Bengio, Aryo Lotfi, and Kevin Rizk. Generalization on the unseen, logic reasoning and degree curriculum. Journal of Machine Learning Research, 25(331):1-58, 2024.
Mubashara Akhtar, Abhilash Shankarampeta, Vivek Gupta, Arpit Patil, Oana Cocarascu, and Elena Simperl. Exploring the numerical reasoning capabilities of language models: A comprehensive analysis on tabular data. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 1399-11405, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2223 .in.d...iinngs--oo
www.dingsemmnlp..... emnlp. 1028.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorskt Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-basee formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 2357-2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10. N0.18633/1/NN19-1255
Yang Chen, Yitao Liang, and Zhouchen Lin. Lov-ddmennsio-----iggdddeennnnn generalization and its implications for length generalization, 2024. URL http:::://aaxi../////b// 08898
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: EMNLP 2021, 3697-3711, 2021.
Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question an- swering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: EMNLP 2022, pp. 6279-6292, 2022.
Hanseul Cho, Jaeyoung Cha, Pranjal Awasthi, Srinadh Bhojanapalli, Anupam Gupta, and Chul- hee Yun. Position coupling: Leveraging task structure for improved length generalization of transformers. In First Workshop Long-Context Foundation Models@ ICMI. 2024, 2024.
11

---

Published as a conference paper at ICLR 2025
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 2368-2378, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: (0.18653/1//119-11
Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: A theoretical perspective. In Advances in Neural Information Processing Systems, volume 36, pp. 70757-70798. Curran Associates, Inc., 2023.
Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 946-958, Online, July 2020. Association for Computational Linguisties. doi: 10.18653//120020.ac--- ma.in..........
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1382-1390, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 18658//3/v 1/2022. http:22.2...2.nnooeiieaaa findings-emnll.99
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-leve bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5828-3850) 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Yi Hu, Xiaojuan Tang, Haotong Yang, and Muhan Zhang. Case-based or rule-based: how do transformers do the math? In Proceedings of the 41st International Conference on Machine Learning, pp. -9438419474, 2024.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy. challenges, and open questions. ACM Trans. Inf. Syst., November 2024. ISSN 1046-8188. doi: 1453345/773
Zhuoxuan Jiang, Haoyuan Peng, Shanshan Feng, Fan Li, and Dongsheng Li. Llms can find mathe- matical reasoning mistakes by pedagogical chain-of thain-tf-thought. In Proceedings of the Phirty-Third International Joint Conference on Artificial Intelligence, pp. 8439-34477 2024.
Devin Johnson, Denise Mak, Andrew Barker, and Lexi Loessberg-Zait Zahl. Probing for multilingual numerical understanding in transformer based language models. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 184-192, 2020.
12

---

Published as a conference paper at ICLR 2025
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 21244222229999 2023.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple sub- word candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers). pp. 66-75, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 00.18653/v1/P11 1007.
Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teach- ing arithmetic to small transformers. In International Conference on Learning Representations, 2024.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. 2024a.
Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, and Fuli Feng. Evaluating mathematical reasoning of large language models: A focus on error identification and correction. In Findings of the Association for Computational Linguistics ACL 2024, pp. 11316-11360, 2024b.
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: EMNLP 2020, pp- 6862-6868, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- ence on Learning Representations, 2019.
Pan Lu, Baolin Peng, Hao Cheng. Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 08343347-43477 2023a.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers). pp. 14605-14631, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10. 10.16655ww1122222accc1onggg
Meta. The llama herd of models, 2024a. URL UKl.:ttpss//aarrrrrrr//aa////11
Meta. Model cards and prompt formats of Ilama 3.1 2024b. 2024b http:b/tttss////o. https://www.11mm..cmmm///////
MistralAI. Mixtral of experts,
Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. Exploring numeracy in
word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3374-3380, Florence, Italy, July 2019. Association for Computational Linguistics. doi: (0.18653/v1/P19 1329. htt_hhttss///ellaaaaaoooo.ooooo.
OpenAI. Gpt-4 technical report, 2023. URL UUL.ntttet//arrii..////////22..2
OpenAl. Gpt-4o system card, 2024a. URL hti.l/ttppp//axxi..r///a////2...
OpenAI. Gpt-ol system card, 2024b. URL http:/pp//aart...r///////2222...
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of
the Association for Computational Linguisties: Human Language Technologies, pp. 2094, Online, June 2021, Association for Computational Linguistics. doi: 10.18653/v1/222 maacl-main. 168.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL https:://ppenrse net wtt.:e/oooromm2oo88800ooo.o
13

---

Published as a conference paper at ICLR 2025
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. BPE-dropout, Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1882-1892, Online, July 2020. Association for Computational Linguistics. doi: 10. 111866//112002.cllmmaaa 170.
Alibaba Group Qwen Team. Qwen2 technical report, 2024. URL http::://axiv..rrg/////o/
10671
Vikas Raunak, Siddharth Dalmia, Vivek Gupta, and Florian Metze. On long-tailed phenomena in neural machine translation. Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3088-3095, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020. 12116665/1220000..nnng--mmmmm indingg-emmlp.276
Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLLJ), pp. 349-361, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/ 11/K19-1033.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 840-854, Abu Dhabi, United Arab Emirates, December 2022.
Association for Computational Linguistics. doi: 10. 118153/v1/20222 htt.////22.innnnsemm///il//oeeeo
Ashutosh Sathe, Divyanshu Aggarwal, and Sunayana Sitaram. Improving self consistency in LLMs through probabilistic tokenization. In ICML 2024 Workshop on LLMs and Cognition, 2024.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2019.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539-68551, 2023.
Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang. Positional description matters for transformers arithmetic, 2023, URL https://aax.v..r/////////// 2311. 14737
Aaditya K Singh and DJ Strouse. Tokenization counts: the impact of tokenization on arithmetic in frontier Ilms. arXiv preprint 1499:3240..14903, 2024. URL/tttp://arxiv..om abs/
2402. 14903
Georgios Spithourakis and Sebastian Riedel. Numeracy for language models: Evaluating and improving their ability to predict numbers. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2104-211111111 2104-2115, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. N0.188533v1/PP 1/13///11P11./// 196.
Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schoelkopf, and Mrinmaya Sachan. A causal framework to quantify the robustness of mathematical reasoning with language models. In Proceedings of the st Annual Meeting of the Association, for Computational Linguistics (Volume 1: Long Papers), pp. 545 4411, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.186533
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomput, 568(C), February 2024, ISSN 09255 2312. doi: 10. me..om...eccommm222.....
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: scaling instruction tuning for mathematical reasoning. Proceedings of the 41st International Conference on Machine Learning, pp. 47885-47900, 2024.
14

---

Published as a conference paper at ICLR 2025
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. Advances in Neural Information Processing Systems, 37:114147-114179, 2025.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. Do NLP models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNIP-IJCNIPP), pp. 5307-5315, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 18653/v1DD19-1554
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
Yechen Xu, Xinhao Kong, Tingjun Chen, and Danyang Zhuo Conveyor: Efficient tool-aware Ilm serving with tool partial execution. arXiv preprint arXiv:2406. 00059, 2024. URL https
//axxiv.org/abb.. 2406. 00059
An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, 2024a. URL httpt:tt///... //ar:////xxvvi org/ h/g//abs. abs / 24099 12122
Haotong Yang, Fanxu Meng, Zhouchen Lin, and Muhan Zhang. Parrot mind: Towards explaining the complex task reasoning of pretrained large language models with template-content structure, 2024b. URL httph:ttt////ttttt org/ abs /2300/2311..mm 05452
Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu-ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye. Interpreting and improving large language models in arithmetic calculation. In Proceedings of the 41st International Conference on Machine Learning, pp. 59932-59950, 2024a.
Xiaotian Zhang, Chunyang Li, Zong. Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models gaokao benchmark, 2024b. URL httph::t/////e //arxiv/
org/abs/2305. 12474
Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Joshua M. Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? study in length generalization. In International Conference on Learning Representations, 2024a.
Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. Trans- formers can achieve length generalization but not robustly. In ICLR 2024 Workshop on Mathemati- cal and Empirical Understanding of Foundation Models, 2024b.
A
APPENDIX
A.1
NUPA TEST
A. 111 REPRESENTATIONS
We present the four representations as follows:
Integer:
we use no comma or point as a digit group separator like 1234567. The integer has only one part as itself. In this paper, we have not considered negative numbers for the time being.
Float: A float has two parts: integer and decimal. We use decimal point to split these two parts and also do not use any digit group separator. An example is 1234.567891. Trailing zeros in the decimal part usually omitted.
Fraction: A fraction has parts: numerator and denominator and a "f" to separate the numerator and denominator parts. Unless otherwise specified, all fractions mentioned in this paper are in their simplest form (that is the numerator and denominator coprime),
15

---

Published as a conference paper at ICLR 2025
but they may be greater than 1. An example is 12/7. Only in the "truediv" task between two fractions, because the "f" is also the division operator, we enclose fractions in a pair of parentheses like (12/7) (2/3) 18/7 to make it clear.
Scientific Notation: A scientific notation has two parts: significand and exponent. In our benchmark, the significand is always float larger than 1 and less than 10 and the exponent should be positive integer (and we also set an upper bound of 99). We use "e" to separate these two parts. An example 1.5e8.
A.1.2 DETAILED INTRODUCTION AND DISCUSSION ABOUT TASKS
In addition to the brief introduction of the 17 tasks in our benchmark, here we provide detailed discussion on why these tasks are significant and the specific abilities they aim to evaluate.
addition, subtraction, multiplication, and division. They are the most fundamental mathematical operations and the first branch of mathematics taught in schools. However, some operations can be complicated when different number representations are involved. For example, fraction addition is more complicated than multiplication because it needs to be reduced to a common denominator first.
Elementary arithmetic:
True division, floor division and modulus:
The division is somewhat unique because not closed for integers and floats. Here, we consider three common division-related calculations. True division: To maintain precision, we represent the division of two integers as simplified fraction. Combined with the "significant digits" task we will mention later, this can approximate the result of dividing two integers as float. Integer division and modulus: Represent approximate multiple relationships, frequently used practical applications, such as dividing individuals into batches.
max and min. Another important aspect of understanding numbers lies in the concept of "order". To truly comprehend a number, we must know how large it is and whether is greater or smaller than another one. Moreover, comparison serves as the foundation for other significant operations. For instance, when adding negative and positive numbers, determine the sign first and then subtract with their absolute values - this involves identifying which of the two numbers has a greater absolute value.
Comparison:
The concept of digit is fundamental. Unlike the "value" of a number, digit is tied to specific representation. When we care about language model's understanding. processing (and generation) of numbers, digit is a crucial concept, as numbers are not read and processed by the language model as a whole, but rather as a sequence of digits. We are curious whether LLMs truly understand the concept of digits. Therefore, specially designed some digit-related tasks, including:
Digit understanding:
Get digit: Given a number and an integer i, return the i-th digit. This task is important when certain digits have special meanings in number (such as a phone number SSN).
LLngtt: Return the total length (i.e., the number of digits) of number.
Count: Count the times that particular digit occurs in an integer.
-Digit compare: Compare and return the larger (smaller) digits one by one.
-Digit add: Perform the normal addition digit by digit but ignore any carrying. For example, digit a1d312 34567) = 46802. It can test a model's understanding of digit alignment and its mastery of single-digit addition.
Through these tasks, we can assess whether models correctly understand the concepts of digits, length, positions, and the alignment of the digits between two numbers.
Conversion between representations: we design tasks for converting a number to two representa- tions: to float and to scientific notation, as they are frequently used to present final results. These two tasks also create transformations between different representations test whether models can understand the relationship between various numerical formats. In particular, since many tasks present answers as approximate values, we designed "significant digit" (sig. fig.) task to evaluate model's ss ability to round long numbers to fixed-length significant digits.
A.1.3 EXAMPLES FOR EACH TASK
We provide each tasks with an example. To test the models, we also add some model specific system messages like "You helpful assistant to process numbers. Please directly answer the question
16

---

Published as a conference paper at ICLR 2025
after the =". The context before "=" is the question and the context after "=" is the groundtruth and is removed when testing.
Add-Integer. Add two numbers: 744 + 543 1287
Add two numbers: 93.81 9.976 103.786
Add- Add-Fraction: Add two numbers: 3/8 + 2/5 31/40
Scientcintiiie: Add two numbers: 9.92e16 + 9.731e18 = 9.8302e18
Sub-Integers Subtract two numbers: 744 543 201
Sub-Float: Subtract two numbers: 93.81 - 9.976 83.834
Sub-Fraction: Subtract two numbers: 2/5 - -3/8 1/40
Sub-Scientific: Subtract two numbers: 9.731c38 9.92c36 = 9.6318c38
Multiply-Integer: Multiply two numbers: 968 x 7744
Multiply-Float: Multiply two numbers: 8.4 x 9.555 = 80.262
Multiply-Fraction: Multiply two numbers: 8/7 5/2 20/7
Multiply-Scientific: Multiply two numbers: 9.92e16 9.731e38 9.653152e55
Truediv-Integer: Divide numbers and return the result fraction. 744 543 248/ 181
Truediv-Fraction: Divide two numbers and return the result as a fraction. (3/8) / (2/5) = 15/16
Floordiv-Integee Divide two numbers and return the result as integer. 845 152 5
Mod- Mod-Integer: Divide two numbers and return the remainder. 845 % 152 85
Max- Max-Integer: Get the maximal number: 50404 and 97871 97871
Max-Floatt Get the maximal number: 44.418 and 65.669 = 65.669
Get the maximal number: 3/5 and 3/8 = 3/5
Max-Scien Scientific: Get the maximal number: 8.15664 and 1.063e73 = 1.063e73
Digit_min eggit_max-nnteee: Compare two numbers digit by digit and return larger digit each position, treating any missing digits as O. 50194 and 14283 54294
Digit_ Digit_max-Float Compare two numbers digit by digit and return the larger digit at each position, treating any missing digits as 35.905 and 8.4 38.905
Digit add-Integer: The task add two given numbers digit by digit and return the result modulo 10 (ignoring carry), treating any missing digits 0. 50404 digit add 97871
47275
Digit add-Float: The task is to add two given numbers digit by digit and return the result modulo 10 (ignoring carry), treating any missing digits as O. 44.418 418 digit add 65.669 9.077
Get digit-Integer: Get the digit the given position (from left right, starting from 0). 50404 at position 4 =4
Get Get_ Lott_ddiiiFoallt Get the digit at the given position (from left to right, starting from 0).
44.418 at position 3
Length-Integer: The total number of digits of 50404 5
Length-Float: The total number of digits of 262.534 6
Count-Integerr Count the number of the given digit in the given number:
27422 count the occurrence time of digit 2 = 3
To float-Fraction Convert the number to float: 9/5 1.8
To_floar float-Scientffic: Convert the number to float: 8.538e2 853.8
To scientific-Integer: Convert the number to scientific notation: 50400 5.04e4
To Do_scieniific-Floa Convert the number scientific notation: 262.534 2.62534e2
17

---

Published as a conference paper at ICLR 2025
Sig.Fig. Hig.FiggIntge Integer: Convert the number to scientific notation: 50194 and keep significant figures as 3 = 5.02e4
Fig.Fittlo Convert the number to scientific notation: 65.669 and keep significant figures as 2 6.6e1
A.1.4 EXPECTED REPRESENTATION IN EACH TASK
Each task in the 41 ones receives one or two input numbers and expects one number as the result. We name the representation by the first input numbers. For simplicity, the second input number shares the same representation as the first one for most tasks. Calculations between different representations can be performed by first converting them the same representation. Two types of tasks are the exception. Tasks "length", "to float" and "to scientific" do not have the second input. The second inputs tasks "get digit", "count", "sig. fig." are always a short Integer, representing a position, length, or digit number from 0 9. To distinguish them from potentially long integers to be processed, we call the former int and the latter integer.
We summarize the second number representation and result representation in each task in Table and Table 5 where means integer, means (shorter) int, Fl means float, Fr means fraction, means scientific notation and N means such number.
Table 4: The second input number representation
Elementary arithmetic
Comparison
Digit Understanding
Conversion
Digit Digit Digit Get Max
Sig. Float Scientific Fig.
To
To
Add Sub Multiply Truediv Floordiv Mod Max
Min
Min Add Digit Length Count
Integer
1
I
1
I
OOO
N
Float
F1
F1
F
X
Fl
FI
FI
H
FI
144
Fraction
Fr
Fr
Fr
Fr
Fr
24
Scientific
S
S
Table 5: Result number representation
Elementary arithmetic
Comparison
Digit Understanding
Conversion
Digit Digit Digit Get Max Min Add Digit Length Count
To
To
Sig.
Add Sub Multiply Truediv Floordiv Mod Max
Min
Float Scientific Fig.
Integer
1
I
Fr
I
Float
F
Fl
F
X
Fl
Fl
FI
FI
FI
S
Fraction
Fr
Fr
Fr
Fr
Fr
Fr
F
Scientific
S
S
S
X
S
S
A.1.5 NON-INCLUDED TASKS
We exclude some compositions between number representations and tasks because of the following three reasons:
X too complex. We exclude the truediv between float and scientific. Division between float numbers is difficult to define accurately in our scenario. is very common to divide two floating point numbers into infinite decimal, which means that even very short decimals can still result in a very long and unpredictable result after division. And in this task we do not want discuss the of rounding the result. (This is another task of ours.) For the same reason, we also exclude division in scientific notation.
O: can be easily transferred to from an included task.
Converting fractions to scientific notation can be done by first converting to conetioonto_sciintfffic float. (Fraction-to. scientific Fraction-to_flost + Float-to_scientific). Fraction- SignificantFigure similar.
Scientific notation retains significant digits and is virtually identical to floating point numbers.
-
18

---

Published as a conference paper at ICLR 2025
count is special task where just consider number "a set of digits" count in a float, fraction and scientific notation is as the same as a integer.
not applicable.
In fraction and scientific notation, the digit concept not well-defined the tasks about digit (digit-compare, digit-add, get-digit and length) are not applicable.
Floordiv and mod is only defined on integer.
Integer and float do not need to be further converted to float. Similarly, scientific has no need converted scientific.
A.1.6 EASY/HARD SPLIT OF NUPA TASKS
We divide the tasks into easy and hard as shown in Table where the hard tasks marked as H with maximal test digit as 20 and the easy tasks marked as E with maximal test digit as 100.
Table 6: Tasks can be divided into Easy and Hard.
Elementary arithmetic
Comparison
Digit Understanding
Conversion
Digit Digit Digit
Sig-
Add Sub Multiply Truediv Floordiv Mod Max
Min
Length Count
Max
Min Add Digit
Float Scientific Fig.
Integer
H
II
H
H
H
H
E
E
E
E
Float
H
H
H
R
F
E
R
E
F.
Fraction
H
H
H
H
H
H
H
Scientific
H
II
H
E
E
E
A.1.7 PREPROCESS AND QUESTION GENERATION NUPA TASKS
We define the length of number as the number of digits in the longest part of a number. The "integer" part and "decimal" part float (as well the significand of a scientific notation), the "numerator" and "denominator" of fraction, the "exponent" of scientific notation are considered as different "parts". order generate a pair of numbers with the larger length L, we first generate L-length number and then generate l-length number where / follows uniform distribution from L/2 to L. If the operation is commutative, we swap the two numbers with probability 0.5.
After we select two random numbers, we have some preprocessing to generate the final questions:
For "Multiply", the difficulty also affected by the shorter number severely, so we split the task into two sub-tasks as "multiply-hard" and "multiply-easy". For the hard subset, we require that the shorter number must be longer than half of the longer one. For an easy subset, we require that the length of the shorter number is less than so that the complexity is (n) instead of Onn?). And because the addition of fractions also involves multiplication, we also add an add-casy for this task in the same way.
For "max" and "min" tasks, additionally provide harder version. For Integers and floats, we make sure that two compared numbers share the same length. At the same time, they should have more digits the same like 12949 and 12961 avoid models that solve the problem by only counting the length or comparing the first digit. For scientific notation, ensure 70% pairs of compared numbers with the models cannot directly get the answer without comparing the significand part. For fractions, ensure the numbers both less than one, avoiding the model can just compare them with to get more than 50% accuracy.
exponential part that
For "to_ float-Fraction". require that the fraction be converted into finite decimal, that is the denominator contains only factors and 5.
For "add/sub-Scientiffe Scientifie". we require the exponential part of each number to have difference less than 5 to make sure that the generated answer will not too long.
The pre-processing could introduce additional duplicated data, so we implement post-filtering step remove duplicates and ensure data integrity.
19

---

Published as a conference paper at ICLR 2025
A.1.8 METRICS
For digit match, we should first align the numbers. For the integers and integer parts in floats, the numerator and denominator of fractions, and the exponential part of the scientific notation, we use the right alignment. For the decimal part in floats (as well as the the significand part in scientific notation), we use the left alignment.
For dlength, we first measure the difference of each part of a number and then add the absolute values up.
Besides the average metrics in each range, we also present the following metries: well-learned digits and rnrformance-preeseneee digits to demonstrate the model's upper and lower performance limits on length. These represent the maximum number of digits that can maintain over 90% and 10% accuracy, respectively. (For digit match, the thresholds are set to 90% and 50%, and for dlength, where smaller better, the thresholds are 0.1 and 1).
We ensure that there is no duplicated sample in dataset, so for some range, the test samples could be less than 1000. We also omit digit or some digit test our testbed to make sure that unit rules can be included in training set.
A.2 PROMPTS AND OTHER DETAILS TO TEST BASELINE MODELS
For all models in our test, we first provide a "format prompt" describing the expected return format (and avoiding models generating complex CoT), and "task prompt" describing the task. We use some easy problems to ensure powerful models (gpt-4o-min) and Llama-3. 1-8B) can correctly understand the tasks and expected return format by the prompts. The expected return representation of each task is referred to in Appendix A.1.4
The format prompt based on the expected return type of the task as follows:
Integer: Directly return the answer as integer without any comma separator, like 123.
float: Directly return the answer as float without any comma separator, like 10.4.
Fraction: Directly return the answer as an **irreducible** fraction without any comma separator, like 7/13
Scientific Notation: Directly return the answer scientific notation without any comma separator, like 1. 1.23e4. The float part should be in the range [1, 10).
The task prompts are listed as follows where <a> and <b> are numbers.
Add: Add two numbers:
<b>
Sub: Subtract two numbers: <a> <b>
Multiply: Multiply two numbers: <a>
<b>> =
Truediv: Divide two numbers and return the result as fraction. <a>/<b>=
Floordiv: Divide two numbers and return the result as an integer. <a> <b>
Mod: Divide two numbers and return the remainder. <a>
<b>
Max: Get the maximal number: <a> and <b>
Min: Get the minimal number: <a> and <b>>
Digit max: Compare two numbers digit by digit and return the larger digit at each position, treating any missing digits as <a> and <b>
Digit min: Compare two numbers digit by digit and return the smaller digit at each position, treating any missing digits as <a> and <b>
Digit add: The task add two given numbers digit by digit and return the result modulo 10 (ignoring carry), treating any missing digits as O. <a> digit add <b>
Get digit: Get the digit at the given position (from left to right, starting from 0). <a> at position <b>>
Length: The total number of digits of <a>>
20

---

Published as a conference paper at ICLR 2025
Count the number of the given digit in the given number: <a> count the occurrence time of digit <b><
Count:
To_flot:: Convert the number to float: <a>
To To_scient: Convert the number to scientific notation: <a>
Sig fig: Convert the number to scientific notation:
<a> and keep sigmificant figures as <b>.
Notice that all prompts are ended with an "=" so that we can easily separate the input question and the generation of models. When the "=" is not involved in the loss calculation.
the texts in supervised finetuning (SFT), the context before
For GPT-4o and GPT-4o-mini, we also add system message as follows and use the aforementioned question as user message:
You are capable math assistant. Return your solution without any process in the format: The answer is (YOUR ANSWERJ. The final answer must strictly match the format <regex>.
where the <regex> is a regular expression based on the expected return format:
Integer: r"Id+"
Float: r"d++Ad+*
Fraction: r'dd+Ad+"
Scientific Notation: r"dd+\d++edd+"
We use the models expect GPT from huggingface and use the default tokenizer, model and generation configuration provided by the models. We test GPT-40 and GPT-4o-mini by the OpenAl API, where GPT-4o means gpt-4o-208 2024-0806 and GPT-4o-min means 72B and 1lama-3.1-70R without quantization and ensure this quantization does not affect generation quality.
For Qwen2- additionally 4-bit quantization but we also test several samples
11--o-mmi-2024--000
We retrieve the first match of the corresponding regular expression after the "=" as the answer. If there is no retrieve, we use an empty answer to calculate the metrics, where exact match and digit match is both zero and the dlength the total length of the groundtruth number.
A.3 FULL TEST RESULTS OF LLMS
We show the full NUPA Test results in Figures 5 (exact match), G (digit match), 7 (dlength) and Table 7 (well-learned digits and performance- preserving digits for each metrics).
With the detailed metrics, we can more clearly understand the behavior of some models on some tasks. For example, we find that the "exact match" and "digit match" of some models like Qwen-2 and GPT-4o on the "integer-max" task are similar, suggesting that when the models know which one is correct, they can always copy the answer from question correctly. So the wrong answer comes from incorrect comparison. Another example is the Llama-2 performance on max-hard. Because the length of two input numbers and the groundtruth answer in the max-hard task all the same, most models show less dlength on this task suggesting they know that "the answer should have the same length of inputs", but we find Llama-2 shows dlength approximately equal the average length in the range, suggesting that Llama-2 cannot generate a valid answer on this task. These are just a few examples illustrate how more detailed metrics help gain deeper understanding of model behavior. There are many possible conclusions, but there are too many to list here.
A.3.1 FEW-SHOT LEARNING
To ensure the output format of models is as precise possible, employ 5-shot learning. For each task, we select one sample from different lengths respectively and test the few-shot performance. Table summmarisssss the exact match score performance across three selected tasks and Figure shows more tasks. Notably, providing an explicit output format results in general performance improvements across tasks and input lengths. In most tasks, the models can usually produce accurately formatted
21

---

Published as a conference paper at ICLR 2025
outputs even in the zero-shot setting, with limited additional benefit observed from few-shot examples, while the few-shot examples can indeed provide some performance improvement.
We find that the conclusions mentioned in main paper have still holds. For example, the performance also significantly decreases as the length increases or the tasks and representations become unfamiliar (like Add-Fraction Add-Sciess Aed-nciintifii floordiv). And the performance of digit-related tasks still unsatisfying.
Table 10: Few-shot performance on selected tasks
Model
Add Int
Max Float
Floordiv Int
S
M
I.
XI.
X
M
I.
XI.
X
M
I.
XI.
2aama-2-7b-hf-5--tte
0.61
0.12
0.00
0.00
0.68
0.55
0.49
0.43
0.04
0.01
0.01
0.00
Llama-2-7b-hf
0.74
0.11
0.00
0.00
0.44
0.47
0.28
0.15
0.04
0.01
0.00
0.00
0.94
0.41
0.10
0.01
0.88
0.81
0.63
0.54
0.23
0.02
0.01
0.00
Llama-3.
0.95
0.38
0.06
0.02
0.70
0.57
0.41
0.36
0.19
0.01
0.01
0.01
0.83
0.82
0.37
0.04
1.00
0.98
0.81
0.64
0.28
0.08
0.03
0.01
Qwen2-7B
0.93
0.70
0.23
0.03
0.68
0.72
0.55
0.43
0.22
0.05
0.01
0.02
Exxt March
Lieea Sraend
Drear 128 cnd
00-
1ager
Irage
Mege
Hea
aage
Hedger
Praden
Mengs
sapger
Foot
Figure 8: Parts of performance of state-of-the LLMs on NUPA benchmark with 5-shot examples.
A.4 TOKENIZER, PE AND DATA FORMATS
A.4.1 EXPERIMENT DETAILS
We train several models test the effectiveness of tokenizers, PEs and data formats. Unless otherwise mentioned, our model architecture uses the Llama-3. architecture (Decoder-only Transformers with causal masking, autoregressive generation, and RoPE as the default PEs). We modify the layer numbers, hidden size and the number of heads change the parameter size of models. See Table TT We keep all hyperparameters, except model size, consistent with the original Llama setup in the implementation from Huggingface, We use the default sampling generation strategy with default hyperparameters, where the temperature is set as 0.6 and top_p is 0.9, About the meaning these settings please refer to Llama technique report Metaa 2024a) and model cards Meta 2024b).
To train these models, we use the Adam W optimizer (Loshchilov & Hutter 2019) with a learning rate of 5e-5, weight decay of 0.01, and batch sizes of 256, 64, and 32 for O.IB 0.9B, and 3B models, respectively. Other optimizer settings follow the default values in the Transformers library. We sample 1e7 samples for each length (where feasible) and concatenate them into single training
22

---

Published as a conference paper at ICLR 2025
set. Models trained for epoch using cosine decay learning rate scheduler, and the best checkpoint on validation data is reported.
cluster equipped with Nvidia A800 GPUs (80GB memory). Training a 100M model takes 5-8 hours, a 1B model approximately day, and a 3B model around days single A800 GPU. Finetuning pretrained model typically takes about day.
Our experiments were conducted
Table 11:
Detailed model settings for experiments.
parameter size
num hidden layers
hidden size
intermediate size
num attention heads
num KV heads
100M
1024
3584
0.9B
2048
7168
3.0B
3072
10752
Exact Match
Match
Match
Exact Match
Match D10
1.0
2.8
2.5
05
3.6
0.6-
1-digt tokon por
0.2-
2-digt tokon cor
1.2
bidigt telon
5.0
0.34
0.3M
183N
27.8M
3.0
0.1N
93M
15.M
27.0M
0.191
9.3M
125M
27.59
0.155
2.2M
115M
27.59
0.19
2.5H
18.54
27 59
O. IB model int add
Exact Match
Exact Match
Match D10
2.24
1.0
0.8
0.200
5.8
0.7-
L:135-
Accuracy
0.6
0.6
65-
0.150-
0.125
0.4.
0.100
3.dggt Laken ser
2-dgit bakon por
1.2
3.0
0.0N
2,6%
2.3 18M 2.4M
0.ON
2.0M
1:3M 1:M
2
0.39 0.6N
2224
1
2.4M
6.30
0.6N
1.2M
1.89
2:4M
0.00
0.5M
1.24
1.5%
2
(b) 0.9B model int add
Exact Match
Exact Match
Exact Match D45
Match D50
1.0
5.50
Accuracy
6.20-
0.025
0.
6.15-
0.020-
L.kge Cakon
2.30-
0.2
2-digtt baken per
12
diiiit tikon
0.3N 1AN 23H 4.6N 2:3M
6.N 7:4M 27H 43M 5:2M
6.1 2:4H 2.M 4.M
32M
6.5
2AM
2.7M
4.0M
5.24
0.39 1
27TH
4.0 5.50
(c) 3B model int add
Figure 9: Accuracy of models of O.IB, 0.9B and 3B parameters trained with 1-3 digit tokenizer on the task of integer addition. X-axis is the number of seen training samples.
A.4.2 TOKENIZATION
We experiment on models of 3 different size, including OBI 0.9B and 3B. For the O.IB and 0.9B models, we train them on integer addition of 1-8 digits; for the 3B model, we train it on the same task of 1-40 digits.
Figure 9 illlstraaes the in-domain performance of these three models in the first three columns and their out-of -f-ooai--minn (OOD) performance in the last two columns. Here we use the exact match metric. In our experiments of the 0.1B and 0.9B models, the one-digit and the two-digit tokenizer demonstrate comparable performance in the in-domain test, while the one-digit tokenizer exceeds the others to large extent in length generalization. in both in-domain and out-of-domain evaluations. Tokenizers with an increasing number of digits significantly hinder subbillion models' NUPA. In the experiments of the 3B model, the two-digit tokenizer matches the one-digit tokenizer in both in-domain and OOD performance. In addition, the three-digit tokenizer shows the potential in length generalization for the first time, yet its performance remains inferior to that of the smaller tokenizers. This indicates that scaling up the model size
contrast, the three-digit tokenizer exhibits poor performance
23

---

Published as a conference paper at ICLR 2025
indeed alleviate the challenges in developing NUPA caused by larger tokenizers. Nevertheless, larger tokenizers do not present any distinct benefits either in-domain or out-of-domain generalization in both small and large models.
We report the results according to different metrics from Figure inncluding digit match and dlength in Figure and Figure
Digit Match D6
Digit Match DD
Dight Match DB
Digit Match D9
Digit Match D10
1.00
1.03
0.22
Acouracy
2.80
0.00
0.00
0.50-
0.50-
0:80
0.02
0.79
0.65
0.03
0.00-
0.42
0.60
0.5
6.02
2.20
newember
0:20
0.12
0.25
0.5M SMM
2
2:H
0.5 SAM
3,79 2:0H
0.5 LiN
(a) 0.9B model int add
Digh Match D6
Digit Match
Digit Match DB
Digit Match D9
Digit Match D10
3.00-
Accuray
0.50
5.70-
5.00-
2.00
9.30-
0.50
0.50
0.50
0.93
0.63
2.40
9.50-
0.43
2-dhht tchenizer
2:494 tokemzerr
2.30
2.40-
5.20-
0.40
0.25
0.1M
3.3H
6.N
9.7M
3.3H
6.N
9.7M
0.3M
3.3M
6SN
5.MM
3.3M
6.5M
6.7M
0.3N
3.3M
6.5H
6.7H
(b) 0.9B model float add
Dight Match
Digit Match
Digit Match
Digit
Digit Match D10
0.55
9.50-
0.35
0.33
2.40
3.40-
0.55
2.30
2:dg4 takamezr
1-412 rokerizer
3.70-
0.15
0.1
1AM
2:M
1 2:7M
adM
6.1M
3,MM
2.M
60M
6.1M
3.4H
2.7M
4.0M
0.3M 3.4H 2.7M 4.6M
(c) 0.9B model int multiply
Figure 10: Accuracy of 0.9B models trained with 1-3 digit tokenizer on three tasks of integer addition, float addition and integer multiplication according digit match. X-axis is the number of seen training samples.
Random tokenizer Introduced as "sub-word regularization" by Kudo (2018); Provilkov et al. (2020), the random tokenizer splits words like "Hello world" into variable tokens such as "He/llo/ world" or "Hell/o/ world". Though not widely used in LLMs, Sathe et al. (2024) found that it enhances reasoning by introducing variability in generation path. Inspired by this, apply this the numbers, segmenting numbers into tokens with lengths randomly chosen between and a predefined maximum, instead of using greedy left-to-right segmentation.
24

---

Published as a conference paper at ICLR 2025
Dlength D6
Diengih D?
Diength DB
Diength D9
Diength D10
0.30
0.23
0.30-
0.35-
2:75
2,05
1.75-
0.23
2.23
1.23
3.00
3.00
2:
0.02
0.10
0.0--
0.55
0.15
6:75
6.52
1/00
0.22
1.2--
0.22
6.
3.5M
1. 7M
3:7M
SM
(a) 0.9B model int add
Diength D6
Diength DV
Diength DB
Diength D9
Diength D20
2.00
2.00
0:50
1.73
2.50-
5.40-
0.40
1.23
1.53
<. takenozer
9:20-
0.20
0.75-
2.20
0.10
0.50
0.1M
9.7M
0.
9.7M
8.7M
3.5M
3.3M
(b) 0.9B model float add
Diength D6
Diengthh D7
Diength D8
Diength DD
Diength D10
3.25-
3.70-
3.36-
5.30-
0.40-
0.80
0.80
L9 12length
0.20
9.10-
2:15
3.05
2:0--
0.23
0.22
2:0gh Ictenizer
0.1M
1.1M
2.7M
0.1M
1.1M 2.7M
4.6M
6.3M
1,4M
2.7M
4.30
0:3N
3.4N
2,7N
4.001
0.00
0.N
3.4N
2,7N
4.5N
(c) 0.9B model int multiply
Figure 11: Accuracy of 0.9B models trained with 1-3 digit tokenizer on three tasks of integer addition. float addition and integer multiplication according to dlength. Here we report log2(dlength + 1).X- axis is the number of seen training samples.
Exact Match D6
Exact Match D7
Exact Match De
Exact Match D9
Exact Match D10
1.00
0.00
0.20
6.40
Arturary
0.60
0.60
k.gg tokerkeer
0.40
0440
0.39
0.00
3.00" ranismm tikenzer
tstan
0.20
0.00
0.00
0.10
0.20
9.00
0.00
0.07
0.3M
6.5M 0dw 1.30 1.w 2:5H
9. 0.3M 5:5H
1. Z.M
7.1 0.3M 639
3,3M 1.7M 2:5M
0.3M 0.5M 0
1.5 L.me 2:3H
6. 0.39 0.5M 1.M 1. 20
Figure 12: Accuracy of 0.9B models trained with 1- to 3- digit tokenizers and 2- to 3- digit random tokenizers integer addition. Shadow shows the standard error. Dn means n digits. X-axis is the number seen training samples.
Figure 12 shows the performance of 1- to 3-digit tokenizers alongside 2- to 3-digit random tokenizers, where n-digit random tokenizer means the one with maximal length n. In terms of in-domain generalization, the three-digit random tokenizer outperforms the three-digit standard tokenizer, while the two-digit random tokenizer shows slight decline compared to its standard counterpart. We believe this is because the 0.9B model is capable of learning the two-digit tokenizer well, and the added perturbation from random tokenization acts as form of regularization, introducing noise that slightly affects performance. The random tokenizers consistently outperform their standard counterparts in OOD generalization, indicating the regularization benefits in that aspect. In the case of the three-digtt digit tokenizer, which is more challenging for 0.9B model to learn, random tokenization generates smaller tokens, making the learning process easier and leading to improved in-domain performance. However, they still fall short of the performance achieved by the one-digit tokenizer.
25

---

Published as a conference paper at ICLR 2025
A.4.3 PEs
Exactt Match
Digit Match
Diength
1.00
2.5
0.80
noрe allbl
lccuracy
0.80
Talength
2.0
0.40
0.40
10-
0.20
0.20
7
11
13
15
17
5
11
13
15
17
7
9
11
13
15
17
Length
Length
Length
Exact Match
Digit Match
Diength
rope
0.80
0.80
log ( Eelecaaie
0.60
allbi
0.60
0.20
0.20
0.00
7
11
13
15
17
11
13
15
13 Length
15
17
Length
Length
Exact Match
Digit Match
Diength
0.80
rope nope
0.80
0.60
Acturacy
0.60
log ldlengthe
0.40
0.20
0.20
0.00
0.00
13 Length
Length
Length
Exact Match
Digit Match
Dlength
1.00
1.00
rope
2.0
Acturaey
0.80
0.60
aili
nope
0.80
8 Ielengthe
1.5
0.40
0.40
1.0
0.20
0.5
0.
15
0.00
13 Length
17
0.0
11
13 Length
15
17
Length
Figure 13: Exact match, digit match and dlength of 100M models trained with various PE, including RoPE, NoPE and Alibi. From top to bottom, the tasks are integer addition, float addition, fraction multiplication and scientific notation.
We show exact match, digit match and dlength of 100M models trained with various PE, including RoPE, NoPE and Alibi in Figure 13 We find NoPE and Alibi achieve better length generalization than RoPE, which is consistent with previous work like Zhou et al. (024b)).
To explain the mechanism of PEs, is necessary to describe what the "generalization" is about. In most tasks, there is an intrinsic "length-agnoste numbers. For example, the addition rules: "align numbers by their least significant digits, add them digit by digit and carry over if the sum exceeds 9" is length-agnostic because it applies universally, regardless of the input length. However, during training on data with restricted length range (like to 8), models may also learn length-related rules that fit the training data, such as combining normal addition rules with constraints like "the output length must range from to 8". Because these two rules are indistinguishable, prior knowledge should be added into the model to help the model learn the
calculating rule, independent of the length of input
inductive bias rules expected in most practical settings (Abbe et al.
2024
Chen al. 2024
26

---

Published as a conference paper at ICLR 2025
Table 12: RoPE performance with standard error from three repeated experiments. Dn means n digits where D8 is the longest in-domain length and D9 is the shortest out-of-domain length.
Exact Match
Digit Match
Dlength
D8
D9
D8
D9
D8
D9
Int-add
1.00++.00 0.00=
1.00=0.00 0.45=+0.02
0,00++0.00
1.07:0.02
Float-add 1.00+0.00 0,00=0.00
1.00=0.00 0.59=0002 0.00 1.06#0.01
Frac-mul
0.70+ 0.01
0.85 0.22:t0.02
0.18:
1.45
Sci-add
1 1.00+0.00 0.23+20.88 1.00=0.00 0.92++001
0.00+:0.00
0.66+0.11
According to our experiments, we find that (1) RoPE encourages the model to rely on the length of the input. The first evidence is that RoPE causes the model's predictive performance to plummet dramatically just beyond the training boundary. We report the RoPE's performance the boundary of training length in Table 12 where D8 (digit 8) is the longest length in the training range while D9 (digit 9) the shortest lenghh out of the training range. In "int-add" task, the exact match drops from nearly 100% to 0% when moving from 8 to 9 digits, while "dlength" rises from 0 to 1.07 (Table[2). This indicates that the model has a significant probability of generating shorter results, avoiding the generation of more than 8-digit answers. At the same time, RoPE not only constrains the model's ' output length but also affects the digit pairing. The performance of 100% for inputs of 8 digits indicates that the model performs calculations for each position unless it can successfully align the corresponding digits. However, when the model encounters 9-digit inputs, digit match drops significantly to 50%, suggesting a considerable probability of failing to align the digits. Similar results on the other three tasks suggest that it is agnosticc behavior. The only exception is the digit match of scientific notation addition. We discuss the results later.
(2) On the other hand, length learning provided by RoPE appears to be a shortcut. In cases where the model is extremely small or has been trained very little, we see the advantages of this "shortcut". In Table 13 we train a 2 2-layer transformer (1.3M parameters) on integer addition using three different PEs on to 8- digit integer addition or the 0. IB model with only 1M samples, we find RoPE shows the best in-domain performance. Experiments on the other three tasks are shown in Table 14 and Table [5] where the RoPE always surpasses others.
Table 13: 8-digit digit-match accuracy with small model small dataset.
1.3M Model
IM Samples
RoPE
0.091
0.97
NoPE Alibi
0.061
0.78
0.056
0.23
As a possible explanation about why Alibi and NoPE achieve better length generalization, our experiments suggest that for length generalization in number tasks, the required inductive bias is to interpret the input as a sequence of digits while deliberately ignoring its overall length. ROPE, as a positional encoding that enables the model to quickly learn position-related information, may lead the model to adopt a length-dependent shortcut (Table 13). causing it to favor length-related rules. In contrast, both Alibi and NoPE diminish this reliance on position and length, encouraging the model to treat each unit's s operation as step-by-step process, thereby achieving better length generalization.
Discuss about scientific addition The results in Table 12 reveal clear trend where performance drops from 8-digit to 9-digit numbers, with one exception: the digit match score in the scientific notation addition task, which remains relatively high at 0.93 even for 9-digit numbers. We believe it is mainly because of the alignment mechanism between two scientific notations which differs from other representations. In other representations, numbers are aligned by position - integers from the most-left digit and the floats by the decimal point. However, in scientific notation, alignment depends on the difference in exponent values, which reduces RoPE's reliance position and mitigates length overfitting. Despite this, the effect of RoPE limiting output length remains apparent, as evidenced by the significant increase in the dlength score.
27

---

Published as a conference paper at ICLR 2025
A.4.4 DATA FORMATS
Table 16: Exact match of O.IB models trained on integer addition and float addition respectively with various compositions of reverse formatting and zero padding.
Integer Addition
Floot Addition
10V
+pad
Tev
pad
nOV total
Des total
pad
rev each
rev cach
pad
rov doc
ToV doc -pdd
rev int
rov int
+ -aad
d9 0. 10 0.98 1.00.001 0.11
0:24
0.12 0.24 0o 0.12 0,24 1.00..00 1.00 0.994.01 1.00
410 0.69+.11 0.91 05 0.16=.11 in 0.50= 34 0.077+003 0.21=001 0.10=.02 02 0.23 no 0.07=.022 0.17+.04 0.97 0.874.36 0.17+.004 0.76+.19
We provide the experiments in Table 16 and the evaluation curves of compositions of reverse formatting, zero padding and index hints in Figure [1]] Figure [15] Figure [6] Figure [7 and Figure [88] We experiment on O.IB models trained on 1- to 8- digit training samples. Here we all use the exact
match metric.
Previous work (Zhou et al. 2024b) believes that reverse formatting can help the calculation of each digit by aligning the calculation order to the right-to-left order that humans are accustomed and solve the carrying problem. That is, from left-to-right, we cannot determine the result of the current digit unless the next digit results and whether there a carrying have been known. However, a more detailed analysis can explain why the order is not as important as previously believed:
Regarding addition, the cases where reverse formatting can make a difference through the effects of assisting carry-over calculations are quite rare. Most of the time, knowing the result of the next digit allows us to determine the answer for the current digit. When the next digit addition is not less than 10 (without considering further carrying from the following digit), there must be carrying from that digit into the current one, no matter what the result of the later digits is. And when the next digit addition is not more than 8, there will never be carrying. The only exception is the next digit addition is 9. In this situation, we must refer to the next two digits to determine the current digit results. Therefore, we point out that, although in the worst-case scenario, performing non-reversed addition requires (n)) (le-gegt looking forward for each digit (44445 + 55556 = 100001), and reversing could solve this problem, such cases are extremely rare. In most instances, the task can be accomplished with very limited local view.
About the experiments of index hint, we show in Table 17 Our conclusion on index hints seems to contradict the findings of Zhou et al. (2024b), where models with index hints appeared to achieve better results. We believe this discrepancy may be related to model size and digit range. In their work, a much smaller model (only 25M parameters) was used, but the training range covered 1-40 digits. This reduced the model's s ability to learn the patterns independently without external hints, resulting in different learning outcome where the model began to rely on index hints. As piece of evidence, when Zhou et al. (2024b) train 1-10 digits, the performance without index hint is OK. (But they did not provide the complete results of 1-10 digit training in their work.)) The efffcttivnness of index hints may involve complex interactions, which could be an interesting direction for future research.
A.4.5 NUPA FINETUNING WITH PE, TOKENIZER AND REPRESENTATION MODIFICATION
We show parts of results of our attempt to finetune Llama-3.1 -31-8B model with PE, tokenizer and data format modification in Table 18 All the checkpoint we select by the lowest valid loss. No one can outperform the naive finetuning or the original Llama.
A.5 RULE-FOLLOWING CHAIN-OF-THOUGHT
A.5.1 DATA FORMAT RULE-ROLLOWING CoT
Rule-following CoT consists of a rule prompt and response to follow the rule. To generate the CoT samples for task, first the computational rules are converted to Python code. We utilize code LLM to write the code in this experiment. Then, a data generation script which can generate the rule-following intermediate steps for any given input is manually written. With the script, given a pair of numbers and a task, both the rules (code prompt) and the response can be generated automatically. In the finetuning stage, both the prompt and the response are provided to the model but only the loss on the response part is calculated. During inference, the code prompt is still provided and the models expected to generate the ule-followwig steps as well as the final answer.
28

---

Published as a conference paper at ICLR 2025
Below is an example of a complete Rule-Following CoT data format, where the model is required to solve integer addition task with a right-to-leff recursion and three digit addition as the unit task.
Prompt:
Follow the given rule to solve the question. Rule:
def add (numl, num2)
result
carry
# Main Loop
while numl or num2:
digitl = int (numl [-3:]] if numl else 0
digit2 int (num2 [-3:]] if num2 else 0
total = digitl digit2 + carry
result str (tottal811000) + result
carry total//1000
numl numl[:-3] if numl else numl num2 num2[:-3] if num2 else num2 if carry:
result str (carry) + result result Aesult.lstri((1 return result
or '0'
Q: Return the answer an integer without any delimiter, like 123. Add two numbers: 28 35
Response:
numl = "28"
num2 "35"
result = ""
carry
while num1 or num2:
check stop criterion
enter the loop
digit1 int (num1 [-3:]) if numl else
digit2 = int (num2 [-3:]] if num2 else
numl = "28"
num2 "35"
digitl =28
digit2 =5
total digitl + digit2 carry
carry= =0
total 28 +5 63
result str (total81000) result
carry = gotal//1000
29

---

Published as a conference paper at ICLR 2025
total % 1000 63%1000 63 result = "063" + "" = "063"
carry 0//1000 0
numl numl [: :3] num2 num2 [ 33
numl "M
num2
while numl or mum2
check the stop criterion numl = and num2
end the loop
if carry: result str (carry) result
carry =0 pass
result 1strippt) or "0"
result = "63"
return result
return "63" So the answer is 63.
A.5.2 MAXIMUM DIGIT LENGTHS WITHIN CONTEXT WINDOW
The selective tasks used to train the RFFT are shown in Table 19 and we also report the maximal length within 2k tokens context windows limitation.
30

---

Published as a conference paper at ICLR 2025
bead Paitch
OPfing
Offmerend
Queval
27geer
Fadin
Irogge
radinn
(a)
Exact Match
Office
Camm--3--68-
(b)
offine
Horrall 0075
0.--
La7m--2-76-41
Mulayy Balr
Hullury Bayy
PM
A64
Exact
04.
67
44-
Owerd
0.--
0.2-
0.1-
1onch
(d)
Uomo 60
0.5
Oieech
10
To Sdbeet
Sount
53
(c)
Figure Exact match of models tested on NUPA Test.
31

---

Published as a conference paper at ICLR 2025
Digit
Gifting
Cynria F21
2-76--44
0.1-
AM
AM
(a)
Digit Match
0.--
0.5-
094
Lanss- 2-03--
Mae
Her
ScansPoresstor
Nor Herd
Pac Red
Musah Hend
Partol Hand
(b)
Dight March
cffice
Larra
Hoore 0Crl
0.5-
2g0
Idlaee
(c)
Digit Match
Liamao L-08-
teliger
Deagee
(d)
(c)
Figure 6: Digit match of models tested on NUPA Test.
32

---

Published as a conference paper at ICLR 2025
GFT--oom
Ho (Peggg
50b
(a)
Dlength
Mullay maa
(b)
Diangth
2-00-
-3
Bow Roe
Hulloyy Dayy
Dic Man
068 me
Diengtth
cont do
Horre 10/70
Logge
saager
Inager
(d)
Figure 7: Dlength of models tested on NUPA Test. Note that we use log, (olengtt + 1) as the ylabel the figure.
33

---

Published as a conference paper at ICLR 2025
Table 7: Well-leamed digits/ performaacce-preservvig digits of models tested on NUPA Test according
to exact match.
Add Int
Add
Add Frac
Add Easy Frac
Add
Sub
Sub Float
Sub Frac
Float
GPT-4o-mini
5/20
4/11
0/2
6/20
5/15
0/1
GPT-4o
5/20
4/11
0/1
0/1
0/0
6/20
4/15
0/1
Qwen2-72B
0/15
0/1
0/1
0/
6/20
0/15
0/1
Qwen2-7B
4/ 14
0/15
0/1
070
0/7
4/14
0/13
0/1
Llana: 3.1-88--0
1212
5/17
0/1
0/1
0/4
4/12
4/17
0/2
1lama-3.1-700
15
0/11
0/1
0/1
0/0
6/11
4/11
0/1
Llama-3.1-88
4/9
3/11
0/1
0/0
0/0
3/10
3/10
0/1
Llama-2-7b-hr 0/6
0/7
0/0
0/0
0/0
0/6
0/5
0/1
Sub
Hard Int
Hard Float
Hard
Sci
Int
Float
Frac
Sci
Sci
GPT-4o-mini
5/20
4/11
0/1
0/2
0/0
6/20
5/15
0/1
GPT-44
6/20
4/15
Qwen2-72B
6/20
1/15
0/1
D/1
0/11
6/200
0/15
0/1
Qwen2-7B
0/0
0/13
0/1
Jlama-3.1-BB-1
4/13 12
5/17
0/1
0/1
0/4
4/12
4/17
0/2
1amaa-3.17000
4/15
0/1
0/0
6/11
4/11
0/1
Llama-3.1-8B
4/9
3/11
0/0
3/10
3/10
0/1
Mixural-xxBB
5/10
4/11
0/1
0/1
0/6
4/15
3/11
0/1
Llama
0/6
0/7
0/6
0/5
Multiply Hard Multiply Hard Multiply Hard Multiply Hard Multiply Easy Multiply Easy Multiply Easy Multiply Easy
Int
Float
Frac
Frac
Sci
GPT-4o-mini
20
4/11
0/1
0/2
0/0
6/
5/15
0/1
GPT-4o
5/20
4/11
0/1
0/1
0/0
6/20
4/15
0/1
Qwen2-72B
6/20
0/15
6/20
0/
0/1
Qwen2:
4 14
0/15
0/1
0/0
0/7
4/
0/13
0/1
1aama-3.1-90
4/17
1a1ma-3.17700
45
0/11
0/1
0/1
0/0
6/11
4/11
0/1
Llama-3.1-80
4.19
3/11
0/1
0/0
0/0
3/10
3/10
0/1
Mixtral-8x7B
1/
4/
0/1
0/1
4/15
3/11
0/1
Llama-2-7b-hf
0/6
0/7
0/0
0/0
0/0
0/6
0/5
0/1
Digit
Digit Add
Get Digit
Get Digit Float
Length Ini
Length
Int
Float
Int
Float
Int
Float
GPT-4o-mini
5/20
4/11
0/1
0/2
0/0
6/20
5/
0/1
GPT44
5 20
4/11
0/1
0/1
0//
6/20
4/15
0/1
Qwen2-72B
6/20
0/15
0/1
0/1
0/11
6/20
0/15
0/1
Qwen2 7B
4/
0/13
Llama-3 3.18-t
4 12
5/17
0/1
0/1
0/4
4/12
4/17
0/2
Llama-3,
4/15
0/1
6/11
4/11
0/1
Llama-3.1-8B
4/9
3/11
0/1
070
0/0
3/10
3/10
0/1
Mixtral 8B7B
4/11
0/1
0/1
0/6
4/15
3/11
0/1
1lama-2-7h-hf 0/6
0/7
0/0
0/0
0/0
0/6
0/.5
0/1
Truediv Int
Truediv Frac
Floordiv Int
Mod
Mod Easy
To Float
To Float
To Scient Int
Frac
Sci
GPT-4o-mini
5/20
4/11
0/1
0/2
0/0
6/20
5/
0/1
GPT-4o
5/20
4/11
4/
0/1
Qwen2-72B
6/ 20
0/
0/1
0/11
6/
0/15
0/1
Qwen2-7B
4
0/15
0/1
0/0
0/7
4/
0/13
0/1
Llama-3.1-8B-
5/17
4/
4/17
0/2
LJama 3. 3.1 70B
4 15
0/11
0/1
0/1
0/0
6/11
4/11
0/1
Llama-3.1-8B
4/9
0/0
3/10
3/10
Mixtral-xxBB
5/10
4/
0/1
0/1
0/6
4/15
3/11
0/1
Llama-2-7b-hf
0/6
0/7
0/0
0/0
0/0
0/6
0/5
0/1
To Scient
Count
Sig
Float
Int
Int
GPP
GPT-4o
5/20
4/11
0/1
Qwen2-72B 72B
6/20
0/15
0/1
Qwen2-7B
4
0/
0/1
Llama-3. 1--B-f 1-8B-ft
4/
5/17
0/1
Llama-3. 3.170B
Llama 3.1 8B
4/9
3/11
0/1
Mixtral-xx7B
5/10
4111
Llama-2 Jam-a-277bhhf
0/6
0/7
0/0
34

---

Published as a conference paper at ICLR 2025
Table 8: Well-leamndd digits performance--resseeee digits models tested on NUPA Test according
to digit match.
Add Int
Add Float
Add Frac
Add Easy Frac
Add
Sub Ini
Sub Float
Sub Frac
GPT-4oo-mni
9/20
6/20
0/2
0/2
11/20
9/20
0/1
GPT-4o
10/20
6/20
0/1
0/2
0/
11/20
10 20
0/1
Qwen2-72B
7/20
0/0
070
0//00
6/15
0/16
0/0
Qwen2
4
3/220
0/0
0/0
0/0
6/
6/20
0/0
9/200
0/1
070
0/7
6/199
12/
0/1
Llama-3.1-700
65
7/20
0/1
0/4
6/17
7/20
0/1
Llama-3.1-88
4/9
6/18
0/0
0/0
0/0
6/11
6/17
0/0
Mixtral-xx7B
6/12
0/0
Llama-2-7o-bf
3/6
0/8
0/0
0/0
0/0
3/5
0/9
0/0
Sub
Sci
Int
Float
Frac
Sci
Int
Float
Sci
GPT-4o-mini
0/3
100 100
10/ 100
0/7
19/98
100 100
8/ 100
18 100
GPT
0/0
16/
Qwen2-72B
0/0
/ 100
30/ 100
0/4
100 100
82 100
32 100
100 100
Qwen2-7B
0/0
/
10/98
0/0
82
13
Llama-3,1-8B-
0/0
831 100
75/ 100
0/20
100 100
100 100
79/100
100 100
Llama: 3.1 70B
0/0
86
07
0/4
0/0
0/0
0/ 100
0/
Llama-3.1-8B
0/0
0/67
0/93
0/54
100
0/96
0/100
Mixtral 8x7B
070
7/21
3/
0/0
0/ 62
20 100
4 4/25
0/ 100
1ama-227h-hrf
0//38
Multiply Hard Multiply Hard Multiply Hard Multiply Hard Multiply Easy Multipyy Easy Multiply Easy Multiply Easy
Int
Float
Frac
Frac
Sci
GPT-4o-mini
0/5
0/0
1/2
0/4
0/6
0/0
1/2
0/4
GPT-4o
0/5
0/0
1/2
0/4
0/6
0/0
1/3
0/4
Qwen2-72h
0/5
0/0
0/0
0/0
0/0
Qwen2-7B
0/3
0/0
0/0
0/0
0/4
0/0
0/0
0/0
Llama-3.1-8B-4
0/4
0/0
0/3
0/4
0/5
3.1ma-311-2 70B
0/55
0/3
0/2
0/3
0/6
0/3
0/2
0/3
Llama-3.1-88 1-BB
0/4
0/0
0/0
0/0
0/5
0/0
0/1
0/0
Mixtral-xx7B
0/4
0/0
0/0
0/2
0/0
Llama-2-76-hf
0/0
0/0
0/0
0/0
0/0
0/0
0/0
0/0
Digit
Digit
Add
Truediv Int
Truediv Frac
Floordiv Int
Mud
Int
Float
Int
Float
Int
GPT-4o mini
D/ 100
0/100
0/0
070
075
1/2
5/15
0/3
GPT-4o
0/ 100
0/ 100
0/0
0/0
0/6
1/2
5/15
0/3
Qwen2-72B
0/ 100
0/100
0/0
0/0
0/0
0/1
3/8
0/3
Qwen2-7B
0/
0/33
0/8
0/1
3/6
0/3
Llama-3.1-8B
0/ 100
13/ 100
0/20
5/73
3/20
0/
6/18
0/3
Llama-3.11700
0/
0/000
0/0
0/0
0/0
0/1
4/12
0/0
Llama-3.1-88
0/92
0/0
0/0
0/0
0/0
0/0
3/7
0/0
Mixtral-8x7B
D/ 100
0/97
0/5
075
0/0
0/1
3/7
0/3
Llama-2-7b-hf
0/
0/42
0/0
0/0
0/0
0/0
0/3
0/0
Mod Easy
To Float
To Float
To Scient
To Scient Float
Sig
Int
Frac
Sci
GPT-4o-mini
0/3
0/8
0/9
23/
8/24
31 /
GPTT-4o
0/3
0/8
0/9
23/ 71
8/28
31 100
Qwen2-72B
0/3
6/9
16/28
100
96
100
Qwen2-7B
0/3
4/8
6/22
100 100
0/ 100
19/ 100
LJama-3. 1-70B
0/0
4/9
1/
0/ 100
0/:1
0/21
Llama-3.1-8B
0/0
3/6
0/
0/0
0/0
Mixtral-8x 8x7B
0/3
4/9
14/36
100 100
680
17//000
Llama
0/0
0/5
0/0
0/
0/22
0/0
35

---

Published as a conference paper at ICLR 2025
petprd/dddddooramaaeeeeeerrrrrie Table 9: Well-learned digits/ performaance-ppreservin digits models tested on NUPA Test according
to dlength.
Add Int
Add
Add
Add Easy
Add
Sub Int
Sub Float
Sub Frac
Float
Frac
Frac
GPT-4o-mini
20/22
4/8
0/2
1/2
0/0
20 20
5/9
0/1
GPT-4o
20/20
4/7
0/2
1/2
0/0
20
0/1
Qwen2-72B
20/20
7/
0/0
070
0/0
20/
0/0
0/0
Qwen2-7B
11/13
0/13
0/0
0/0
0/0
10/ 12
0/12
0/0
Llama-3.1-8B-
19/20
11/20
0/7
20/20
13//20
0/5
LJama 3.1 3.1-70B
20/20
6/13
0/1
0/1
0/0
20 20
4/13
0/1
Llama-3.1-8B
/
6/10
Mixtral-xxBB
10/ 14
4/
0/1
0/1
0/0
16/20
5/13
0/0
Llama-2-7b-bf
0/10
0/0
070
4/20
0/11
0/1
Sub
Max
Max Float
Max
Max
Max Hard
Max Hard
Max Hard
Sci
Int
Fine
Sci
Float
Sci
GPT mini
0/0
GPT-4o
0/0
31/72
6/9
0/2
12/17
100 100
6/
8 16
Qwen2-72B
0/0
18/32
13/30
0/0
43/
16/ 26
35
Qwen2.7B
0/0
11/23
0/
0/0
0/14
86 100
0/20
11
Jlama-3.--BB-ft 1-8B-ft
0/0
/7/
35/54
075
97/100
100 100
25
3
Llama-3.1-700
0/0
0/5
0/
0/0
0/5
0/
0/133
0/4
Llama 3.1-8B
0/0
0/0
0/7
0/0
0/0
0/18 18
0/6
0/0
8x7B
Lla-m-2-76-bf
0/0
0/10
0/0
0/0
0/0
100 / 100
0/0
0/ 11
Multiply Hard Multiply Hard Multiply Hard Multiply Hard Multiply Easy Multiply Easy Multiply Easy Multiply Easy
Int
Float
Frac
Sci
Int
Float
Frac
Sci
GPT-4o-mini
7/11
0/0
1/2
0/0
9/20
0/0
1/2
0/0
GPT-4o
0/0
1/2
0/0
Qwen2-72B
0/6
0/0
0/0
0/0
5/7
0/0
0/0
0/0
Qwen2-7B
0/0
0/0
0/0
070
0/0
0/0
0/0
Llama L.1ma-3.188B 3.1-8B it
11/18
3/9
0/11
0/7
16/
8/17
1/15
0/20
LJama-3.1-70 1-70B
5/9
0/3
0/1
070
7/16
0/3
0/1
0/0
Llama-3.1-8B
0/4
0/0
0/0
0/0
0/1
0/0
Mixtral-x7BB
4/6
0/0
0/2
0/0
4/8
0/0
0/2
0/0
Llama-2-7b-bf
0/0
0/0
0/0
Digit Max
Digit Max
Digit Add
Digit Add
Triediv
Truediv
Floordiv
Mod
Float
Int
GPT-4oo-mini
0/7
0/8
0/0
8/
3/
4P
10/ 16
3/7
0/8
0/0
0/5
1/2
8/14
3
Qwen2-72B
0/5
Qwen2-7B
6/ 11
0/4
20
0/9
0/0
0/
4/6
0/5
Llama-3.1-8B-4
//6
19/29
29145
3/20
0/3
9/18
0/
Llama-3.1-22 3.1-7B
6/ 16
0/7
0/11
0/0
0/0
0/1
3/10
0/4
Llama-3.1-88
0/6
0/0
0/10
0/7
0/0
0/0
4/8
0/5
Mixtral-8x.7
0/4
0/8
0/1
4/7
0/5
Llama 2-7h hf
0/7
0/3
0/9
0/7
0/0
0/1
0/3
0/6
Mod Easy
To Float
To Float
Scient Int
Scient Float
Int
Frac
Sci
Int
GPT-4o-mini
3/20
0/4
15
9/21
5/8
0/23
GPT-4o
3/20
0/4
15
9/21
0/23
Qwen2-72B
0/ 10
3/8
12/46
0/44
0/0
6/
Qwen2-7P
3/7
Llama-3.1-8B-4
0/
3/9
14/78
27
16/38
0/0
Llama-33 3.1--0B
0/5
3/7
0/15
070
0/0
0/0
1.ama-3.1-88
0/6
3/6
0/12
0/0
0/0
0/0
Mixtral-8x7B
0/6
3/6
15/16
0/18
0/0
0/0
Llama-2-76-baf
0/6
0/4
93/98
0/0
0/0
Table 14: 8-digit digit match accuracy with small model (1.3M) with RoPE, NoPE and Alibi. Fraction-multipiicate
Int-add
Float-add
Scientific-add
RoPE
0.091
0.88
0.23
0.75
NoPE
0.061
0.39
0.17
0.52
Alibi
0.056
0.31
0.18
0.50
36

---

Published as a conference paper at ICLR 2025
Table 15: 8-digit digit match accuracy with small dataset (IM samples) with RoPE, NoPE and Alibi. Fraction-multillcction
Int-add
Float-add
Scientific-add
RoPE
0.97
0.99
0.33
0.99
NoPE
0.78
0.98
0.29
0.96
Alibi
0.23
0.80
0.17
0.79
Table 17: Exact match of O. IB models trained on integer addition, multiply and maximum respectively with various compositions of reverse formatting and index hints.
Integer Addition
Integer Multiply
Integer Max
reverse + idx
reverse idx
rev
idx
no
idx
rev
no
idx
reverse only
no
idx
0.41
0.43
0.00
0.13
0.00
1.00
0.99
0.99
d10
0.80
0.01
0.13
0.02
0.04
0.02
1.00
0.97
0.98
Table 18: Finetuning with PE, data format, and tokenizer modification will degrade the performance. The first two lines are a naive finetuned Llama and the original Llama without finetuning, which are the baseline. "Id" means using the one-digit tokenizer for numbers otherwise the original tokenizer. "rev" means reverse representation, where the integer parts are reversed. All the checkpoint we select by the lowest valid loss. The accuracy reported the average "exact match" in each range. Metric "wld" is used to denote well-leamed digit: "ppd" is used to denote performance pewcicee-reeeviingg digit.
Integer Addition
Float Addition
M
XL
wld
ppd
S
M
L
XL
wld
ppd
FT w/o FT
0.95
0.65
0.12
0.01
4
12
0.96
0.71
0.27
0.08
5
17
0.95
0.38
0.06
0.02
4
9
0.90
0.47
0.10
0.02
3
11
NoPE
0.00
5
0.37
0.06
0.00
0.00
0
NoPE + rev ld
0.89
0.35
0.06
0.02
3
9
0.81
0.38
0.09
0.01
0
NoPE + rev + pad ld
0.87
0.34
0.05
0.02
0
9
0.74
0.38
0.06
0.01
0
RoPE ld
0.93
0.59
0.05
0.00
4
9
0.33
0.30
0.06
0.01
0
RoPE rov
0.40
0.20
0.04
0.00
0
0.35
0.30
0.09
0.02
Fraction Multiplication (easy)
Scientific Notation Addition
S
M
XL
wld
ppd
S
M
L
XL
wld
ppd
FT w/o FT
0.43
0.00
0.00
0.00
1
3
0.09
0.08
0.02
0.00
0
0.28
0.00
0.00
0.00
0
3
0.02
0.02
0.01
0.00
0
NoPE
0.14
0.00
0.00
0.00
0
3
0.00
0.00
0.00
0.00
0
NoPE +rev ld
0.25
0
NoPE + rev pad + ld
0.27
0.00
0.00
0.00
0
3
0.01
0.02
0.01
0.00
0
RoPE + ld
0.09
0.00
0.00
0.00
0
0.08
0.03
0.02
0.00
0
3
RoPE rev ld
0.06
0.00
0.00
0.00
0
0.08
0.02
0.01
0.00
0
Exact Match
1.0
1.0
0.5
6.8
0.8
Accuracy
6.6
0.6
0.4
0.4
reverse only reverse
0.2
only
6.1M
10.9M
21.6M
32.4M
45.1M
0.1M
10.0M
21.6M
32:4M
43.1N
0.1M
10.0M
21.8M
32.4M
43.1M
Figure 14: Exact match of O. models trained on 1- to 8- digit integer addition with different compositions of reverse formatting and zero padding on 8- to 10- digit tests. X-axis the number of seen training samples.
37

---

Published as a conference paper at ICLR 2025
Exact Match D8
Exact Match D9
Exact Match D10
1.9
M
reverse 1063
fotial
0.8-
0.8-
Accuraty
0.6
0.6
Heverse dec
0.4
reverse int
Teuenru int pad
pod only
0.8
0.0
Figure 15: Exact match of 0.1B models trained on 1- to 8- digit float addition with different compositions of reverse formatting and zero padding on 8- to 10- digit tests. X-axis is the number of seen training samples.
Exact
1.0
1.0
6.8
0.8
Accuracy
0.6-
0.6-
0.4
reuetsv only reverse index Index
0.2
0.1M
0.11
0.1M
Figure 16: Exact match of O.1B models trained on to 8- digit integer addition with different compositions of reverse formatting and index hints on 8- to 10- digit tests. X-axis is the number of seen training samples.
Exact Match
Exact Match
Exact Match D10
0.5
0.10
0.4-
Accurecy
0.06
reverse only reverse index
0.2
6.1
0.02
only
0.3M
10.3M
20.0M
30.8M
0.1M
10.3M
20.6M
30.8M
0.1M
10.3M
20.0M
30.8M
Figure 17: Exact match of O. models trained on 1- to 8- digit integer multiplication with different compositions of reverse formatting and index hints on to 10- digit tests. X-axis is the number of seen training samples.
Exact Match
Exact Match
Exact Match D10
1.0
1.0
0.8
Accurecy
0.6
0.6
3.6
0.4
0.4
0.4
reverse only
index
only
0.1M
7.6M
15.5M
0.1M
7.6M
35.5M
0.1M
7.8M
15.5M
Figure 18: Exact match of O. models trained on 1- to 8- digit integer maximum with different compositions of reverse formatting and index hints on to 10- digit tests. X-axis is the number of seen training samples.
38

---

Published as a conference paper at ICLR 2025
Table 19: Maximum length of each task that 2k context window can afford with RF-CoT
Add
Sub
Multiply
Floordiv
Mod
Max
DigitMax
GetDigit
Length
Integer
20
20
12
20
100
17
100
34
Float
6
5
4
50
100
Fraction
3
2
3
20
Scientific
3
3
3
100
39


Reviews:
[{"review_id": "4uChDzva78", "reviewer": "Reviewer_MwX6", "summary": "This paper investigates the empirical abilities of LLMs to solve numerical reasoning tasks of varying complexity. It proposes a benchmark, called NUPA, incorporating several different number representations (integers, fractions, etc) and reasoning tasks (arithmetic, comparisons, etc). Its experiments show that several well-known LLMs are prone to errors on this benchmark, particularly as the digits get larger. The paper also analyzes how factors like tokenization, data formats, and finetuning affect performance on NUPA.", "questions": "How do you get ground truth reasoning traces for RFFT?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper does a good job of broadening the evaluation of numerical reasoning from integers—which is often the focus in these kinds of studies—to more general classes of numbers. In general, there are a lot of experiments here, including several tasks and language models. The claim that models struggle with basic numerical understanding is well supported by the results. \n2. The discussion on tokenization is quite interesting, surprisingly few studies that I am aware of consider how tokenization affects numerical reasoning. Although not very surprising, it is nice to have empirical evidence that 1-digit tokenization yields stronger generalization to more complex examples than those in the training set. \n3. The paper goes beyond being purely “descriptive”, it also investigates methods that could *improve* numerical reasoning.", "weaknesses": "1. Numerical reasoning in LLMs is well-studied by now. Yet, the paper lacks a related work section and has very few references to similar studies overall. It is therefore unclear to what extent the insights here are novel. Some of the claims of novelty also come off as overly strong, e.g., the last sentence of the abstract “our work takes an initial step towards understanding and improving NUPA [numerical understanding and processing ability] of LLMs.” I include a short list of relevant references below; however, I would suggest the authors to perform a more comprehensive literature review so that they can properly situate their work.\n2. The paper is somewhat poorly written. First, there are many grammatical errors in this text (e.g., l62-3, l123). Second, many parts lack explanation or justification. For instance, it is not explained how the random tokenizer works. The technique appears to be adapted from Sathe et al. (2024), but that is a recent and probably not very well-known paper. On another note, random tokenizers were not introduced by that paper as suggested in the text; see for instance Kudo (2018). Another part that is unclear to me is the section on positional encodings. It is not explained what is meant by length-agnostic and length-related rules. It also doesn’t contextualize the methods studied—RoPE and Alibi. What are they and why are they studied? Is there some plausible explanation for why you observe the results you do?\n3. The paper is experiment-driven; there is no underlying theory guiding the selection of tasks or techniques for improvement. The scope is also very broad—the paper attempts to do many things at once. While that may be seen as a strength, I feel that it comes at the sacrifice of depth of analysis and clarity. I would suggest the authors to construct a more focused narrative and provide justifications that are grounded either in previous work or theory. \n4. The paper lacks important statistical reporting like confidence intervals or p-values. \n5. As a minor additional note, Figure 2 (and the similar figures in appendix) is quite hard to parse. There is a lot of information and it is difficult to distinguish the models since some of the colors are rather similar. It is also confusing that the results for the finetuned model are here since finetuning is discussed much later in the paper. \n\n\n----\n\nDziri et al. 2023. Faith and Fate: Limits of Transformers on Compositionality.\n\nKudo et al. 2023. Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?\n\nRazeghi et al. 2022. Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning.\n\nZhang et al. 2024. Interpreting and Improving Large Language Models in Arithmetic Calculation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YU0QJBhN2s", "reviewer": "Reviewer_T1ZM", "summary": "This paper evaluates language models on tasks involving numerical and arithmetic reasoning. The authors introduce a test suite with 17 numerical tasks (e.g., addition, subtraction, modulo, max, min, and digit retrieval) across four numerical representations, using it to evaluate nine language models. The paper also analyzes the impact of tokenization, positional embedding type, and data format on a model’s ability to perform addition and multiplication. Finally, the authors assess whether fine-tuning, with and without chain-of-thought, can improve LLaMA-3 8B’s performance.", "questions": "The lack of discussion on previous work is a significant drawback. I would be willing to reconsider my score if this is addressed in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The evaluation procedure is well-structured and comprehensive.\n- Some tasks in the evaluation suite offer useful insights into model failure modes.", "weaknesses": "- Attribution and discussion of previous work is extremely poor:\n    - Notably, a discussion of the related work is missing. The the proposed evaluation suite, the design choices for it, and the results obtained should be discussed in light of previous studies that evaluated and analyzed the performance of language models on task involving numerical reasoning [e.g., 1,2,3,4,5,6]\n    - The authors fail to reference papers that introduce methods that they directly adopt: chain-of-thought prompting was introduced by Wei et al. [7], and LoRA was proposed by Hu et al. [8]. Both citations are missing.\n- Although the authors study the impact of tokenization on the performance of a model that they train form scratch, a discussion of how the tokenization of the pre-trained models evaluated might affect the results is missing. Additionally, here again, any reference to previous work that studied the impact of tokenization on numerical reasoning [e.g., 9] is absent.\n- Some presentation/clarity issues:\n    - Presenting the fine-tuning results in Table 2 is confusing, as they are discussed only much later in Section 4.\n    - Figure 2 can be improved, especially in the choice of color for the LLaMA models, which are quite hard to distinguish in the bar plot.\n    - Line 512.5: I understand that rule-following fine-tuning is introduced by Hu et al., but further details about the fine-tuning process (potentially in the appendix) would be helpful.\n    - The use of negative \\vspace is excessive, resulting in cramped spacing between tables/figures and text, especially on pages 7 and 9.\n\n---\n[1] A Survey of Deep Learning for Mathematical Reasoning (Lu et al., ACL 2023)    \n[2] Do NLP Models Know Numbers? Probing Numeracy in Embeddings (Wallace et al., EMNLP-IJCNLP 2019)    \n[3] Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models (Lin et al., EMNLP 2020)    \n[4] A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models (Stolfo et al., ACL 2023)    \n[5] Injecting Numerical Reasoning Skills into Language Models (Geva et al., ACL 2020)    \n[6] Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning (Razeghi et al., Findings 2022)    \n[7] Wei, Jason, et al. \"Chain-of-thought prompting elicits reasoning in large language models.\" Advances in neural information processing systems 35 (2022): 24824-24837.    \n[8] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021).    \n[9] Singh, A.K. and Strouse, D.J., 2024. Tokenization counts: the impact of tokenization on arithmetic in frontier llms. arXiv preprint arXiv:2402.14903.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5lGc5VCtYd", "reviewer": "Reviewer_m8mv", "summary": "Designs a comprehensive benchmark for number understanding, covering four different number presentations (e.g. integers) and 17 tasks (such as multiplication). The benchmarks allows for testing a range of different numerical understanding skills, at different levels of difficulty (through longer numbers in terms of digits for example). The authors do a comprehensive range of experiments on the benchmarks, evaluating off-the-shelf LLMs on it, as well as training their own models to understand the effects of aspects of common LLMs (such as tokenizers) and existing methods for improving numerical understanding (such as representing in reverse order). The findings are the many models still struggle in elementary understanding tasks, especially for larger numbers. Moreover, the existing tricks to deal with this don't fully mitigate the problems.", "questions": "Main questions is can you give details on how the experiments that use training of fine-tuning are done, and what in-domain and out-of-domain refers to (what examples do you hold out for this)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper has the potential to be an important contribution to the field, for two main reasons:\n\n- Great benchmark proposed with a comprehensive range of tasks and difficulty levels, and good metrics for evaluating performance\n\n- Excellent coverage of experiments. Testing comprehensively many aspects relevant to this problem, like different types of generalisation, different architectural changes, different existing models, effects of scale, existing methods for improving numeric understanding, prompting, etc. I would be very interested in these results, if the weaknesses below could be addressed.", "weaknesses": "Unfortunately, the presentation is lacking and it's very hard to interpret the results because of missing details on how models are trained.\n\n**Main weaknesses**\n- There is no information at all in the paper about how the models are trained for the experiments in section 3.1 and 3.2. How much examples do you train on? How long? Optimizers, hyperparameters, repeats, etc.? How do you define in and out-of-domain? It's very difficult to interpret the results of this experiment without knowing more.\n\n- The paper needs substantial rewrites, and will probably benefit from corrections by someone whose first language is English or by using an LLM to revise and point out mistakes. \n\n- There is a long discussion of things like the relevance of integers and how fractions arise (section 2.1 and 2.2), but this seems unnecessary and the space could be used instead to represent the results better. For example, Figure 2 presenting the main results in the text is very small and almost impossible to read without zooming in 200%. The colors used are also hard to distinguish. I would suggest shortening section 2.2 and almost entirely removing 2.1 and use the space to represent the results better. Especially because this is the great part of this paper, you have so many results but you do not discuss the details of the experiments and present the results in a way that are hard to follow.\n\n- Some results are interpreted in ways that are not substantiated by the evidence found. For example, line 298 to 309; I disagree that this result means the model does not understand the concept of digit. The fact that it becomes harder to return the right digit for longer numbers actually hints at something else going on that might be more related to the model's inherent difficulties with retrieving something from a position in longer sequences, which has nothing to do with it's understanding of the concept digit. I also disagree that this points to case-based reasoning, because again it might be due to some aspects of its architecture. Additionally, the fact that models of different sizes show the same performance on a task does not necessarily indicate the performance depends on the training approach over size, it might also depend on architecture.\n\n- There is no section on related work, and the authors say in the beginning of section 2 that they will show limitations of prior benchmarks while discussing the coverage of their own, but this does not happen. This makes it difficult for the reader to place this contribution w.r.t. existing literature.\n\n**Minor points**\n\n- Would be great to already get some more concrete information in the abstract (what probability of error? which 3 factors influence it?). Or at least in the intro some more concrete info.\n\n- Figure 4 and 5: it's unclear what's on the X-axis (though one can assume it's training steps), and also unclear what D6 to D10 refers to. It's also unclear what is in-domain and what is OOD without reading the text.\n\n- *\"These types of errors are a major cause of hallucinations when dealing with math, reasoning, and data analysis tasks, as the model presents seemingly correct problem solving approaches but ultimately produces incorrect results\"* -> this statement requires a citation\n\n- line 117 to 119; a discussion on the innateness of integer understanding seems somewhat out of scope for this work and not too relevant to the contribution.\n\n- Cite / reference openai chatgpt when using it line 135\n\n- One approach that seems missing is few-shot examples, which might significantly boost performance for all models. Not just because they might learn from the examples, but because they get primed on the output format required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xpeUbdbYan", "reviewer": "Reviewer_Z2vn", "summary": "This work introduces a benchmark suite of \"numerical understanding and processing ability\" (NUPA) tasks for transformer-based LLMs. The task suite separates numeracy from potentially confounding logical reasoning problems, and presents a variety of numerical formats. Additionally, various tricks are explored to improve LLM performance in these numeracy tasks; no silver bullets.", "questions": "1. Why is it important that LLMs be good at NUPA when we python, and even LLM code-generation that can write python to deal with NUPA? I.e. Why is this a worthwhile problem, and who says it is?\n\n2. To what extent do your findings suggest that the issue is architectural or to do with tokenization? To the degree that this is the case, to what extent do your findings suggest that the issue is paradigmatic rather than technical?\n\n3. Is it necessary to provide normative justifications for the benchmarking suite? If so, I would like to know why, because I think the paper could be stronger if the benchmarking tests were presented matter-of-factly, without detours.\n\n4. Experimentally, how much compute (as an estimate) was used throughout?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It is clear that some thought has gone into curating the benchmark tasks and justifying their inclusion. Even though I do not personally agree with some of the justifications and normative statements made, I believe that this sort of conscientious effort to justify a benchmark is commendable. I rate this work highly in terms of integrity, quality of presentation, and clarity.\n\nI particularly appreciated the authors' investigation of CoT, as it is generally interesting to understand whether \"prompt-level\" interventions have a lot of room for improving performance on complex tasks. A recent (contemporaneous, so not factoring into my decisions) paper by Apple on GSM8k qualifying LLM performance as memorisation seems to agree with the findings of this work that numeracy abilities do not appear to scale; potentially a way to frame and situate this work with respect to the broader discourse on generalisation and understanding in LLMs.", "weaknesses": "I have no issues with the benchmark and evaluation. I do not currently find the conceptual motivations and wrapping compelling; I could be convinced otherwise for each of these with the help of citations, or of better arguments, and I would be happy to raise my score accordingly.\n\n1. I am unconvinced that numeracy in LLMs is a problem in need of a solution. First, surely there is a citable source for LLM inadequacy for numeracy. Second, even if they were terrible at numeracy, the onus is on the authors to convince the reader that this a problem worth caring about, for at least two obvious reasons: 1) all of these tasks are already trivially done by a calculator or a python program, and 2) commercially available LLMs can probably do alright at numerical tasks indirectly via code-generation and execution. As it stands, it reads as if the authors are insisting that this is a problem deserving of attention --- I'm sure it could be, but this argument can be better made.\n\n2. I am unconvinced that numeracy in LLMs is a problem in search of deeper understanding. Consider that \"How many r's in strawberry\" is a meme that normal people know about; the idea that tokenization impairs the ability of transformers to reason at a letter-, digit-, or word-unit level is well-digested. So when the authors claim that the weakness of LLMs at digit understanding is surprising (line 298), I find this a bit sensationalist and not grounded with respect to a broader and basic level of discourse around the capabilities of LLMs.\n\n3. I am unconvinced of the normative rationales supporting the benchmarks, which seem ad-hoc. Here is an example: the authors claim that multiplication (not a particular algorithm) is O(L^2) in the length L counted in digits (of presumably both inputs; line 169), and take this as justification that models ought to find multiplication difficult. This doesn't hold up to armchair introspection: lookup is O(1), and we have all learned our times tables as children. The competent-human process of mental multiplication is mostly a sequence of lookups along with additive de/recomposition, so the real exponent is probably less than 2. This is a jarring oversight when contrasted with the effort with which the authors bring up cognitive and developmental psychology (lines 116 to 127), recasting what I assumed was a scholarly inclusion as a kind of checklisting exercise. There are other unsupported normative claims about what a good benchmark ought to encompass, such as \"NUPAs are necessary and expected to learn by LLMs [sic]\" (line 114), \"any student who has completed primary and secondary education should be able to accomplish them\" (line 160), and \"for most practical tasks involving numbers (like arithmetic or math tests), all we care about is whether the answer is right, and there is no significant difference between being almost right and being completely wrong\" (line 228). While I understand the importance of establishing normative standards to justify a benchmark with respect to, I find it hard to believe as a reader that the authors have the expertise and authority to do so.\n\n4. I am unconvinced that this is worth \"taking seriously\"; let us suppose that everyone agrees that NUPA in LLMs is a problem worth solving and that improving on this benchmark is the way to solve the problem. We know beforehand that transformers have problems with NUPA due to tokenization, and now the finding of this paper is that fine-tuning and CoT as tricks don't solve the problem. There are many graphs in this paper that must have taken a fair amount of compute to obtain; is it worth it if everyone decided to run similar tests, or fine-tune with respect to the NUPA suite? I don't mean this in an unnecessarily antagonistic way, but one concerned with scientific interest and downstream impacts; if LLMs are just bad at numeracy for architectural or structural reasons as the current evidence suggests, could the introduction of a benchmark be inviting high-compute low-innovation approaches to the problem, and does the problem even merit this kind of resource investment?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)