Number Cookbook: Number Understanding of Language Models and How to Improve It

## Abstract

Abstract Large language models (LLMs) can solve an increasing number of complex reasoning tasks while making surprising mistakes in basic numerical understanding and processing (such as 9.11 > 9.9 9.11 9.9 9.11>9.9 9.11 > 9.9 ). The latter ability is essential for tackling complex arithmetic and mathematical problems and serves as a foundation for most reasoning tasks, but previous work paid little attention to it or only discussed several restricted tasks (like integer addition). In this paper, we comprehensively investigate the numerical understanding and processing ability (NUPA) of LLMs. Firstly, we introduce a benchmark covering four common numerical representations and 17 distinct numerical tasks in four major categories, resulting in 41 meaningful combinations in total. These tasks are derived from primary and secondary education curricula, encompassing nearly all everyday numerical understanding and processing scenarios, and the rules of these tasks are very simple and clear.
Through the benchmark, we find that current LLMs fail frequently in many of the tasks. To study the problem, we train small models with existing and potential techniques for enhancing NUPA (such as special tokenizers, PEs, and number formats), comprehensively evaluating their effectiveness using our testbed. We also finetune practical-scale LLMs on our proposed NUPA tasks and find that 1) naive finetuning can improve NUPA a lot on many but not all tasks, and 2) surprisingly, techniques designed to enhance NUPA prove ineffective for finetuning pretrained models. We further explore the impact of chain-of-thought techniques on NUPA. Our work takes a preliminary step towards understanding and improving NUPA of LLMs. Our benchmark and code are released at https://github.com/GraphPKU/number_cookbook .

## 1 Introduction

The mathematical and reasoning abilities of large language models (LLMs) are currently quite impressive (OpenAI, ; Meta, ; OpenAI, ; Yang et al., ), capable of solving problems at the level of a graduate student or even more difficult ones like olympiad-level problems (He et al., ), GAOKAO (a nationwide examination of high school students applying to universities
in China) (Zhang et al., ) and college mathematics (Tang et al., ). However, upon closer examination of the models’ outputs, we found that although the models demonstrate remarkable proficiency in problem-solving approaches, they often struggle with basic numerical understanding and processing — like a careless student who claims, “I know how to do it, but I didn’t get it right.” Some of these errors are quite surprising, such as believing that $9.11>9.9$ or making mistakes in simple addition $8/7+3/5$. These errors are a major cause of hallucinations when dealing with math, reasoning, and data analysis tasks, as the model presents seemingly correct problem-solving approaches, but ultimately produces incorrect results. Therefore, investigating and improving the fundamental “numerical understanding and processing abilities” (NUPA) of models is crucial.

However, in current research, reasoning ability and NUPA are often tested together, both on classic datasets such as GSM8k (Cobbe et al., ), MATH (Hendrycks et al., ), MMLU (Hendrycks et al., ), and in more challenging tests mentioned above. For example, a problem in GSM8k is: “Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?” Solving this problem requires two aspects: on the one hand, mathematical reasoning including understanding the text, extracting relevant information, formulating mathematical equations (or finding other solution methods), solving the equations or executing an algorithm, and obtaining the result; on the other hand, it also requires understanding and processing the numbers provided in the problem or produced as intermediate results at each step, like $48/2=24$ and $48+24=72$. While these two abilities are both essential to correctly solving the problems, tests on such datasets do not distinguish between them.

A more severe issue is that the numerical content is often deliberately simplified in these datasets. In various exam questions (like in the American Invitational Mathematics Examination (Li et al., )), to focus on assessing students’ understanding of mathematical concepts — such as how to set up the correct equations and apply the right theorems — the numbers in both the questions and answers are often specially chosen to be integers. However, this is not the case in real-world scenarios (Chen et al., ).

Despite the importance of NUPA, there is still a lack of accurate, detailed, and comprehensive formalization, measurement, and analysis of this fundamental capability. In this paper, we take the preliminary step towards formalizing the NUPA of LLMs.
We categorize the numerical concepts and operations commonly taught in primary and secondary education into four types of numerical representations: integers, floating-point numbers (finite decimals), fractions, and scientific notation, along with four ability categories comprising 17 tasks. Pairing these representations results in 41 meaningful tasks, forming our NUPA benchmark (Table 1). These representations and tasks cover the most common scenarios involving number understanding and processing, which are typically not challenging for humans, as we read, use, or process such numbers nearly everyday.

On this benchmark, we rigorously test several state-of-the-art LLMs containing GPT-4o (OpenAI, ), Llama-3.1 (Meta, ) and Qwen2 (Qwen Team, ). We ask models to directly output the answers without calling external tools. Although the latest LLMs perform well on some easiest tasks, their performance declines significantly as tasks become slightly more complex (such as multiplication, modulo operations, or digit-based calculations), or as the representation of numbers extends beyond basic integers. See Figure 2 of Section 2.4. The overall unsatisfactory performance highlights a pronounced mismatch between the claimed strong mathematical reasoning abilities and the poor practical, everyday numerical understanding and processing abilities of today’s LLMs.

To address this issue, we explore three categories of approaches to enhance the NUPA of models. The first category of techniques aim at improving models’ NUPA during the pretraining stage, including alternative tokenization, specially designed positional encoding (PE) (Haviv et al., ; Kazemnejad et al., ; Zhou et al., ), changing number formats (like zero-padding, index-hint (Zhou et al., ) and reverse representation (Lee et al., ; Zhou et al., )).
We evaluate and analyze them on our newly introduced benchmark, verifying their effectiveness/ineffectiveness on respective tasks/representations, which extends over previous evaluation mainly on the integer addition/multiplication tasks. Further, we summarize these techniques into three mechanisms: simplifying the reasoning process, aiding digit alignment, and providing regularization, and discuss the potential of these mechanisms to be applied across a broader range of numerical representations.

**Table 1: Task overview of NUPA Test. The four rows represent four numerical representations, and the 17 columns correspond to different tasks. ✓: 41 tasks included in our test. ✗: Not included, too complex. $\bigcirc$: Not directly included but can be easily adapted from an included task. $-$: Not applicable. The detailed explanation for these non-included tasks is provided in Appendix A.1.4**
|  | Elementary arithmetic | Comparison | Digit Understanding | Conversion |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Add | Sub | Multiply | Truediv | Floordiv | Mod | Max | Min | Digit<br>Max | Digit<br>Min | Digit<br>Add | Get<br>Digit | Length | Count | To<br>Float | To<br>Scientific | Sig.<br>Fig. |
| Integer | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | $-$ | ✓ | ✓ |
| Float | ✓ | ✓ | ✓ | ✗ | $-$ | $-$ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | $\bigcirc$ | $-$ | ✓ | ✓ |
| Fraction | ✓ | ✓ | ✓ | ✓ | $-$ | $-$ | ✓ | ✓ | $-$ | $-$ | $-$ | $-$ | $-$ | $\bigcirc$ | ✓ | $\bigcirc$ | $\bigcirc$ |
| Scientific | ✓ | ✓ | ✓ | ✗ | $-$ | $-$ | ✓ | ✓ | $-$ | $-$ | $-$ | $-$ | $-$ | $\bigcirc$ | ✓ | $-$ | $\bigcirc$ |

The second category of approaches aim to improve NUPA for an already trained model. We find that while simple direct finetuning can significantly enhance NUPA performance, applying the aforementioned techniques (PEs, data formats and tokenizers) at this stage may have adverse effects. We test various settings and fine-tuning configurations, but none are able to achieve performance equal to or better than the original model. Our results suggest that these modifications can significantly disrupt the models’ established behavior or conflict with its pre-existing knowledge, leading to a decrease in performance.

Finally, we discuss the potential of using chain-of-thought (CoT) techniques for numerical processing. Although CoT methods can break down complex problems into simpler sub-tasks and significantly increase the likelihood of obtaining correct answers, their drawbacks — such as consuming a large context window and requiring extended processing time — become particularly apparent in numerical tasks. We test a general CoT method known as RFFT (Hu et al., ), and find that for more complex tasks (such as those with $O(n^{2})$ complexity, including multiplication, division, and fraction simplification), chain-of-thought methods face scalability challenges, making them difficult to be applied in practical scenarios. It is noteworthy that in this paper, we do not discuss tool use methods (Schick et al., ; Lu et al., ) for NUPA as 1) we want to study the self-contained NUPA of LLMs, 2) calling external tools whenever encountering numbers increases the inference latency (Xu et al., ), and 3) we believe NUPA without tools is a necessary ability of AGI.

In summary,
we propose a more comprehensive benchmark on the basic numerical understanding and processing abilities (NUPA) of LLMs, evaluate several SOTA LLMs’ performance on it, and further study three categories of approaches to improve NUPA: pretraining, finetuning and CoT.
Our results reveal that the current research is insufficient to fully address the NUPA problem, despite it being a fundamental capability for solving many more complex tasks. We hope that by introducing a systematic classification and more comprehensive evaluation of NUPA, we can bring greater attention from the community to this important but overlooked fundamental capability.

## 2 NUPA test: a benchmark for number understanding and processing ability

In this section, we will introduce our NUPA benchmark from the following four aspects: number representations, tasks, metrics, and result analysis of current LLMs. We will explain the rationale behind the inclusion (or exclusion) of specific representations and tasks in our benchmark, highlighting their distinctive features. By comparing these representations and tasks with previous benchmarks, we will demonstrate that several common tasks and abilities have been overlooked in prior evaluations.

### 2.1 Number representation

As we discuss above, we believe that the educational curricula on the primary and secondary school levels is a valuable reference for determining the essential NUPAs that LLMs should master. We identify four number formats that are both common and sufficient to cover most practical scenarios.

- •
Integer: Integer is the first set of numbers that we met in primary school and the most common number representation. Some experiments on animals (Davis & Memmott, ) and infants (Strauss & Curtis, ) have found that understanding of (simple) integers is somewhat innate because it is used naturally in counting various objects. Understanding and operations of more complex integers are also acquired very early in human civilization (Smith & Karpinski, ; Dauben, ). At the same time, integers serve as the foundation for other mathematical representations.
- •
Fraction: We consider fractions with integer numerators and denominators. Once the concept of “division” is introduced within the set of integers, the notion of a fraction naturally arises. Extending from integers to fractions (i.e., rational numbers) represents the first major expansion in our understanding of numerical systems. In practical situations, especially those involving distribution, fractions become unavoidable. However, while the concept of fractions is intuitive, performing operations with them is more intricate. As discussed later, mastering fraction operations remains a significant challenge for current LLMs.
- •
Floating-Point Number (Float): Floats, or finite decimals, are a useful subset of fractions. Calculations with floats like addition and comparison, work similarly to integers, making them common in daily life. Most fractions are eventually converted to floats when exact precision isn’t needed. Floats are also one of the most prevalent formats in computer. Converting fractions to floats may introduce unacceptable inaccuracies in some cases. Consider the following problem:
The bookstore sells newspapers and magazines. Bob bought 7 newspapers and 3 magazines for 13 dollars, and Alice bought 3 newspapers and 7 magazines for 23 dollars. How much are the newspapers and magazines, respectively?
ChatGPT sets up the equations: $7x+3y=13$ and $3x+7y=17$ and uses the elimination method, $x=\frac{13}{7}-\frac{3}{7}y$ to find the integer solution $x=1$, $y=2$. In this case, the fractions $\frac{13}{7}$ and $\frac{3}{7}$ cannot be approximated as floats (1.857 and 0.429), as this would prevent us from arriving at the exact integer solution and get $y=2.001$ instead. Thus, fractions and floats are tested separately.
- •
Scientific Notation: Scientific notation is a common way to represent floats, characterized by separating a number’s precise value from its order of magnitude. The magnitude is expressed using a base-10 exponent, while the float (called significand) represents the precise value. It is widely used in fields like physics, economics, and computer science because it efficiently handles a wide range of numbers and clearly conveys significant figures and precision. For LLMs, mastering scientific notation can significantly enhance their ability to handle practical tasks, such as interpreting financial reports or understanding scientific texts. However, this format introduces complexity, as numbers with different exponents cannot be easily aligned by the decimal point.

Details of these four representations in our benchmark can be found in Appendix A.1.1.
There are possible representations of numbers that are not included in these four formats, like complex numbers, infinite decimal representation (repeating and non-repeating), radical expression (like $\sqrt{2}$), … These representations either occur infrequently in practical conversations (e.g., complex numbers) or present significant challenges for language models to process without the aid of external tools (e.g., radicals). For these reasons, we have opted not to include them in our benchmark at this stage.

### 2.2 Tasks in four ability categories

Another aspect of NUPA is defining the tasks that the model needs to handle. The tasks should have clear calculation rules — any student who has completed a full primary and secondary education should be able to accomplish them. Furthermore, most practical numerical processing tasks should either fall within these tasks or can be easily transformed into some of them. We propose 17 tasks across four ability categories. The complete task list is shown in Table 1 and we provide an example for each task in Appendix A.1.2.

Below we discuss the rationales for including some tasks in detail.

- •
Elementary arithmetic: addition, subtraction, multiplication, and division. They are the most fundamental mathematical operations and the first branch of mathematics taught in schools. However, some operations can be complicated when different number representations are involved. For example, fraction addition is more complicated than multiplication because it needs to be reduced to a common denominator first.
–
True division, floor division and modulus: The division is somewhat unique because it is not closed for integers and floats. Here, we consider three common division-related calculations. True division: To maintain precision, we represent the division of two integers as a simplified fraction. Combined with the “significant digits” task we will mention later, this can approximate the result of dividing two integers as a float. Integer division and modulus: Represent approximate multiple relationships, frequently used in practical applications, such as dividing individuals into batches.
- •
Comparison: max and min. Another important aspect of understanding numbers lies in the concept of “order”. To truly comprehend a number, we must know how large it is and whether it is greater or smaller than another one. Moreover, comparison serves as the foundation for other significant operations. For instance, when adding negative and positive numbers, we determine the sign first and then subtract with their absolute values — this involves identifying which of the two numbers has a greater absolute value.
- •
Digit understanding: The concept of a digit is fundamental. Unlike the “value” of a number, a digit is tied to its specific representation. When we care about a language model’s understanding, processing (and generation) of numbers, digit is a crucial concept, as numbers are not read and processed by the language model as a whole, but rather as a sequence of digits. We are curious whether LLMs truly understand the concept of digits. Therefore, we specially designed some digit-related tasks, including:
–
Get digit: Given a number and an integer $i$, return the $i$-th digit. This task is important when certain digits have special meanings in a number (such as a phone number or SSN).
–
Length: Return the total length (i.e., the number of digits) of a number.
–
Count: Count the times that a particular digit occurs in an integer.
–
Digit compare: Compare and return the larger (smaller) digits one by one.
–
Digit add: Perform the normal addition digit by digit but ignore any carrying. For example, $\operatorname{digit_add}(12345,34567)=46802$. It can test a model’s understanding of digit alignment and its mastery of single-digit addition.
Through these tasks, we can assess whether models correctly understand the concepts of digits, length, positions, and the alignment of the digits between two numbers.
- •
Conversion between representations: we design tasks for converting a number to two representations: to float and to scientific notation, as they are frequently used to present final results. These two tasks also create transformations between different representations to test whether models can understand the relationship between various numerical formats. In particular, since many tasks present answers as approximate values, we designed a “significant digit” (sig. fig.) task to evaluate a model’s ability to round long numbers to fixed-length significant digits.

The combination of representations and tasks ultimately result in a total of 41 meaningful pairs. Without confusion, we also call them tasks. The tasks receive either one or two numbers as inputs and return a number as result and the input numbers and results share the same representation for most tasks unless otherwise stated (refer to Appendix A.1.3). The remaining combinations are excluded due to being excessively complex, uncommon, inapplicable, or redundant with other tasks. For further details, see the discussion in Appendix A.1.4.

The difficulty of each task depends not only on the nature of the task itself but also on the length of the numbers
to be processed — longer tasks involve more steps of internal operations, as well as longer inputs and outputs. Therefore, we test on different problem lengths. For tasks that are inherently more difficult, we limit the size of the problem to 1-20 digits, and for easier tasks to 1-100 digits. (For which tasks are considered difficult or easy, please refer to the Appendix A.1.5.)

We generated 1,000 questions for each task and each length. Unlike some previous works that set the lengths of two numbers to be the same, in our tests, the length $L$ of a question is determined by the longer of the two numbers, while the length of the shorter number follows a uniform distribution between $1$ and $L$.
We implemented additional handling to ensure that generated problems do not result in overly simple, complex, or meaningless results. Some tasks are further split into a hard and an easy version. More details about generating the benchmark are provided in Appendix A.1.6.

### 2.3 Metrics about NUPA

Measuring the performance of NUPA benchmarks on these tasks is not trivial. “Exact match” accuracy is undoubtedly the golden standard of the performance where the answer is considered as correct when it exactly matches the groundtruth. For most practical tasks involving numbers

(like arithmetic or math tests),
all we care about is whether the answer is right, and there is no significant difference between being almost right and being completely wrong.
However, a smoother and more detailed metric is useful to understand the behavior and capabilities of a model and how to improve them. Therefore, we also report the “digit match” and “dlength” (difference of length) metrics, as metrics of digit accuracy and length accuracy respectively.
We first split numbers into parts (e.g., integer and decimal parts of a float, numerator and denominator of a fraction) and align the generated answer with the groundtruth digit by digit. Integer parts (such as integer itself, the integer part of a float, the fraction numerator and denominator, the exponential of a scientific notation) are aligned from the least significant digit; and for the decimal part of float (and the float in scientific notation), they are aligned from the most significant digit. For “digit match”, we measure the correctness of each digit, with missing digits considered as errors, and report the overall accuracy. For “dlength”, we report the sum of absolute difference in length between each part of the prediction and the groundtruth. Figure LABEL:fig:metrics illustrates these three metrics.

For each task, we divide the digits into four intervals (S, M, L, XL). For tasks with lengths 1-20, the four intervals correspond to 1-4  5-8, 9-14, 15-20 digits respectively. For tasks with lengths 1-100, they correspond to 1-10, 11-20, 21-60, 61-100 digits respectively. We average the results in each interval for each task and metric.
More details of our metrics are given in Appendix A.1.7

### 2.4 Performance of current LLMs

We test some commonly used LLMs on our benchmark, including three Llama models: Llama-2-7b, Llama-3.1-8b and Llama-3.1-70b (Meta, ), one of the most popular open-source model families from Meta; Mixtral-8$\times$7B (Jiang et al., ), a strong MoE model; and Qwen2-2B and Qwen-72B (Qwen Team, ) which are also open-source models that are believed to have strong math abilities. Finally, we also test state-of-the-art commercial models GPT-4o-2024-08-06 and GPT-4o-mini-2024-07-18. We use prompts to control models to directly output result numbers without relying on external tools or CoT. The prompts used for each model and task are included in Appendix A.2. We select the results of some typical tasks in each category in Figure 2, while the complete results and discussion on all metrics are shown in Appendix A.3. We have several observations regarding the results:

Figure: Figure 2: Parts of performance of state-of-the-art LLMs on NUPA benchmark.
Refer to caption: x1.png

The best model performs well on typical tasks, but its performance declines on more specialized tasks. We find that GPT-4o, GPT-4o-mini and Qwen2 handle typical tasks, such as integer addition, float addition, integer max, and integer length, with high accuracy in the S and M ranges. This aligns with their strong performance on various mathematical datasets. However, their accuracy drops sharply when working with less common representations, like fractions and scientific notation, with average accuracy falling below 20%, even for the shortest S-range (1-4 digits). Similarly, for tasks such as significant figures, modulo operations, and digit-based calculations, their performance was unsatisfactory. This highlights the current limitations of LLMs in understanding numerical diversity and complexity. Despite their good performance on a narrow set of numerical tasks, they struggle with many others, failing to produce accurate results in these areas.

Length remains a significant challenge for NUPA of LLMs. We observe a noticeable decline in accuracy for even simple tasks like integer addition as the problem length increases. For instance, GPT-4o’s accuracy drops from nearly 100% in the S range and 80% in the M range to around 40% in the L range and just 15% in the XL range. In the more complex task float addition, the accuracy decreases from 90% (S) and 60% (M) to merely 15% (L) and less than 5% (XL). This trend is consistent across other models and tasks. For example, Qwen2’s performance in the integer-length task declines from almost 100% in the S range to 50% in the M range, and falls below 5% in the L and XL ranges.

Length impedes learning both individual digits and overall length.
To understand why models struggle with longer input numbers, we examine digit match and dlength performance in Figure 7 and Figure 8 in Appendix A.3. These metrics reveal that length affects both the accuracy of individual digits (digit match) and the answer’s overall length (dlength), with variations across tasks. For example, GPT-4o and Llama-3.1 display consistently low dlength in the add-integer task, with digit match decreasing sharply as length increases, suggesting that length primarily impacts per-digit accuracy on this task. Conversely, in the max-float task, dlength increases significantly with length (about 30-60 in the XL range), while digit match has still 60% in the XL range. Note that since missing digits are treated as errors, this 0.6 digit match is likely due to these missing digits. This suggests that the main challenge here lies in generating answers of the correct length, rather than individual digit accuracy. In other tasks like fraction, both length and digit accuracy issues arise, as reflected in rising dlength and declining digit match.

The concept of “digit” is more challenging than expected. We were surprised to find that LLMs struggle to fully grasp the concept of “digit”. For instance, in the “get digit” task, where the model is asked to return the $i$-th digit of a long integer, performance drops significantly as the length of the number increases. This suggests that current LLMs lack a consistent understanding of what a digit represents. Note that the performance is good in the shorter S-range, which indicates that the models can at least comprehend the task instruction.
In the XL-range, GPT-4o achieves only 20% accuracy, barely above the random guessing 10% baseline (since the correct answer is always a digit between 0 and 9). This fundamental limitation may explain why current LLMs struggle with numerical understanding and processing, especially as task complexity and input length increase. If a model cannot reliably identify a specific digit in a given number, it casts doubt on its ability to generalize to more complex arithmetic tasks, such as addition. The performance on such tasks may heavily rely on case-based reasoning, as suggested by Hu et al. ().

We also have some interesting observations:
(1) LLMs find the “max-hard” task easier than “max” with integer inputs. The difference between these tasks is that in the max task, the two numbers often differ in length, whereas in max-hard, they are always the same length and share some left-most digits, requiring more digits to be compared. While max-hard intuitively seems more difficult, models actually perform better on it. This is likely because they struggle to effectively use sequence length information, as reflected in their weaker performance on the “length” tasks in the longer ranges. It suggests that models might process tasks in different ways from humans. They could have to compare two numbers digit by digit. In this situation, the “harder” subtasks are actually easier because the numbers are already aligned.
(2) GPT-4o and GPT-4o-mini show nearly identical performance across most tasks, similar to the comparison between Qwen2-72B and Qwen2-7B. This suggests that once a model reaches a certain size, NUPA performance relies more on factors like training strategies, data diversity, and post-training refinements, rather than simply on increasing model size.

## 3 How do tokenizers, PEs and data formats affect NUPA?

We have observed that the NUPA test poses significant challenges even for the most advanced LLMs. In this section, we aim to investigate the factors that can influence the NUPA of LLMs during their pretraining phase.
We utilize the architecture of decoder-only transformers and alter the number of attention heads and layers, the hidden size and the intermediate size to create models of various parameter sizes, like 0.1B, 0.9B and 3B parameters.
These models are trained from scratch, incorporating a wide range of techniques that could potentially impact NUPA. In this section, each model is trained on a single task because training a model from scratch that can recognize and perform different tasks is difficult and beyond the scope of our discussion here. Through these experiments, we aim to confirm the effects of various methods, including tokenization strategies, PEs, and different data formats.

Figure: Figure 3: Different tokenization of a long number. (a) mixed digit tokenizer, used in GPT-2. (b) one-digit tokenizer, used in Llama-2. (c) three-digit tokenizer, used in GPT-3.5, GPT-4 and Llama-3. (d) Aligned three-digit tokenizer.
Refer to caption: x2.png

### 3.1 Tokenizer: One-digit tokenizers are good enough

LLMs interpret numbers as segmented tokens rather than whole numbers. With the development of language models, various tokenization strategies have emerged, including mixed tokenizers, one-digit tokenizers, and $k$-digit tokenizers ($k\geq 2$), as shown in Figure 3.
In the BPE tokenizer used by GPT-2, the numbers are not specially treated, which resulted in irregular number cutting and is harmful to digit alignment. The cutting of numbers in modern tokenizers has become more aligned. These tokenizers greedily segment a number from left to right into $k$-digit tokens until a remainder shorter than $k$ digits is left, which is then segmented into a single token. Llama-2 use a one-digit tokenizer, but all of the latest LLMs use a tokenizer with $k=3$, subsequently coming with an extended vocabulary for numbers. Additonally, Singh & Strouse () discover that just alternating the greedy direction from “left-to-right” to “right-to-left” (for integers) can improve performance of Llama-3 and GPT-4.

There is a growing tendency to expand the vocabulary size as the number of parameters in LLMs rapidly increases. Recent work has shown that a larger vocabulary is more suitable for larger LLMs (Tao et al., ) because longer tokens can encapsulate more complex and precise meanings for general word tokens. But number is a different question:

- •
The long-tail phenomenon (Raunak et al., ), common in word tokens, is not as pronounced for the number tokens. The distribution of number tokens is closer to a uniform distribution.
- •
Two smaller number tokens can always be combined into a valid new number token (e.g., 3 and 7 form 37), which is not true for general word tokens (e.g., “hello” and “hi” cannot form “hellohi”). This means the number of possible number tokens grows exponentially as $k$ increases, much faster than word tokens.
- •
The entropy in language generation is quite uneven. In autoregressive generation, once the context and a word’s prefix are given, the subsequent generation is often nearly deterministic. For example, in THIS paper, once “transform” appears, it is almost certain that “-er” will follow to form “transformer”. The phenomenon, known as “optimal viewing-position effect” (Brysbaert & Nazir, ) leads to similar difficulty for the model in generating the word “transformer” in one step or generating “transform” and “-er” in two steps, which encourages vocabulary expansion. However, for numerical tokens, the generation is more independent and the entropy distribution is more uniform. In an addition problem like 1234 + 5678 = 6…, knowing the first digit provides no useful information for predicting the subsequent digits. In this scenario, breaking down the tokens into smaller units reduces the complexity of each step significantly, whereas generating longer tokens in one step means solving several independent unit operations simultaneously, exponentially increasing the difficulty.
- •
The next token prediction of number tokens is harder than predicting the next word token because number prediction often involves calculation and operations, whereas word mapping tends to be more intuitive. Because of these differences between word tokens and number tokens, we wondered whether increasing the size of the number vocabulary would be effective in improving NUPA, as it is for language tasks.

Figure: (a) 0.9B model int add
Refer to caption: x3.png

We trained 0.9B models on 1- to 8-digit length samples including integer addition, float addition, and integer multiplication, using aligned $k$-digit tokenizers where $k=1,2,3$ (d in Figure 3).
Figure 4 shows the in-domain performance of these models in the first three columns and their out-of-domain (OOD) performance in the last two columns, evaluated using the exact match metric.

From the figure, the one-digit tokenizer shows best in-domain performance in these three tasks, while three-digit tokenizer exhibits poor performance. In out-of-domain tests, one-digit tokenizer even exceeds the others by large margins. Tokenizers with an increasing number of digits significantly hinder sub-billion models’ NUPA. We also performed experiments on models of 3 different sizes including 0.1B, 0.9B, and 3B in Appendix A.4.1 and got similar results. Even as the model size increases, the performance of 2- or 3- digit tokenizer improves but remains either similar or worse than that of the one-digit tokenizers. For these experiments, we also report the digit match and dlength results in Appendix A.4.1 Figure 10 and 11, where one-digit tokenizer performs better both on digit learning (larger digit match) and length learning (less dlength). In conclusion, we find no evidence to support the idea that increasing the vocabulary size improves NUPA performance. On the contrary, larger vocabularies significantly increase the model size requirements. Given that we used a nearly billion-parameter model just to learn integer addition, a larger tokenizer is unnecessary and may actually hinder performance.

Figure: Figure 5: Accuracy of 0.9B models trained with 1- to 3- digit tokenizers and 2- to 3- digit random tokenizers on integer addition. X-axis is the number of seen training samples.
Refer to caption: x6.png

##### Random tokenizer.

Additionally, we investigate the random tokenizer, as introduced by Sathe et al. () for language reasoning tasks.
They claim that this technique can enhance reasoning ability by providing different generation processes for the same reasoning path.
We apply this idea to the number domain, where the tokenizer segments a number into tokens with digit lengths randomly chosen between 1 and a predefined maximum, instead of greedily segmenting from left to right. Unlike GPT-2 tokenizer (Figure 3 (b)), random tokenizers can generate multiple tokenizations for the same number while the GPT-2 tokenizer cannot. We denote this random tokenizer with a maximum digit length of $n$ as the $n$-digit random tokenizer. Notably, the one-digit random tokenizer is equivalent to the one-digit tokenizer.

Figure 5 shows the performance of 1- to three-digit tokenizers alongside 2- to three-digit random tokenizers. In terms of in-domain generalization, the three-digit random tokenizer outperforms the three-digit standard tokenizer, while the two-digit random tokenizer shows a slight decline compared to its standard counterpart. We believe this is because the 0.9B model is capable of learning the two-digit tokenizer well, and the added perturbation from random tokenization acts as a form of regularization, introducing noise that slightly affects performance. The random tokenizers consistently outperform their standard counterparts in OOD generalization, indicating the regularization benefits in that aspect. In the case of the three-digit tokenizer, which is more challenging for a 0.9B model to learn, random tokenization generates smaller tokens, making the learning process easier and leading to improved in-domain performance.
However, they still fall short of the performance achieved by the one-digit tokenizer.

### 3.2 PEs and data formats to improve NUPA

In this section, we discuss the effects of using special PEs and data formats on NUPA. Previous work studying these tricks has primarily focused on one or two specific tasks such as integer addition. Moreover, the mechanisms behind the techniques remain an open question. In this paper, we extend the discussion of some of the techniques to more arithmetic tasks across different number domains and discuss their internal mechanisms. We categorize the techniques into three types: assisting computation, assisting alignment, and length regularization.

#### 3.2.1 Special PEs are length regularization

In most tasks, there is an intrinsic calculating rule not related to the length of input numbers, which we call the “length-agnostic” rule. However, because the model can only access a portion of the lengths during training, there can be a “length-related” rule that is not distinguishable by the model. For example, the model can learn a “length-related” addition rule that is the normal addition rules and “the output should be of length ranging from 1 to 8”. We name this phenomenon as length over-fitting. Previous work has shown that PE is a key factor affecting length generalization.
To further investigate how PEs influence the length generalization and whether the influence is specific on a certain task, we train 100M models with different PEs: RoPE (Su et al., ), NoPE (Kazemnejad et al., ) and Alibi (Press et al., ) on four tasks: integer addition, float addition, fraction multiplication (easy) and scientific notation addition respectively. Models are trained on 1-8 lengths (S and M range), then test them on full range (S to XL, 1-20). RoPE is the most classic relative
PE and Alibi is also a relative PE that claims to be more effective in length generalization, while NoPE denotes transformers without positional encoding (thus relying purely on the causal mask to use position information).

Figure: Table 2: RoPE performance.

We find that (1) RoPE encourages the model to rely on the length of the input. The first evidence is that RoPE causes the model’s predictive performance to plummet dramatically just beyond the training boundary. In “int-add” task, the exact match drops from nearly 100% to 0% when moving from 8 to 9 digits, while “dlength” rises from 0 to 0.67 (Table 2).
This indicates that the model has a significant probability of generating shorter results, avoiding the generation of more than 9-digit answers.
At the same time, RoPE not only constrains the model’s output length but also affects the digit pairing. The performance of 100% for inputs of 8 digits indicates that the model performs calculations for each position unless it can successfully align the corresponding digits. However, when the model encounters 9-digit inputs, digit match drops significantly to 50%, suggesting a considerable probability of failing to align the digits. Similar results on the other three tasks suggest that it is a task-agnostic behavior. The only exception is the digit match of scientific notation addition. We discuss the results in Appendix A.4.2.

Figure: Table 3: 8-digit digit-match accuracy with small model or small dataset.

(2) On the other hand, length learning provided by RoPE appears to be a shortcut. In cases where the model is extremely small or has been trained very little, we see the advantages of this “shortcut”. In Table 3, we train a 2-layer transformer (1.3M parameters) on integer addition using three different PEs on 1- to 8- digit integer addition or the 0.1B model with only 1M samples, we find RoPE shows the best in-domain performance.
However, in actual training, this shortcut, as a case-based learning mechanism for length, fails to achieve a generalization in length. In contrast, although NoPE and Alibi slow down learning, they enable the model to learn a more generalizable behavior. Experiments on the other three tasks are shown in Table 13 and Table 14 in Appendix A.4.2, where the RoPE always surpasses others. Therefore, our conclusion is that RoPE causes over-fitting on length shortcuts while these specially-designed PEs can be understood as length regularization.

#### 3.2.2 Data formats help digit alignment

**Table 4: Exact match of 0.1B models trained on integer addition and float addition respectively with various compositions of reverse formatting and zero padding.**
|  | Integer Addition | Float Addition |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | rev | rev<br>+pad | no | pad | rev total | rev total<br>+ pad | rev each | rev each<br>+ pad | rev dec | rev dec<br>+ pad | rev int | rev int<br>+ pad | no | pad |
| d9 | 1.00 | 1.00 | 1.00 | 1.00 | 0.12 | 0.24 | 0.12 | 0.24 | 0.12 | 0.24 | 1.00 | 1.00 | 1.00 | 1.00 |
| d10 | 0.80 | 0.92 | 0.32 | 0.82 | 0.10 | 0.22 | 0.12 | 0.23 | 0.09 | 0.22 | 0.98 | 0.98 | 0.22 | 0.98 |

A series of works focusing on length generalization of LLMs, especially integer addition, have proposed specific data formats. Typical techniques include reverse formatting, zero padding and index hints. Reverse formatting (Lee et al., ; Shen et al., ; Zhou et al., , ; Cho et al., ) presents numbers in reverse order from the least significant digit to the most significant one to align with the models’ autoregressive mechanism, simplifying the learning process for addition. Zero padding (Lee et al., ; Shen et al., ; Zhou et al., ; Cho et al., ) adds leading zeros to numbers to standardize the lengths of operands, helping models align operands. Index Hints (Zhou et al., ) explicitly include positional information in input and output sequences by representing each digit along with its position index prefixing the digit.

While previous work mainly focuses on integer addition or multiplication, we extend the techniques to various tasks in NUPA Test of different number domains for the first time.
To compare the effects of reverse formatting and zero padding, we demonstrate in Table 4 how the combination of reverse formatting and zero padding impacts length generalization. Reverse formatting, zero padding, and their combination all show better performance than vanilla formats in integer and float addition, while their performance is comparable to each other, suggesting that their functionality largely overlaps. Zero padding provides the function of helping alignment, and reverse formatting also plays a large role in alignment. The previously believed “helping calculation” function of reverse formatting is minor. Previous work (Zhou et al., ) believes that reverse formatting can help the calculation of each digit by aligning the calculation order to the right-to-left order that humans are accustomed to and solve the carrying problem. That is, from left-to-right, we cannot determine the result of the current digit unless the next digit results and whether there is a carrying have been known. However, a more detailed analysis can explain why the order is not as important as previously believed:

Regarding addition, the cases where reverse formatting can make a difference through the effects of assisting carry-over calculations are quite rare. Most of the time, knowing the result of the next digit allows us to determine the answer for the current digit. When the next digit addition is not less than 10 (without considering further carrying from the following digit), there must be a carrying from that digit into the current one, no matter what the result of the later digits is. And when the next digit addition is not more than 8, there will never be a carrying. The only exception is the next digit addition is $9$. In this situation, we must refer to the next two digits to determine the current digit results. Therefore, we point out that, although in the worst-case scenario, performing non-reversed addition requires $O(n)$-length looking forward for each digit ($44445+55556=100001$), and reversing could solve this problem, such cases are extremely rare.
In most instances, the task can be accomplished with a very limited local view.

As for index hint, we find it doesn’t work for our models, because it expands the vocabulary and thus causes an extra burden for models. On the contrary, reverse formatting or zero-padding raises the performance even with 0.1B models in multiple tasks. The experiments and discussion about index hint are included in appendix A.4.3. In conclusion, we find that the simplest “reverse” or “zero-padding” trick is effective for digit alignment and THIS alignment is the main reason why these data formats work.

### 3.3 Does finetuning improve NUPA performance of LLMs?

The existing techniques aimed at enhancing NUPA have rarely been applied to practical LLMs, mostly being tested on toy models and isolated tasks. This raises the question of whether it is possible to enhance the NUPA capabilities of large models through post-training finetuning. To explore this, we generate training sets ($10^{5}$ samples for each digit and each task) and validation sets for our NUPA tasks, ensuring no overlap with the original test set. We then used them to perform finetuning on a pre-trained model. Specifically,
we finetune a Meta-Llama-3.1-8B model with lora (rank 128, $\alpha$=32) on a mixed training set comprising all of our NUPA tasks. Remarkably, we find only 800 steps training (about 50M training samples, $\ll 1$ epoch) leads to significant improvement, as shown in Figure 2 with the finetuned model labeled as “Llama-8B-ft”. Though Llama-3.1-8B is not a strong baseline, this finetuned version achieves much better performance. For example, in max, max-hard, add-float and turediv tasks, this model even surpassed or matched GPT-4o, confirming our hypothesis: for many NUPA tasks, the model’s base capacity may not be the main limiting factor, but rather the lack of numerical diversity and task variety in the training data.

However, we also found that such finetuning does not provide much improvement on certain tasks, such as understanding digits.
Furthermore, when we tried to incorporate the various tricks, such as modifying the model’s original PEs, tokenizers, or number formats, into an already trained model, these methods proved ineffective. When we altered the PE or adjusted the tokenization and representation of the model, the changes significantly disrupted the model’s original behavior, causing a substantial performance drop. This suggests that enhancing a model’s NUPA capabilities through post-training may require more revolutionary innovations beyond the current tricks. The detailed results of these attempts are presented in Table 16 in Appendix A.4.4.

## 4 Is CoT suitable and valid for NUPA?

CoT has been proven to be effective to enhance the capacity of LLMs both theoretically (Feng et al., ; Yang et al., ) and experimentally (Wei et al., ; OpenAI, ). Thus, we are also interested in whether CoT is the ultimate solution for improving NUPA.
Due to the task and representation diversity in our benchmark, it is hard to cover all issues with a single form of CoT. So we adapt a special CoT form called Rule-Following CoT (Hu et al., ) (RF-CoT), where LLMs are trained to follow code or pseudo-code that outlines the procedure to solve the task. Since each step can be broken down into recurrences and basic unit operations, RF-CoT is capable of handling any problem that can be solved algorithmically, making it well-suited for our benchmark tasks.

**Table 5: Average inference time.**
|  | batchsize | sec / sample |
| --- | --- | --- |
| RF CoT | 128 | 5.625 |
| Direct | 128 | 0.371 |
| Direct | 256 | 0.336 |

**Table 6: Performance of RF CoT. “-” means exceeding context window limitation (2k tokens).**
| Exact Match | Add Float | Multiply Fraction | Max Scientific | Mod Integer |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Digit | 5 | 6 | 7 | 2 | 3 | 4 | 38 | 39 | 40 | 6 | 7 | 8 |
| RF CoT | 1.00 | 1.00 | - | 0.90 | 0.85 | - | 1.00 | 0.85 | 0.90 | 0.70 | 0.35 | - |
| GPT-4o | 0.78 | 0.66 | 0.49 | 0.53 | 0.20 | 0.00 | 0.37 | 0.46 | 0.36 | 0.01 | 0.00 | 0.00 |
| Qwen2-72B | 0.62 | 0.50 | 0.70 | 0.05 | 0.00 | 0.00 | 0.96 | 0.98 | 0.95 | 0.03 | 0.00 | 0.00 |
| Llama-3.1-8B-ft | 0.91 | 0.7 | 0.65 | 0.49 | 0.28 | 0.02 | 1.00 | 1.00 | 1.00 | 0.14 | 0.15 | 0.16 |

To evaluate the performance of the CoT method, we finetuned the LLaMA 3.1-8B model on a subset of the NUPA tasks, using rule-following finetuning (RFFT) (Hu et al., ). During both training and testing, we set a context window of 2000 tokens, with any data exceeding this limit being ignored. The performance on selected tasks is presented in Table 6. Within the context length limit, the rule-following finetuned LLaMA 3.1-8B significantly outperformed GPT-4o and Qwen2-72B as well as the one finetuned without RF-CoT in most situations.

At the same time, it requires a significantly longer context window and causes much slower inference speed compared to directly generating the answer. As shown in Table 6, with the 2000-token limit, CoT can only handle fraction addition involving numbers up to three digits. We provide the maximal digit length within the 2k context window limitation for each task in Appendix A.5 to show the context window limitation for complex tasks. As for inference time, Table 5 demonstrates the average inference time for generating each sample using “RF-CoT” and “direct answer” during the NUPA test where both experiments are operated on an A800 GPU. In the table, the “direct answer” with batch size $256$ shares a similar CUDA memory to RF-CoT with batch size $128$. The RF-CoT method is approximately 17 times slower than directly generating the answer, causing an unsustainable burden for such a basic operation that is frequently encountered in solving real-world problems, especially considering that number calculations may only account for a small part of a complex, real-world reasoning problem (such as analyzing a financial report).

## 5 Related work

In addition to the related work mentioned in the main text, our paper also includes some related works, which are:

##### Probing numerical understanding in natural language

Earlier studies (Wallace et al., ; Johnson et al., ) have explored LMs’ understanding of numbers presented in natural languages, such as English or Chinese. These studies employ probes to extract hidden states from LMs to evaluate their numerical comprehension, focusing on classic representations and tasks like adding two or three-digit numbers. In contrast, we primarily focus on Arabic numeral representations, as they are more prevalent in complex reasoning and support a wider variety of formats and tasks.

##### Number dataset in specific domains

Lin et al. () propose a dataset centered on questions involving numerical commonsense (e.g., how many legs bird have). Akhtar et al. () present a tabular dataset containing information such as duration, year and monetary amount.
Chen et al. () introduce a dataset focused on numerical reasoning with financial data, which Chen et al. () expand into a conversational format.
These works utilize Q&A probing methods to assess whether models can understand numerical meanings and perform sorting, estimation, and simple calculations. While these papers begin to address more complex and diverse numerical representations and tasks, they focus on specific numerical domains that diverge from pure numerical understanding and processing tasks. Therefore, this paper emphasizes tasks directly related to numbers themselves, rather than those limited to specific contexts. Furthermore, we opt to assess the model’s generative capabilities, as this aligns more closely with practical applications of LLMs.

##### Mathematical Reasoning Datasets

Numerous datasets now focus on mathematical reasoning, including GSM8k (Cobbe et al., ), MATH (Hendrycks et al., ), and SVAMP (Patel et al., ). These datasets consist of math word problems and primarily evaluate a model’s ability to analyze problems, identify mathematical tools, and organize problem-solving processes. Few studies distinguish between numerical processing capabilities and mathematical reasoning abilities. Saxton et al. () present preliminary work that introduces a dataset of numerical processing tasks, such as adding long floating-point numbers. However, it lacks categorization and analysis of task difficulty by length. Additionally, the dataset mixes in tasks like equation solving, algebraic calculations, and probability assessments, complicating our understanding of the model’s pure numerical capabilities.

## 6 Conclusion

We investigate NUPA of LLMs and introduce a comprehensive benchmark, the NUPA test, to reveal that numerical problems remain challenging for modern LLMs. Our comprehensive test suite, which includes a variety of numerical representations and tasks, has exposed the surprising vulnerability of LLMs in this fundamental area. To explore ways to improve NUPA, we extend and evaluate previous pre-training techniques on the NUPA benchmark. While direct finetuning on the NUPA tasks does improve the performance, using those tricks specifically designed for NUPA in the finetuning tend to harm the performance, suggesting that these methods are not easily transferable to practical LLMs. We also explore the potential of chain-of-thought techniques to enhance NUPA and discuss their limitations.

## 7 Limitation

As a benchmark that specifically focuses on number understanding and processing abilities, we acknowledge that the range of tasks could still be incomplete and biased towards certain aspects. We will continue updating our benchmark, including but not limited to adding new tasks and refining existing ones to ensure appropriate difficulty. Additionally, the number of models we have tested so far is limited, and we plan to include more promising pre-trained models in future evaluations.

On the other hand, although we have identified the limitations of LLMs’ NUPA, the existing solutions each have their own drawbacks. We have yet to find a path that fully addresses the problem. Solving this issue may require research across multiple fields, such as enhancing the diversity of pre-training corpora, developing new techniques, or enabling more efficient reasoning paradigms that make more complex CoT approaches feasible. We hope our work can contribute to and be complemented by advancements in these areas.

## Appendix A Appendix

### A.1 NUPA Test

#### A.1.1 Representations

We present the four representations as follows:

- •
Integer: we use no comma or point as a digit group separator like 1234567. The integer has only one part as itself. In this paper, we have not considered negative numbers for the time being.
- •
Float: A float has two parts: integer and decimal. We use a decimal point to split these two parts and also do not use any digit group separator. An example is 1234.567891. Trailing zeros in the decimal part are usually omitted.
- •
Fraction: A fraction has two parts: numerator and denominator and we use a “/” to separate the integer and decimal parts. Unless otherwise specified, all fractions mentioned in this paper are in their simplest form (that is the numerator and denominator are coprime), but they may be greater than 1. An example is 12/7. Only in the “truediv” task between two fractions, because the “/” is also the division operator, We enclose fractions in a pair of parentheses like (12/7) / (2/3) = 18/7 to make it clear.
- •
Scientific Notation: A scientific notation has two parts: significand and exponent. In our benchmark, the significand is always a float larger than 1 and less than 10 and the exponent should be a positive integer (and we also set an upper bound of 99). We use a “e” to seperate these two parts. An example is 1.5e8.

#### A.1.2 Examples for each task

We provide each tasks with an examples. To test the models, we also add some model specific system messages like “You are a helpful assistant to process numbers. Please directly answer the question after the =”. The context before “$=$” is the question and the context after “$=$” is the groundtruth and be removed when testing.

- •
Add-Integer: Add two numbers: $744+543=1287$
- •
Add-Float: Add two numbers: $93.81+9.976=103.786$
- •
Add-Fraction: Add two numbers: $3/8+2/5=31/40$
- •
Add-Scientific: Add two numbers: $9.92e16+9.731e18=9.8302e18$
- •
Sub-Integer: Subtract two numbers: $744-543=201$
- •
Sub-Float: Subtract two numbers: $93.81-9.976=83.834$
- •
Sub-Fraction: Subtract two numbers: $2/5-3/8=1/40$
- •
Sub-Scientific: Subtract two numbers: $9.731e38-9.92e36=9.6318e38$
- •
Multiply-Integer: Multiply two numbers: $968\times 8=7744$
- •
Multiply-Float: Multiply two numbers: $8.4\times 9.555=80.262$
- •
Multiply-Fraction: Multiply two numbers: $8/7\times 5/2=20/7$
- •
Multiply-Fraction: Multiply two numbers: $9.92e16\times 9.731e38=9.653152e55$
- •
Truediv-Integer: Divide two numbers and return the result as a fraction. $744\ /\ 543=248/181$
- •
Truediv-Fraction: Divide two numbers and return the result as a fraction. $(3/8)\ /\ (2/5)=15/16$
- •
Floordiv-Integer: Divide two numbers and return the result as an integer. $845\ //\ 152=5$
- •
Mod-Integer: Divide two numbers and return the remainder. $845\ \%\ 152=85$
- •
Max-Integer: Get the maximal number: $50404\text{ and }97871=97871$
- •
Max-Float: Get the maximal number: $44.418\text{ and }65.669=65.669$
- •
Max-Fraction: Get the maximal number: $3/5\text{ and }3/8=3/5$
- •
Max-Scientific: Get the maximal number: $8.15e64\text{ and }1.063e73=1.063e73$
- •
Digit_max-Integer: Compare two numbers digit by digit and return the larger digit at each position, treating any missing digits as 0. $50194\text{ and }14283=54294$
- •
Digit_max-Float: Compare two numbers digit by digit and return the larger digit at each position, treating any missing digits as 0. $35.905\text{ and }8.4=38.905$
- •
Digit_add-Integer: The task is to add two given numbers digit by digit and return the result modulo 10 (ignoring carry), treating any missing digits as 0. $50404\text{ digit add }97871=47275$
- •
Digit_add-Float: The task is to add two given numbers digit by digit and return the result modulo 10 (ignoring carry), treating any missing digits as 0. $44.418\text{ digit add }65.669=9.077$
- •
Get_digit-Integer: Get the digit at the given position (from left to right, starting from 0). $50404\text{ at position }4=4$
- •
Get_digit-Float: Get the digit at the given position (from left to right, starting from 0). $44.418\text{ at position }3=1$
- •
Length-Integer: The total number of digits of $50404=5$
- •
Length-Float: The total number of digits of $262.534=6$
- •
Count-Integer: Count the number of the given digit in the given number: $27422\text{ count the occurance time of digit }2=3$
- •
To_float-Fraction: Convert the number to float: $9/5=1.8$
- •
To_float-Scientific: Convert the number to float: $8.538e2=853.8$
- •
To_scientific-Integer: Convert the number to scientific notation: $50400=5.04e4$
- •
To_scientific-Float: Convert the number to scientific notation: $262.534=2.62534e2$
- •
Sig.Fig-Integer: Convert the number to scientific notation: $50194$ and keep significant figures as $3=5.02e4$
- •
Sig.Fig-Float: Convert the number to scientific notation: $65.669$ and keep significant figures as $2=6.6e1$

#### A.1.3 Expected representation in each task

Each task in the 41 ones receives one or two input numbers and expects one number as the result. We name the representation by the first input numbers. For simplicity, the second input number shares the same representation as the first one for most tasks. Calculations between different representations can be performed by first converting them to the same representation. Two types of tasks are the exception. Tasks “length”, “to float” and “to scientific” do not have the second input. The second inputs in tasks “get digit”, “count”, “sig. fig.” are always a short Integer, representing a position, length, or a digit number from 0 to 9. To distinguish them from potentially long integers to be processed, we call the former int and the latter integer.

We summary the second number representation and result representation in each tasks in Table 7 and Table 8 where I means integer, i means (shorter) int, Fl means float, Fr means fraction, S means scientific notation and N means no such a number.

**Table 7: The second input number representation**
|  | Elementary arithmetic | Comparison | Digit Understanding | Conversion |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Add | Sub | Multiply | Truediv | Floordiv | Mod | Max | Min | Digit<br>Max | Digit<br>Min | Digit<br>Add | Get<br>Digit | Length | Count | To<br>Float | To<br>Scientific | Sig.<br>Fig. |
| Integer | I | I | I | I | I | I | I | I | I | I | I | i | N | i | $-$ | N | i |
| Float | Fl | Fl | Fl | ✗ | $-$ | $-$ | Fl | Fl | Fl | Fl | Fl | i | N | $\bigcirc$ | $-$ | N | i |
| Fraction | Fr | Fr | Fr | Fr | $-$ | $-$ | Fr | Fr | $-$ | $-$ | $-$ | $-$ | $-$ | $\bigcirc$ | N | $\bigcirc$ | $\bigcirc$ |
| Scientific | S | S | S | ✗ | $-$ | $-$ | S | S | $-$ | $-$ | $-$ | $-$ | $-$ | $\bigcirc$ | N | $-$ | $\bigcirc$ |

**Table 8: Result number representation**
|  | Elementary arithmetic | Comparison | Digit Understanding | Conversion |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Add | Sub | Multiply | Truediv | Floordiv | Mod | Max | Min | Digit<br>Max | Digit<br>Min | Digit<br>Add | Get<br>Digit | Length | Count | To<br>Float | To<br>Scientific | Sig.<br>Fig. |
| Integer | I | I | I | Fr | I | I | I | I | I | I | I | i | i | i | $-$ | S | S |
| Float | Fl | Fl | Fl | ✗ | $-$ | $-$ | Fl | Fl | Fl | Fl | Fl | i | i | $\bigcirc$ | $-$ | S | S |
| Fraction | Fr | Fr | Fr | Fr | $-$ | $-$ | Fr | Fr | $-$ | $-$ | $-$ | $-$ | $-$ | $\bigcirc$ | Fl | $\bigcirc$ | $\bigcirc$ |
| Scientific | S | S | S | ✗ | $-$ | $-$ | S | S | $-$ | $-$ | $-$ | $-$ | $-$ | $\bigcirc$ | Fl | $-$ | $\bigcirc$ |

#### A.1.4 Non-included tasks

We exclude some compositions between number representations and tasks because of the following three reasons:

- •
✗ too complex. We exclude the truediv between float and scientific. Division between float numbers is difficult to define accurately in our scenario. It is very common to divide two floating point numbers into an infinite decimal, which means that even very short decimals can still result in a very long and unpredictable result after division. And in this task we do not want to discuss the case of rounding the result. (This is another task of ours.) For the same reason, we also exclude division in scientific notation.
- •
$\bigcirc$: can be easily transferred to from an included task.
–
Converting fractions to scientific notation can be done by first converting to a float. (Fraction-to_scientific = Fraction-to_float + Float-to_scientific). Fraction-SiginificantFigure is similar.
–
Scientific notation retains significant digits and is virtually identical to floating point numbers.
–
count is a special task where we just consider a nubmer as “a set of digits” so count in a float, fraction and scientific notation is as the same as in a integer.
- •
$-$: not applicable.
–
In fraction and scientific notation, the digit concept is not well-defined so the tasks about digit (digit-compare, digit-add, get-digit and length) are not applicable.
–
Floordiv and mod is only defined on integer.
–
Integer and float do not need to be further converted to float. Similarly, scientific has no need to converted to scientific.

#### A.1.5 Easy/hard split of NUPA tasks

We divide the tasks into easy and hard as shown in Table 9, where the hard tasks marked as H with maximal test digit as 20 and the easy tasks marked as E with maximal test digit as 100.

**Table 9: Tasks can be divided into Easy and Hard.**
|  | Elementary arithmetic | Comparison | Digit Understanding | Conversion |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Add | Sub | Multiply | Truediv | Floordiv | Mod | Max | Min | Digit<br>Max | Digit<br>Min | Digit<br>Add | Get<br>Digit | Length | Count | To<br>Float | To<br>Scientific | Sig.<br>Fig. |
| Integer | H | H | H | H | H | H | E | E | E | E | E | E | E | E |  | E | E |
| Float | H | H | H |  |  |  | E | E | E | E | E | E | E |  |  | E | E |
| Fraction | H | H | H | H |  |  | H | H |  |  |  |  |  |  | H |  |  |
| Scientific | H | H | H |  |  |  | E | E |  |  |  |  |  |  | E |  |  |

#### A.1.6 Preprocess and question generation for NUPA tasks

We define the length of a number as the number of digits in the longest part of a number. The “integer” part and “decimal” part of a float (as well as the significand of a scientific notation), the “numerator” and “denominator” of a fraction, the “exponent” of a scientific notation are considered as different “parts”. In order to generate a pair of numbers with the larger length $L$, we first generate a $L$-length number and then generate a $l$-length number where $l$ follows a uniform distribution from $L/2$ to $L$. If the operation is commutative, we swap the two numbers with probability 0.5.

After we select two random numbers, we have some pre-procession to generate the final questions:

- •
For “Multiply”, the difficulty also affected by the shorter number severely, so we split the task into two sub-tasks as “Multiply-hard” and “multiply-easy”. For the hard subset, we require that the shorter number must be longer than half of the longer one. For an easy subset, we require that the length of the shorter number is less than 3, so that the complexity is $O(n)$ instead of $O(n^{2})$. And because the addition of fractions also involves multiplication, we also add an add-easy for this task in the same way.
- •
For “max” and “min” tasks, we additionally provide a harder version. For Integers and floats, we make sure that two compared numbers share the same length. At the same time, they should have more digits as the same like $12949$ and $12961$ to avoid models that can solve the problem by only counting the length or comparing the first digit.
For scientific notation, we ensure 70% pairs of compared numbers with the same exponential part so that models cannot directly get the answer without comparing the significand part. For fractions, we ensure the numbers are both less than one, avoiding the model can just compare them with 1 to get more than 50% accuracy.
- •
For “to_float-Fraction”, we require the fraction must can be convert into a finite decimal, that is the denominator contains only factors 2 and 5.
- •
For “add/sub-Scientific”, we require the exponential part of each number to have a difference less than 5 to make sure that the generated answer will not be too long.

The pre-procession could introduce additional duplicated data. we facilitate a post-filtering of duplication and decontainment

#### A.1.7 Metrics

For digit match, we should first align the numbers. For the integers and integer parts in floats, the numerator and denominator of fractions, and the exponential part of the scientific notation, we use the right alignment. For the decimal part in floats (as well as the in the significand part in scientific notation), we use the left alignment.

For dlength, we first measure the difference of each part of a number and then add the absolute values up.

Besides the average metrics in each range, we also present the following metrics: well-learned digits and performance-preserving digits to demonstrate the model’s upper and lower performance limits on length. These represent the maximum number of digits that can maintain over 90% and 10% accuracy, respectively. (For digit match, the thresholds are set to 90% and 50%, and for dlength, where smaller is better, the thresholds are 0.1 and 1).

We ensure that there is no duplicated sample in dataset, so for some range, the test samples could be less than 1000. We also omit 1 digit or some 2 digit test in our testbed to make sure that unit rules can be included in a training set.

### A.2 Prompts and other details to test baseline models

For all models in our test, we first provide a “format prompt” describing the expected return format (and avoiding models generating complex CoT), and a “task prompt” describing the task. We use some easy problems to ensure powerful models (gpt-4o-mini and Llama-3.1-8B) can correctly understand the tasks and expected return format by the prompts. The expected return representation of each task is referred to in Appendix A.1.3.

The format prompt based on the expected return type of the task is as follows:

- •
Integer: Directly return the answer as an integer without any comma separator, like 123 .
- •
float: Directly return the answer as a float without any comma separator, like 10.4 .
- •
Fraction: Directly return the answer as an **irreducible** fraction without any comma separator, like 7/13 .
- •
Scientific Notation: Directly return the answer as a scientific notation without any comma separator, like 1.23e4 . The float part should be in the range [1, 10).

The task prompts are listed as follows where <a> and <b> are numbers.

- •
Add: Add two numbers: <a> + <b> =
- •
Sub: Subtract two numbers: <a> - <b> =
- •
Multiply: Multiply two numbers: <a> * <b> =
- •
Truediv: Divide two numbers and return the result as a fraction. <a> / <b> =
- •
Floordiv: Divide two numbers and return the result as an integer. <a> // <b> =
- •
Mod: Divide two numbers and return the remainder. <a> % <b>
- •
Max: Get the maximal number: <a> and <b> =
- •
Min: Get the maximal number: <a> and <b> =
- •
Digit max: Compare two numbers digit by digit and return the larger digit at each position, treating any missing digits as 0. <a> and <b> =
- •
Digit min: Compare two numbers digit by digit and return the smaller digit at each position, treating any missing digits as 0. <a> and <b> =
- •
Digit add: The task is to add two given numbers digit by digit and return the result modulo 10 (ignoring carry), treating any missing digits as 0. <a> digit add <b> =
- •
Get digit: Get the digit at the given position (from left to right, starting from 0). <a> at position <b> =
- •
Length: The total number of digits of <a> =
- •
Count: Count the number of the given digit in the given number: <a> count the occurrence time of digit <b> =
- •
To_float: Convert the number to float: <a> =
- •
To_scient: Convert the number to scientific notation: <a> =
- •
Sig_fig: Convert the number to scientific notation: <a> and keep significant figures as <b>.

Notice that all prompts are ended with an “=” so that we can easily separate the input question and the generation of models. When we use the texts in supervised finetuning (SFT), the context before the “=” is not involved in the loss calculation.

For GPT-4o and GPT-4o-mini, we also add a system message as follows and use the aforementioned question as user message:

> You are a capable math assistant. Return your solution without any process in the format: The answer is [YOUR ANSWER]. The final answer must strictly match the format <regex>.

where the <regex> is a regular expression based on the expected return format:

- •
Integer: r"\d+"
- •
Float: r"\d+\.\d+"
- •
Fraction: r"\d+/\d+"
- •
Scientific Notation: r"\d+\.\d+e\d+"

We use the models expect GPT from huggingface and use the default tokenizer, model and generation configuration provided by the models. We test GPT-4o and GPT-4o-mini by the OpenAI API, where GPT-4o means gpt-4o-2024-0806 and GPT-4o-mini means GPT-4o-mini-2024-07-18. For Qwen2-72B and Llama-3.1-70B, we additionally use 4-bit quantization but we also test several samples without quantization and ensure this quantization does not affect generation quality.

We retrieve the first match of the corresponding regular expression after the “=” as the answer. If there is no retrieve, we use an empty answer to calculate the metrics, where exact match and digit match is both zero and the dlength is the total length of the groundtruth number.

### A.3 Full test results of LLMs

We show the full NUPA test results in Figures 6 (exact match), 7 (digit match), 8 (dlength) and Table 10, 11, 12 (well-learned digits and performance-preserving digits for each metrics).

With the detailed metrics, we can more clearly understand the behavior of some models on some tasks. For example, we find that the “exact match” and “digit match” of some models like Qwen-2 and GPT-4o on the “integer-max” task are similar, suggesting that when the models know which one is correct, they can always copy the answer from question correctly. So the wrong answer comes from incorrect comparison. Another example is the Llama-2 performance on max-hard. Because the length of two input numbers and the groundtruth answer in the max-hard task are all the same, most models show less dlength on this task suggesting they know that “the answer should have the same length of inputs”, but we find Llama-2 shows dlength approximately equal to the average length in the range, suggesting that Llama-2 cannot generate a valid answer on this task. These are just a few examples to illustrate how more detailed metrics can help us gain a deeper understanding of model behavior. There are many possible conclusions, but there are too many to list here.

Figure: Figure 6: Exact match of models tested on NUPA Test.
Refer to caption: x7.png

Figure: Figure 7: Digit match of models tested on NUPA Test.
Refer to caption: x12.png

Figure: Figure 8: Dlength of models tested on NUPA Test. Note that we use $\boldmath{\log_{2}}(\textbf{\text{dlength}}+1)$ as the ylabel in the figure.
Refer to caption: x17.png

**Table 10: Well-learned digits / performance-preserving digits of models tested on NUPA Test according to exact match.**
|  | Add | Add | Add | Add Easy | Add | Sub | Sub | Sub |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Int | Float | Frac | Frac | Sci | Int | Float | Frac |  |
| GPT-4o-mini | 5 / 20 | 4 / 11 | 0 / 1 | 0 / 2 | 0 / 0 | 6 / 20 | 5 / 15 | 0 / 1 |  |
| GPT-4o | 5 / 20 | 4 / 11 | 0 / 1 | 0 / 1 | 0 / 0 | 6 / 20 | 4 / 15 | 0 / 1 |  |
| Qwen2-72B | 6 / 20 | 0 / 15 | 0 / 1 | 0 / 1 | 0 / 11 | 6 / 20 | 0 / 15 | 0 / 1 |  |
| Qwen2-7B | 4 / 14 | 0 / 15 | 0 / 1 | 0 / 0 | 0 / 7 | 4 / 14 | 0 / 13 | 0 / 1 |  |
| Llama-3.1-8B-ft | 4 / 12 | 5 / 17 | 0 / 1 | 0 / 1 | 0 / 4 | 4 / 12 | 4 / 17 | 0 / 2 |  |
| Llama-3.1-70B | 4 / 15 | 0 / 11 | 0 / 1 | 0 / 1 | 0 / 0 | 6 / 11 | 4 / 11 | 0 / 1 |  |
| Llama-3.1-8B | 4 / 9 | 3 / 11 | 0 / 1 | 0 / 0 | 0 / 0 | 3 / 10 | 3 / 10 | 0 / 1 |  |
| Mixtral-8x7B | 5 / 10 | 4 / 11 | 0 / 1 | 0 / 1 | 0 / 6 | 4 / 15 | 3 / 11 | 0 / 1 |  |
| Llama-2-7b-hf | 0 / 6 | 0 / 7 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 6 | 0 / 5 | 0 / 1 |  |
|  | Sub | Max | Max | Max | Max | Max Hard | Max Hard | Max Hard |  |
|  | Sci | Int | Float | Frac | Sci | Int | Float | Sci |  |
| GPT-4o-mini | 5 / 20 | 4 / 11 | 0 / 1 | 0 / 2 | 0 / 0 | 6 / 20 | 5 / 15 | 0 / 1 |  |
| GPT-4o | 5 / 20 | 4 / 11 | 0 / 1 | 0 / 1 | 0 / 0 | 6 / 20 | 4 / 15 | 0 / 1 |  |
| Qwen2-72B | 6 / 20 | 0 / 15 | 0 / 1 | 0 / 1 | 0 / 11 | 6 / 20 | 0 / 15 | 0 / 1 |  |
| Qwen2-7B | 4 / 14 | 0 / 15 | 0 / 1 | 0 / 0 | 0 / 7 | 4 / 14 | 0 / 13 | 0 / 1 |  |
| Llama-3.1-8B-ft | 4 / 12 | 5 / 17 | 0 / 1 | 0 / 1 | 0 / 4 | 4 / 12 | 4 / 17 | 0 / 2 |  |
| Llama-3.1-70B | 4 / 15 | 0 / 11 | 0 / 1 | 0 / 1 | 0 / 0 | 6 / 11 | 4 / 11 | 0 / 1 |  |
| Llama-3.1-8B | 4 / 9 | 3 / 11 | 0 / 1 | 0 / 0 | 0 / 0 | 3 / 10 | 3 / 10 | 0 / 1 |  |
| Mixtral-8x7B | 5 / 10 | 4 / 11 | 0 / 1 | 0 / 1 | 0 / 6 | 4 / 15 | 3 / 11 | 0 / 1 |  |
| Llama-2-7b-hf | 0 / 6 | 0 / 7 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 6 | 0 / 5 | 0 / 1 |  |
|  | Multiply Hard | Multiply Hard | Multiply Hard | Multiply Hard | Multiply Easy | Multiply Easy | Multiply Easy | Multiply Easy |  |
|  | Int | Float | Frac | Sci | Int | Float | Frac | Sci |  |
| GPT-4o-mini | 5 / 20 | 4 / 11 | 0 / 1 | 0 / 2 | 0 / 0 | 6 / 20 | 5 / 15 | 0 / 1 |  |
| GPT-4o | 5 / 20 | 4 / 11 | 0 / 1 | 0 / 1 | 0 / 0 | 6 / 20 | 4 / 15 | 0 / 1 |  |
| Qwen2-72B | 6 / 20 | 0 / 15 | 0 / 1 | 0 / 1 | 0 / 11 | 6 / 20 | 0 / 15 | 0 / 1 |  |
| Qwen2-7B | 4 / 14 | 0 / 15 | 0 / 1 | 0 / 0 | 0 / 7 | 4 / 14 | 0 / 13 | 0 / 1 |  |
| Llama-3.1-8B-ft | 4 / 12 | 5 / 17 | 0 / 1 | 0 / 1 | 0 / 4 | 4 / 12 | 4 / 17 | 0 / 2 |  |
| Llama-3.1-70B | 4 / 15 | 0 / 11 | 0 / 1 | 0 / 1 | 0 / 0 | 6 / 11 | 4 / 11 | 0 / 1 |  |
| Llama-3.1-8B | 4 / 9 | 3 / 11 | 0 / 1 | 0 / 0 | 0 / 0 | 3 / 10 | 3 / 10 | 0 / 1 |  |
| Mixtral-8x7B | 5 / 10 | 4 / 11 | 0 / 1 | 0 / 1 | 0 / 6 | 4 / 15 | 3 / 11 | 0 / 1 |  |
| Llama-2-7b-hf | 0 / 6 | 0 / 7 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 6 | 0 / 5 | 0 / 1 |  |
|  | Digit Max | Digit Max | Digit Add | Digit Add | Get Digit | Get Digit | Length | Length |  |
|  | Int | Float | Int | Float | Int | Float | Int | Float |  |
| GPT-4o-mini | 5 / 20 | 4 / 11 | 0 / 1 | 0 / 2 | 0 / 0 | 6 / 20 | 5 / 15 | 0 / 1 |  |
| GPT-4o | 5 / 20 | 4 / 11 | 0 / 1 | 0 / 1 | 0 / 0 | 6 / 20 | 4 / 15 | 0 / 1 |  |
| Qwen2-72B | 6 / 20 | 0 / 15 | 0 / 1 | 0 / 1 | 0 / 11 | 6 / 20 | 0 / 15 | 0 / 1 |  |
| Qwen2-7B | 4 / 14 | 0 / 15 | 0 / 1 | 0 / 0 | 0 / 7 | 4 / 14 | 0 / 13 | 0 / 1 |  |
| Llama-3.1-8B-ft | 4 / 12 | 5 / 17 | 0 / 1 | 0 / 1 | 0 / 4 | 4 / 12 | 4 / 17 | 0 / 2 |  |
| Llama-3.1-70B | 4 / 15 | 0 / 11 | 0 / 1 | 0 / 1 | 0 / 0 | 6 / 11 | 4 / 11 | 0 / 1 |  |
| Llama-3.1-8B | 4 / 9 | 3 / 11 | 0 / 1 | 0 / 0 | 0 / 0 | 3 / 10 | 3 / 10 | 0 / 1 |  |
| Mixtral-8x7B | 5 / 10 | 4 / 11 | 0 / 1 | 0 / 1 | 0 / 6 | 4 / 15 | 3 / 11 | 0 / 1 |  |
| Llama-2-7b-hf | 0 / 6 | 0 / 7 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 6 | 0 / 5 | 0 / 1 |  |
|  | Truediv | Truediv | Floordiv | Mod | Mod Easy | To Float | To Float | To Scient |  |
|  | Int | Frac | Int | Int | Int | Frac | Sci | Int |  |
| GPT-4o-mini | 5 / 20 | 4 / 11 | 0 / 1 | 0 / 2 | 0 / 0 | 6 / 20 | 5 / 15 | 0 / 1 |  |
| GPT-4o | 5 / 20 | 4 / 11 | 0 / 1 | 0 / 1 | 0 / 0 | 6 / 20 | 4 / 15 | 0 / 1 |  |
| Qwen2-72B | 6 / 20 | 0 / 15 | 0 / 1 | 0 / 1 | 0 / 11 | 6 / 20 | 0 / 15 | 0 / 1 |  |
| Qwen2-7B | 4 / 14 | 0 / 15 | 0 / 1 | 0 / 0 | 0 / 7 | 4 / 14 | 0 / 13 | 0 / 1 |  |
| Llama-3.1-8B-ft | 4 / 12 | 5 / 17 | 0 / 1 | 0 / 1 | 0 / 4 | 4 / 12 | 4 / 17 | 0 / 2 |  |
| Llama-3.1-70B | 4 / 15 | 0 / 11 | 0 / 1 | 0 / 1 | 0 / 0 | 6 / 11 | 4 / 11 | 0 / 1 |  |
| Llama-3.1-8B | 4 / 9 | 3 / 11 | 0 / 1 | 0 / 0 | 0 / 0 | 3 / 10 | 3 / 10 | 0 / 1 |  |
| Mixtral-8x7B | 5 / 10 | 4 / 11 | 0 / 1 | 0 / 1 | 0 / 6 | 4 / 15 | 3 / 11 | 0 / 1 |  |
| Llama-2-7b-hf | 0 / 6 | 0 / 7 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 6 | 0 / 5 | 0 / 1 |  |
|  | To Scient | Count | Sig |  |  |  |  |  |  |
|  | Float | Int | Int |  |  |  |  |  |  |
| GPT-4o-mini | 5 / 20 | 4 / 11 | 0 / 1 |  |  |  |  |  |  |
| GPT-4o | 5 / 20 | 4 / 11 | 0 / 1 |  |  |  |  |  |  |
| Qwen2-72B | 6 / 20 | 0 / 15 | 0 / 1 |  |  |  |  |  |  |
| Qwen2-7B | 4 / 14 | 0 / 15 | 0 / 1 |  |  |  |  |  |  |
| Llama-3.1-8B-ft | 4 / 12 | 5 / 17 | 0 / 1 |  |  |  |  |  |  |
| Llama-3.1-70B | 4 / 15 | 0 / 11 | 0 / 1 |  |  |  |  |  |  |
| Llama-3.1-8B | 4 / 9 | 3 / 11 | 0 / 1 |  |  |  |  |  |  |
| Mixtral-8x7B | 5 / 10 | 4 / 11 | 0 / 1 |  |  |  |  |  |  |
| Llama-2-7b-hf | 0 / 6 | 0 / 7 | 0 / 0 |  |  |  |  |  |  |

**Table 11: Well-learned digits / performance-preserving digits of models tested on NUPA Test according to digit match.**
|  | Add | Add | Add | Add Easy | Add | Sub | Sub | Sub |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Int | Float | Frac | Frac | Sci | Int | Float | Frac |  |
| GPT-4o-mini | 9 / 20 | 6 / 20 | 0 / 2 | 0 / 2 | 0 / 14 | 11 / 20 | 9 / 20 | 0 / 1 |  |
| GPT-4o | 10 / 20 | 6 / 20 | 0 / 1 | 0 / 2 | 0 / 14 | 11 / 20 | 10 / 20 | 0 / 1 |  |
| Qwen2-72B | 7 / 16 | 7 / 20 | 0 / 0 | 0 / 0 | 0 / 20 | 6 / 15 | 0 / 16 | 0 / 0 |  |
| Qwen2-7B | 4 / 12 | 3 / 20 | 0 / 0 | 0 / 0 | 0 / 0 | 6 / 14 | 6 / 20 | 0 / 0 |  |
| Llama-3.1-8B-ft | 6 / 16 | 9 / 20 | 0 / 1 | 0 / 0 | 0 / 7 | 6 / 19 | 12 / 20 | 0 / 1 |  |
| Llama-3.1-70B | 6 / 15 | 7 / 20 | 0 / 1 | 0 / 1 | 0 / 4 | 6 / 17 | 7 / 20 | 0 / 1 |  |
| Llama-3.1-8B | 4 / 9 | 6 / 18 | 0 / 0 | 0 / 0 | 0 / 0 | 6 / 11 | 6 / 17 | 0 / 0 |  |
| Mixtral-8x7B | 6 / 12 | 5 / 18 | 0 / 1 | 0 / 1 | 0 / 6 | 5 / 15 | 5 / 18 | 0 / 0 |  |
| Llama-2-7b-hf | 3 / 6 | 0 / 8 | 0 / 0 | 0 / 0 | 0 / 0 | 3 / 5 | 0 / 9 | 0 / 0 |  |
|  | Sub | Max | Max | Max | Max | Max Hard | Max Hard | Max Hard |  |
|  | Sci | Int | Float | Frac | Sci | Int | Float | Sci |  |
| GPT-4o-mini | 0 / 3 | 100 / 100 | 10 / 100 | 0 / 7 | 19 / 98 | 100 / 100 | 8 / 100 | 18 / 100 |  |
| GPT-4o | 0 / 0 | 100 / 100 | 10 / 100 | 0 / 7 | 19 / 98 | 100 / 100 | 8 / 100 | 16 / 100 |  |
| Qwen2-72B | 0 / 0 | 21 / 100 | 30 / 100 | 0 / 4 | 100 / 100 | 82 / 100 | 32 / 100 | 100 / 100 |  |
| Qwen2-7B | 0 / 0 | 11 / 86 | 10 / 98 | 0 / 0 | 0 / 82 | 13 / 100 | 7 / 100 | 6 / 69 |  |
| Llama-3.1-8B-ft | 0 / 0 | 83 / 100 | 75 / 100 | 0 / 20 | 100 / 100 | 100 / 100 | 79 / 100 | 100 / 100 |  |
| Llama-3.1-70B | 0 / 0 | 0 / 86 | 0 / 98 | 0 / 4 | 0 / 0 | 0 / 0 | 0 / 100 | 0 / 100 |  |
| Llama-3.1-8B | 0 / 0 | 0 / 67 | 0 / 93 | 0 / 0 | 0 / 54 | 0 / 100 | 0 / 96 | 0 / 100 |  |
| Mixtral-8x7B | 0 / 0 | 7 / 21 | 3 / 19 | 0 / 0 | 0 / 62 | 20 / 100 | 4 / 25 | 0 / 100 |  |
| Llama-2-7b-hf | 0 / 0 | 0 / 100 | 0 / 22 | 0 / 0 | 0 / 17 | 99 / 100 | 0 / 38 | 0 / 100 |  |
|  | Multiply Hard | Multiply Hard | Multiply Hard | Multiply Hard | Multiply Easy | Multiply Easy | Multiply Easy | Multiply Easy |  |
|  | Int | Float | Frac | Sci | Int | Float | Frac | Sci |  |
| GPT-4o-mini | 0 / 5 | 0 / 0 | 1 / 2 | 0 / 4 | 0 / 6 | 0 / 0 | 1 / 2 | 0 / 4 |  |
| GPT-4o | 0 / 5 | 0 / 0 | 1 / 2 | 0 / 4 | 0 / 6 | 0 / 0 | 1 / 3 | 0 / 4 |  |
| Qwen2-72B | 0 / 5 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 6 | 0 / 0 | 0 / 0 | 0 / 0 |  |
| Qwen2-7B | 0 / 3 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 4 | 0 / 0 | 0 / 0 | 0 / 0 |  |
| Llama-3.1-8B-ft | 0 / 4 | 0 / 0 | 0 / 3 | 0 / 4 | 0 / 6 | 0 / 3 | 1 / 3 | 0 / 5 |  |
| Llama-3.1-70B | 0 / 5 | 0 / 3 | 0 / 2 | 0 / 3 | 0 / 6 | 0 / 3 | 0 / 2 | 0 / 3 |  |
| Llama-3.1-8B | 0 / 4 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 5 | 0 / 0 | 0 / 1 | 0 / 0 |  |
| Mixtral-8x7B | 0 / 4 | 0 / 0 | 0 / 1 | 0 / 0 | 0 / 5 | 0 / 0 | 0 / 2 | 0 / 0 |  |
| Llama-2-7b-hf | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |  |
|  | Digit Max | Digit Max | Digit Add | Digit Add | Truediv | Truediv | Floordiv | Mod |  |
|  | Int | Float | Int | Float | Int | Frac | Int | Int |  |
| GPT-4o-mini | 0 / 100 | 0 / 100 | 0 / 0 | 0 / 0 | 0 / 5 | 1 / 2 | 5 / 15 | 0 / 3 |  |
| GPT-4o | 0 / 100 | 0 / 100 | 0 / 0 | 0 / 0 | 0 / 6 | 1 / 2 | 5 / 15 | 0 / 3 |  |
| Qwen2-72B | 0 / 100 | 0 / 100 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 1 | 3 / 8 | 0 / 3 |  |
| Qwen2-7B | 0 / 100 | 0 / 33 | 0 / 5 | 0 / 8 | 0 / 0 | 0 / 1 | 3 / 6 | 0 / 3 |  |
| Llama-3.1-8B-ft | 0 / 100 | 13 / 100 | 0 / 20 | 5 / 73 | 3 / 20 | 0 / 1 | 6 / 18 | 0 / 3 |  |
| Llama-3.1-70B | 0 / 100 | 0 / 100 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 1 | 4 / 12 | 0 / 0 |  |
| Llama-3.1-8B | 0 / 92 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 3 / 7 | 0 / 0 |  |
| Mixtral-8x7B | 0 / 100 | 0 / 97 | 0 / 5 | 0 / 5 | 0 / 0 | 0 / 1 | 3 / 7 | 0 / 3 |  |
| Llama-2-7b-hf | 0 / 100 | 0 / 42 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 3 | 0 / 0 |  |
|  | Mod Easy | To Float | To Float | To Scient | To Scient | Sig |  |  |  |
|  | Int | Frac | Sci | Int | Float | Int |  |  |  |
| GPT-4o-mini | 0 / 3 | 0 / 8 | 0 / 9 | 23 / 67 | 8 / 24 | 31 / 100 |  |  |  |
| GPT-4o | 0 / 3 | 0 / 8 | 0 / 9 | 23 / 71 | 8 / 28 | 31 / 100 |  |  |  |
| Qwen2-72B | 0 / 3 | 6 / 9 | 16 / 28 | 100 / 100 | 96 / 100 | 100 / 100 |  |  |  |
| Qwen2-7B | 0 / 3 | 4 / 8 | 6 / 22 | 100 / 100 | 0 / 100 | 19 / 100 |  |  |  |
| Llama-3.1-8B-ft | 0 / 3 | 3 / 8 | 10 / 36 | 95 / 100 | 25 / 83 | 19 / 100 |  |  |  |
| Llama-3.1-70B | 0 / 0 | 4 / 9 | 0 / 12 | 0 / 100 | 0 / 21 | 0 / 21 |  |  |  |
| Llama-3.1-8B | 0 / 0 | 3 / 6 | 0 / 12 | 0 / 0 | 0 / 0 | 0 / 0 |  |  |  |
| Mixtral-8x7B | 0 / 3 | 4 / 9 | 14 / 36 | 100 / 100 | 68 / 100 | 17 / 100 |  |  |  |
| Llama-2-7b-hf | 0 / 0 | 0 / 5 | 0 / 0 | 0 / 34 | 0 / 22 | 0 / 0 |  |  |  |

**Table 12: Well-learned digits / performance-preserving digits of models tested on NUPA Test according to dlength.**
|  | Add | Add | Add | Add Easy | Add | Sub | Sub | Sub |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Int | Float | Frac | Frac | Sci | Int | Float | Frac |  |
| GPT-4o-mini | 20 / 20 | 4 / 8 | 0 / 2 | 1 / 2 | 0 / 0 | 20 / 20 | 5 / 9 | 0 / 1 |  |
| GPT-4o | 20 / 20 | 4 / 7 | 0 / 2 | 1 / 2 | 0 / 0 | 20 / 20 | 5 / 10 | 0 / 1 |  |
| Qwen2-72B | 20 / 20 | 7 / 13 | 0 / 0 | 0 / 0 | 0 / 0 | 20 / 20 | 0 / 0 | 0 / 0 |  |
| Qwen2-7B | 11 / 12 | 0 / 13 | 0 / 0 | 0 / 0 | 0 / 0 | 10 / 12 | 0 / 12 | 0 / 0 |  |
| Llama-3.1-8B-ft | 19 / 20 | 11 / 20 | 0 / 7 | 0 / 5 | 0 / 7 | 20 / 20 | 13 / 20 | 0 / 5 |  |
| Llama-3.1-70B | 20 / 20 | 6 / 13 | 0 / 1 | 0 / 1 | 0 / 0 | 20 / 20 | 4 / 13 | 0 / 1 |  |
| Llama-3.1-8B | 11 / 20 | 4 / 11 | 0 / 1 | 0 / 0 | 0 / 0 | 10 / 20 | 6 / 10 | 0 / 1 |  |
| Mixtral-8x7B | 10 / 14 | 4 / 12 | 0 / 1 | 0 / 1 | 0 / 0 | 16 / 20 | 5 / 13 | 0 / 0 |  |
| Llama-2-7b-hf | 5 / 12 | 0 / 10 | 0 / 0 | 0 / 0 | 0 / 0 | 4 / 20 | 0 / 11 | 0 / 1 |  |
|  | Sub | Max | Max | Max | Max | Max Hard | Max Hard | Max Hard |  |
|  | Sci | Int | Float | Frac | Sci | Int | Float | Sci |  |
| GPT-4o-mini | 0 / 0 | 39 / 98 | 6 / 9 | 0 / 4 | 13 / 17 | 100 / 100 | 6 / 8 | 6 / 16 |  |
| GPT-4o | 0 / 0 | 31 / 72 | 6 / 9 | 0 / 2 | 12 / 17 | 100 / 100 | 6 / 8 | 8 / 16 |  |
| Qwen2-72B | 0 / 0 | 18 / 32 | 13 / 30 | 0 / 0 | 43 / 86 | 68 / 85 | 16 / 26 | 35 / 63 |  |
| Qwen2-7B | 0 / 0 | 11 / 23 | 0 / 17 | 0 / 0 | 0 / 14 | 86 / 100 | 0 / 20 | 0 / 11 |  |
| Llama-3.1-8B-ft | 0 / 0 | 47 / 83 | 35 / 54 | 0 / 5 | 97 / 100 | 100 / 100 | 25 / 52 | 35 / 69 |  |
| Llama-3.1-70B | 0 / 0 | 0 / 5 | 0 / 13 | 0 / 0 | 0 / 5 | 0 / 8 | 0 / 13 | 0 / 4 |  |
| Llama-3.1-8B | 0 / 0 | 0 / 0 | 0 / 7 | 0 / 0 | 0 / 0 | 0 / 18 | 0 / 6 | 0 / 0 |  |
| Mixtral-8x7B | 0 / 0 | 7 / 10 | 3 / 9 | 0 / 0 | 0 / 5 | 20 / 28 | 3 / 11 | 0 / 6 |  |
| Llama-2-7b-hf | 0 / 0 | 0 / 10 | 0 / 0 | 0 / 0 | 0 / 0 | 100 / 100 | 0 / 0 | 0 / 11 |  |
|  | Multiply Hard | Multiply Hard | Multiply Hard | Multiply Hard | Multiply Easy | Multiply Easy | Multiply Easy | Multiply Easy |  |
|  | Int | Float | Frac | Sci | Int | Float | Frac | Sci |  |
| GPT-4o-mini | 7 / 11 | 0 / 0 | 1 / 2 | 0 / 0 | 9 / 20 | 0 / 0 | 1 / 2 | 0 / 0 |  |
| GPT-4o | 7 / 11 | 0 / 0 | 1 / 2 | 0 / 0 | 16 / 20 | 0 / 0 | 1 / 2 | 0 / 0 |  |
| Qwen2-72B | 0 / 6 | 0 / 0 | 0 / 0 | 0 / 0 | 5 / 7 | 0 / 0 | 0 / 0 | 0 / 0 |  |
| Qwen2-7B | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 5 | 0 / 0 | 0 / 0 | 0 / 0 |  |
| Llama-3.1-8B-ft | 11 / 18 | 3 / 9 | 0 / 11 | 0 / 7 | 16 / 19 | 8 / 17 | 1 / 15 | 0 / 20 |  |
| Llama-3.1-70B | 5 / 9 | 0 / 3 | 0 / 1 | 0 / 0 | 7 / 16 | 0 / 3 | 0 / 1 | 0 / 0 |  |
| Llama-3.1-8B | 0 / 4 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 8 | 0 / 0 | 0 / 1 | 0 / 0 |  |
| Mixtral-8x7B | 4 / 6 | 0 / 0 | 0 / 2 | 0 / 0 | 4 / 8 | 0 / 0 | 0 / 2 | 0 / 0 |  |
| Llama-2-7b-hf | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 5 | 0 / 0 | 0 / 0 | 0 / 0 |  |
|  | Digit Max | Digit Max | Digit Add | Digit Add | Truediv | Truediv | Floordiv | Mod |  |
|  | Int | Float | Int | Float | Int | Frac | Int | Int |  |
| GPT-4o-mini | 5 / 16 | 0 / 7 | 0 / 8 | 0 / 0 | 0 / 5 | 1 / 2 | 8 / 14 | 3 / 15 |  |
| GPT-4o | 10 / 16 | 3 / 7 | 0 / 8 | 0 / 0 | 0 / 5 | 1 / 2 | 8 / 14 | 3 / 15 |  |
| Qwen2-72B | 7 / 16 | 0 / 9 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 1 | 5 / 7 | 0 / 5 |  |
| Qwen2-7B | 6 / 11 | 0 / 4 | 0 / 20 | 0 / 9 | 0 / 0 | 0 / 1 | 4 / 6 | 0 / 5 |  |
| Llama-3.1-8B-ft | 23 / 61 | 19 / 29 | 26 / 61 | 29 / 45 | 3 / 20 | 0 / 3 | 9 / 18 | 0 / 15 |  |
| Llama-3.1-70B | 6 / 16 | 0 / 7 | 0 / 11 | 0 / 0 | 0 / 0 | 0 / 1 | 3 / 10 | 0 / 4 |  |
| Llama-3.1-8B | 0 / 6 | 0 / 0 | 0 / 10 | 0 / 7 | 0 / 0 | 0 / 0 | 4 / 8 | 0 / 5 |  |
| Mixtral-8x7B | 0 / 12 | 0 / 4 | 0 / 8 | 0 / 7 | 0 / 0 | 0 / 1 | 4 / 7 | 0 / 5 |  |
| Llama-2-7b-hf | 0 / 7 | 0 / 3 | 0 / 9 | 0 / 7 | 0 / 0 | 0 / 1 | 0 / 3 | 0 / 6 |  |
|  | Mod Easy | To Float | To Float | To Scient | To Scient | Sig |  |  |  |
|  | Int | Frac | Sci | Int | Float | Int |  |  |  |
| GPT-4o-mini | 3 / 20 | 0 / 4 | 0 / 15 | 9 / 21 | 5 / 8 | 0 / 23 |  |  |  |
| GPT-4o | 3 / 20 | 0 / 4 | 0 / 15 | 9 / 21 | 3 / 8 | 0 / 23 |  |  |  |
| Qwen2-72B | 0 / 10 | 3 / 8 | 12 / 46 | 0 / 44 | 0 / 0 | 6 / 19 |  |  |  |
| Qwen2-7B | 0 / 7 | 3 / 7 | 76 / 93 | 0 / 0 | 0 / 0 | 0 / 5 |  |  |  |
| Llama-3.1-8B-ft | 0 / 17 | 3 / 9 | 14 / 78 | 27 / 33 | 16 / 38 | 0 / 0 |  |  |  |
| Llama-3.1-70B | 0 / 5 | 3 / 7 | 0 / 15 | 0 / 0 | 0 / 0 | 0 / 0 |  |  |  |
| Llama-3.1-8B | 0 / 6 | 3 / 6 | 0 / 12 | 0 / 0 | 0 / 0 | 0 / 0 |  |  |  |
| Mixtral-8x7B | 0 / 6 | 3 / 6 | 15 / 16 | 0 / 18 | 0 / 0 | 0 / 0 |  |  |  |
| Llama-2-7b-hf | 0 / 6 | 0 / 4 | 93 / 98 | 0 / 0 | 0 / 0 | 0 / 0 |  |  |  |

### A.4 Tokenizer, PE and data formats

#### A.4.1 Tokenization

Figure: (a) 0.1B model int add
Refer to caption: x22.png

We experiment on models of 3 different size, including 0.1B, 0.9B and 3B. For the 0.1B and 0.9B models, we train them on integer addition of 1-8 digits; for the 3B model, we train it on the same task of 1-40 digits.

Figure 9 illustrates the in-domain performance of these three models in the first three columns and their out-of-domain (OOD) performance in the last two columns. Here we use the exact match metric. In our experiments of the 0.1B and 0.9B models, the one-digit and the two-digit tokenizer demonstrate comparable performance in the in-domain test, while the one-digit tokenizer exceeds the others to a large extent in length generalization. In contrast, the three-digit tokenizer exhibits poor performance in both in-domain and out-of-domain evaluations. Tokenizers with an increasing number of digits significantly hinder subbillion models’ NUPA. In the experiments of the 3B model, the two-digit tokenizer matches the one-digit tokenizer in both in-domain and OOD performance. In addition, the three-digit tokenizer shows the potential in length generalization for the first time, yet its performance remains inferior to that of the smaller tokenizers. This indicates that scaling up the model size indeed alleviate the challenges in developing NUPA caused by larger tokenizers. Nevertheless, larger tokenizers do not present any distinct benefits in either in-domain or out-of-domain generalization in both small and large models.

We report the results according to different metrics from Figure 4 including digit match and dlength in Figure 10 and Figure 11.

Figure: (a) 0.9B model int add
Refer to caption: x25.png

Figure: (a) 0.9B model int add
Refer to caption: x28.png

#### A.4.2 PEs

We show exact match, digit match and dlength of 100M models trained with various PE, including RoPE, NoPE and Alibi in Figure 12. The shadowed areas denote the in-domain length range.

Figure: Figure 12: Exact match, digit match and dlength of 100M models trained with various PE, including RoPE, NoPE and Alibi. The shadowed areas denote the in-domain length range.
Refer to caption: x31.png

The results of the 2-layer transformer on four tasks are shown in Table 13 and the results of the 0.1B model with only 1M samples are shown in Table 14.

**Table 13: 8-digit digit match accuracy with small model (1.3M) with RoPE, NoPE and Alibi.**
|  | Int-add | Float-add | Fraction-multiplication | Scientific-add |
| --- | --- | --- | --- | --- |
| RoPE | 0.091 | 0.88 | 0.23 | 0.75 |
| NoPE | 0.061 | 0.39 | 0.17 | 0.52 |
| Alibi | 0.056 | 0.31 | 0.18 | 0.50 |

**Table 14: 8-digit digit match accuracy with small dataset (1M samples) with RoPE, NoPE and Alibi.**
|  | Int-add | Float-add | Fraction-multiplication | Scientific-add |
| --- | --- | --- | --- | --- |
| RoPE | 0.97 | 0.99 | 0.33 | 0.99 |
| NoPE | 0.78 | 0.98 | 0.29 | 0.96 |
| Alibi | 0.23 | 0.80 | 0.17 | 0.79 |

##### Discuss about scientific addition

The results in Table 2 reveal a clear trend where performance drops from 8-digit to 9-digit numbers, with one exception: the digit match score in the scientific notation addition task, which remains relatively high at 0.93 even for 9-digit numbers. We believe it is mainly because of the alignment mechanism between two scientific notations which differs from other representations. In other representations, numbers are aligned by position — integers from the most-left digit and the floats by the decimal point. However, in scientific notation, alignment depends on the difference in exponent values, which reduces RoPE’s reliance on position and mitigates length overfitting. Despite this, the effect of RoPE limiting output length remains apparent, as evidenced by the significant increase in the dlength score.

#### A.4.3 Data Formats

We provide the evaluation curves of compositions of reverse formatting, zero padding and index hints in
Figure 13, Figure 14,
Figure 15,
Figure 16 and
Figure 17. We experiment on 0.1B models trained on 1- to 8- digit training samples. Here we all use the exact match metric.

About the experiments of index hint, we show in Table 15. Our conclusion on index hints seems to contradict the findings of Zhou et al. (), where models with index hints appeared to achieve better results. We believe this discrepancy may be related to model size and digit range. In their work, a much smaller model (only 25M parameters) was used, but the training range covered 1-40 digits. This reduced the model’s ability to learn the patterns independently without external hints, resulting in a different learning outcome where the model began to rely on index hints. As a piece of evidence, when Zhou et al. () train 1-10 digits, the performance without index hint is OK. (But they did not provide the complete results of 1-10 digit training in their work.) The effectiveness of index hints may involve complex interactions, which could be an interesting direction for future research.

**Table 15: Exact match of 0.1B models trained on integer addition, multiply and maximum respectively with various compositions of reverse formatting and index hints.**
|  | Integer Addition | Integer Multiply | Integer Max |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | rev | rev<br>+ idx | no | idx | rev | reverse<br>+ idx | no | idx | reverse only | reverse<br>+ idx | no | idx |
| d9 | 1.00 | 0.93 | 0.98 | 0.41 | 0.43 | 0.00 | 0.13 | 0.00 | 1.00 | 0.99 | 1.00 | 0.99 |
| d10 | 0.80 | 0.06 | 0.32 | 0.01 | 0.13 | 0.02 | 0.04 | 0.02 | 1.00 | 0.97 | 1.00 | 0.98 |

#### A.4.4 NUPA fintuning with PE, tokenizer and representation modification

We show parts of results of our attempt to finetune a Llama-3.1-8B model with PE, tokenizer and data format modification in Table 16. All the checkpoint we select by the lowest valid loss. No one can outperform the naive finetuning or the original Llama.

**Table 16: Finetining with PE, data format, and tokenizer modification will degrade the performance. The first two lines are a naive finetuned Llama and the original Llama without finetuning, which are the baseline. “1d” means using the one-digit tokenizer for numbers otherwise the original tokenizer. “rev” means reverse representation, where the integer parts are reversed. All the checkpoint we select by the lowest valid loss. The accuracy reported is the average “exact match” in each range. Metric “wld” is used to denote well-learned digit; “ppd” is used to denote performance-preserving digit.**
|  | Integer Addition | Float Addition |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | S | M | L | XL | wld | ppd | S | M | L | XL | wld | ppd |
| FT | 0.95 | 0.65 | 0.12 | \ul0.01 | 4 | 12 | 0.96 | 0.71 | 0.27 | 0.08 | 5 | 17 |
| w/o FT | 0.95 | \ul0.38 | \ul0.06 | 0.02 | 4 | \ul9 | \ul0.90 | \ul0.47 | \ul0.10 | \ul0.02 | \ul3 | \ul11 |
| NoPE | 0.67 | 0.04 | 0.00 | 0.00 | 3 | 5 | 0.37 | 0.06 | 0.00 | 0.00 | 0 | 0 |
| NoPE + rev + 1d | \ul0.89 | 0.35 | \ul0.06 | 0.02 | \ul3 | \ul9 | 0.81 | 0.38 | 0.09 | 0.01 | 0 | \ul11 |
| NoPE + rev + pad + 1d | 0.87 | 0.34 | 0.05 | 0.02 | 0 | \ul9 | 0.74 | 0.38 | 0.06 | 0.01 | 0 | 9 |
| RoPE + 1d | 0.93 | 0.59 | 0.05 | 0.00 | 4 | 9 | 0.33 | 0.30 | 0.06 | 0.01 | 0 | 9 |
| RoPE + rev + 1d | 0.40 | 0.20 | 0.04 | 0.00 | 0 | 7 | 0.35 | 0.30 | 0.09 | \ul0.02 | 0 | \ul11 |
|  | Fraction Multiplication (easy) | Scientific Notation Addition |  |  |  |  |  |  |  |  |  |  |
|  | S | M | L | XL | wld | ppd | S | M | L | XL | wld | ppd |
| FT | 0.43 | 0.00 | 0.00 | 0.00 | 1 | 3 | 0.09 | 0.08 | 0.02 | 0.00 | 0 | 4 |
| w/o FT | \ul0.28 | 0.00 | 0.00 | 0.00 | 0 | 3 | 0.02 | 0.02 | \ul0.01 | 0.00 | 0 | 0 |
| NoPE | 0.14 | 0.00 | 0.00 | 0.00 | 0 | 3 | 0.00 | 0.00 | 0.00 | 0.00 | 0 | 0 |
| NoPE +rev + 1d | 0.25 | 0.00 | 0.00 | 0.00 | 0 | 3 | 0.01 | 0.02 | \ul0.01 | 0.00 | 0 | 0 |
| NoPE + rev + pad + 1d | 0.27 | 0.00 | 0.00 | 0.00 | 0 | 3 | 0.01 | 0.02 | \ul0.01 | 0.00 | 0 | 0 |
| RoPE + 1d | 0.09 | 0.00 | 0.00 | 0.00 | 0 | 1 | \ul0.08 | \ul0.03 | 0.02 | 0.00 | 0 | \ul3 |
| RoPE + rev + 1d | 0.06 | 0.00 | 0.00 | 0.00 | 0 | 1 | \ul0.08 | 0.02 | \ul0.01 | 0.00 | 0 | 4 |

Figure: Figure 13: Exact match of 0.1B models trained on 1- to 8- digit integer addition with different compositions of reverse formatting and zero padding on 8- to 10- digit tests. X-axis is the number of seen training samples.
Refer to caption: x32.png

Figure: Figure 14: Exact match of 0.1B models trained on 1- to 8- digit float addition with different compositions of reverse formatting and zero padding on 8- to 10- digit tests. X-axis is the number of seen training samples.
Refer to caption: x33.png

Figure: Figure 15: Exact match of 0.1B models trained on 1- to 8- digit integer addition with different compositions of reverse formatting and index hints on 8- to 10- digit tests. X-axis is the number of seen training samples.
Refer to caption: x34.png

Figure: Figure 16: Exact match of 0.1B models trained on 1- to 8- digit integer multiplication with different compositions of reverse formatting and index hints on 8- to 10- digit tests. X-axis is the number of seen training samples.
Refer to caption: x35.png

Figure: Figure 17: Exact match of 0.1B models trained on 1- to 8- digit integer maximum with different compositions of reverse formatting and index hints on 8- to 10- digit tests. X-axis is the number of seen training samples.
Refer to caption: x36.png

### A.5 Rule-following Chain-of-thought

The selective tasks used to train the RFFT are shown in Table 17 and we also report the maximal length within 2k tokens context windows limitation. For the detailed prompt of these tasks we cannot put them into papers so we include them in supplementary.

**Table 17: Maximum length of each task that 2k context window can afford with RF-CoT**
|  | Add | Sub | Multiply | Floordiv | Mod | Max | DigitMax | GetDigit | Length |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Integer | 20 | 20 | 12 | 20 | 6 | 100 | 17 | 100 | 34 |
| Float | 6 | 5 | 4 | - | - | 50 | - | 100 | - |
| Fraction | 3 | 2 | 3 | - | - | 20 | - | - | - |
| Scientific | 3 | 3 | 3 | - | - | 100 | - | - | - |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: How do you get ground truth reasoning traces for RFFT?
soundness: 3
strengths: 1. The paper does a good job of broadening the evaluation of numerical reasoning from integers—which is often the focus in these kinds of studies—to more general classes of numbers. In general, there are a lot of experiments here, including several tasks and language models. The claim that models struggle with basic numerical understanding is well supported by the results. 
2. The discussion on tokenization is quite interesting, surprisingly few studies that I am aware of consider how tokenization affects numerical reasoning. Although not very surprising, it is nice to have empirical evidence that 1-digit tokenization yields stronger generalization to more complex examples than those in the training set. 
3. The paper goes beyond being purely “descriptive”, it also investigates methods that could *improve* numerical reasoning.
summary: This paper investigates the empirical abilities of LLMs to solve numerical reasoning tasks of varying complexity. It proposes a benchmark, called NUPA, incorporating several different number representations (integers, fractions, etc) and reasoning tasks (arithmetic, comparisons, etc). Its experiments show that several well-known LLMs are prone to errors on this benchmark, particularly as the digits get larger. The paper also analyzes how factors like tokenization, data formats, and finetuning affect performance on NUPA.
weaknesses: 1. Numerical reasoning in LLMs is well-studied by now. Yet, the paper lacks a related work section and has very few references to similar studies overall. It is therefore unclear to what extent the insights here are novel. Some of the claims of novelty also come off as overly strong, e.g., the last sentence of the abstract “our work takes an initial step towards understanding and improving NUPA [numerical understanding and processing ability] of LLMs.” I include a short list of relevant references below; however, I would suggest the authors to perform a more comprehensive literature review so that they can properly situate their work.
2. The paper is somewhat poorly written. First, there are many grammatical errors in this text (e.g., l62-3, l123). Second, many parts lack explanation or justification. For instance, it is not explained how the random tokenizer works. The technique appears to be adapted from Sathe et al. (2024), but that is a recent and probably not very well-known paper. On another note, random tokenizers were not introduced by that paper as suggested in the text; see for instance Kudo (2018). Another part that is unclear to me is the section on positional encodings. It is not explained what is meant by length-agnostic and length-related rules. It also doesn’t contextualize the methods studied—RoPE and Alibi. What are they and why are they studied? Is there some plausible explanation for why you observe the results you do?
3. The paper is experiment-driven; there is no underlying theory guiding the selection of tasks or techniques for improvement. The scope is also very broad—the paper attempts to do many things at once. While that may be seen as a strength, I feel that it comes at the sacrifice of depth of analysis and clarity. I would suggest the authors to construct a more focused narrative and provide justifications that are grounded either in previous work or theory. 
4. The paper lacks important statistical reporting like confidence intervals or p-values. 
5. As a minor additional note, Figure 2 (and the similar figures in appendix) is quite hard to parse. There is a lot of information and it is difficult to distinguish the models since some of the colors are rather similar. It is also confusing that the results for the finetuned model are here since finetuning is discussed much later in the paper. 


----

Dziri et al. 2023. Faith and Fate: Limits of Transformers on Compositionality.

Kudo et al. 2023. Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?

Razeghi et al. 2022. Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning.

Zhang et al. 2024. Interpreting and Improving Large Language Models in Arithmetic Calculation.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: The lack of discussion on previous work is a significant drawback. I would be willing to reconsider my score if this is addressed in the paper.
soundness: 2
strengths: - The evaluation procedure is well-structured and comprehensive.
- Some tasks in the evaluation suite offer useful insights into model failure modes.
summary: This paper evaluates language models on tasks involving numerical and arithmetic reasoning. The authors introduce a test suite with 17 numerical tasks (e.g., addition, subtraction, modulo, max, min, and digit retrieval) across four numerical representations, using it to evaluate nine language models. The paper also analyzes the impact of tokenization, positional embedding type, and data format on a model’s ability to perform addition and multiplication. Finally, the authors assess whether fine-tuning, with and without chain-of-thought, can improve LLaMA-3 8B’s performance.
weaknesses: - Attribution and discussion of previous work is extremely poor:
    - Notably, a discussion of the related work is missing. The the proposed evaluation suite, the design choices for it, and the results obtained should be discussed in light of previous studies that evaluated and analyzed the performance of language models on task involving numerical reasoning [e.g., 1,2,3,4,5,6]
    - The authors fail to reference papers that introduce methods that they directly adopt: chain-of-thought prompting was introduced by Wei et al. [7], and LoRA was proposed by Hu et al. [8]. Both citations are missing.
- Although the authors study the impact of tokenization on the performance of a model that they train form scratch, a discussion of how the tokenization of the pre-trained models evaluated might affect the results is missing. Additionally, here again, any reference to previous work that studied the impact of tokenization on numerical reasoning [e.g., 9] is absent.
- Some presentation/clarity issues:
    - Presenting the fine-tuning results in Table 2 is confusing, as they are discussed only much later in Section 4.
    - Figure 2 can be improved, especially in the choice of color for the LLaMA models, which are quite hard to distinguish in the bar plot.
    - Line 512.5: I understand that rule-following fine-tuning is introduced by Hu et al., but further details about the fine-tuning process (potentially in the appendix) would be helpful.
    - The use of negative \vspace is excessive, resulting in cramped spacing between tables/figures and text, especially on pages 7 and 9.

---
[1] A Survey of Deep Learning for Mathematical Reasoning (Lu et al., ACL 2023)    
[2] Do NLP Models Know Numbers? Probing Numeracy in Embeddings (Wallace et al., EMNLP-IJCNLP 2019)    
[3] Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models (Lin et al., EMNLP 2020)    
[4] A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models (Stolfo et al., ACL 2023)    
[5] Injecting Numerical Reasoning Skills into Language Models (Geva et al., ACL 2020)    
[6] Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning (Razeghi et al., Findings 2022)    
[7] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35 (2022): 24824-24837.    
[8] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685 (2021).    
[9] Singh, A.K. and Strouse, D.J., 2024. Tokenization counts: the impact of tokenization on arithmetic in frontier llms. arXiv preprint arXiv:2402.14903.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 2
questions: Main questions is can you give details on how the experiments that use training of fine-tuning are done, and what in-domain and out-of-domain refers to (what examples do you hold out for this)?
soundness: 3
strengths: This paper has the potential to be an important contribution to the field, for two main reasons:

- Great benchmark proposed with a comprehensive range of tasks and difficulty levels, and good metrics for evaluating performance

- Excellent coverage of experiments. Testing comprehensively many aspects relevant to this problem, like different types of generalisation, different architectural changes, different existing models, effects of scale, existing methods for improving numeric understanding, prompting, etc. I would be very interested in these results, if the weaknesses below could be addressed.
summary: Designs a comprehensive benchmark for number understanding, covering four different number presentations (e.g. integers) and 17 tasks (such as multiplication). The benchmarks allows for testing a range of different numerical understanding skills, at different levels of difficulty (through longer numbers in terms of digits for example). The authors do a comprehensive range of experiments on the benchmarks, evaluating off-the-shelf LLMs on it, as well as training their own models to understand the effects of aspects of common LLMs (such as tokenizers) and existing methods for improving numerical understanding (such as representing in reverse order). The findings are the many models still struggle in elementary understanding tasks, especially for larger numbers. Moreover, the existing tricks to deal with this don't fully mitigate the problems.
weaknesses: Unfortunately, the presentation is lacking and it's very hard to interpret the results because of missing details on how models are trained.

**Main weaknesses**
- There is no information at all in the paper about how the models are trained for the experiments in section 3.1 and 3.2. How much examples do you train on? How long? Optimizers, hyperparameters, repeats, etc.? How do you define in and out-of-domain? It's very difficult to interpret the results of this experiment without knowing more.

- The paper needs substantial rewrites, and will probably benefit from corrections by someone whose first language is English or by using an LLM to revise and point out mistakes. 

- There is a long discussion of things like the relevance of integers and how fractions arise (section 2.1 and 2.2), but this seems unnecessary and the space could be used instead to represent the results better. For example, Figure 2 presenting the main results in the text is very small and almost impossible to read without zooming in 200%. The colors used are also hard to distinguish. I would suggest shortening section 2.2 and almost entirely removing 2.1 and use the space to represent the results better. Especially because this is the great part of this paper, you have so many results but you do not discuss the details of the experiments and present the results in a way that are hard to follow.

- Some results are interpreted in ways that are not substantiated by the evidence found. For example, line 298 to 309; I disagree that this result means the model does not understand the concept of digit. The fact that it becomes harder to return the right digit for longer numbers actually hints at something else going on that might be more related to the model's inherent difficulties with retrieving something from a position in longer sequences, which has nothing to do with it's understanding of the concept digit. I also disagree that this points to case-based reasoning, because again it might be due to some aspects of its architecture. Additionally, the fact that models of different sizes show the same performance on a task does not necessarily indicate the performance depends on the training approach over size, it might also depend on architecture.

- There is no section on related work, and the authors say in the beginning of section 2 that they will show limitations of prior benchmarks while discussing the coverage of their own, but this does not happen. This makes it difficult for the reader to place this contribution w.r.t. existing literature.

**Minor points**

- Would be great to already get some more concrete information in the abstract (what probability of error? which 3 factors influence it?). Or at least in the intro some more concrete info.

- Figure 4 and 5: it's unclear what's on the X-axis (though one can assume it's training steps), and also unclear what D6 to D10 refers to. It's also unclear what is in-domain and what is OOD without reading the text.

- *"These types of errors are a major cause of hallucinations when dealing with math, reasoning, and data analysis tasks, as the model presents seemingly correct problem solving approaches but ultimately produces incorrect results"* -> this statement requires a citation

- line 117 to 119; a discussion on the innateness of integer understanding seems somewhat out of scope for this work and not too relevant to the contribution.

- Cite / reference openai chatgpt when using it line 135

- One approach that seems missing is few-shot examples, which might significantly boost performance for all models. Not just because they might learn from the examples, but because they get primed on the output format required.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. Why is it important that LLMs be good at NUPA when we python, and even LLM code-generation that can write python to deal with NUPA? I.e. Why is this a worthwhile problem, and who says it is?

2. To what extent do your findings suggest that the issue is architectural or to do with tokenization? To the degree that this is the case, to what extent do your findings suggest that the issue is paradigmatic rather than technical?

3. Is it necessary to provide normative justifications for the benchmarking suite? If so, I would like to know why, because I think the paper could be stronger if the benchmarking tests were presented matter-of-factly, without detours.

4. Experimentally, how much compute (as an estimate) was used throughout?
soundness: 2
strengths: It is clear that some thought has gone into curating the benchmark tasks and justifying their inclusion. Even though I do not personally agree with some of the justifications and normative statements made, I believe that this sort of conscientious effort to justify a benchmark is commendable. I rate this work highly in terms of integrity, quality of presentation, and clarity.

I particularly appreciated the authors' investigation of CoT, as it is generally interesting to understand whether "prompt-level" interventions have a lot of room for improving performance on complex tasks. A recent (contemporaneous, so not factoring into my decisions) paper by Apple on GSM8k qualifying LLM performance as memorisation seems to agree with the findings of this work that numeracy abilities do not appear to scale; potentially a way to frame and situate this work with respect to the broader discourse on generalisation and understanding in LLMs.
summary: This work introduces a benchmark suite of "numerical understanding and processing ability" (NUPA) tasks for transformer-based LLMs. The task suite separates numeracy from potentially confounding logical reasoning problems, and presents a variety of numerical formats. Additionally, various tricks are explored to improve LLM performance in these numeracy tasks; no silver bullets.
weaknesses: I have no issues with the benchmark and evaluation. I do not currently find the conceptual motivations and wrapping compelling; I could be convinced otherwise for each of these with the help of citations, or of better arguments, and I would be happy to raise my score accordingly.

1. I am unconvinced that numeracy in LLMs is a problem in need of a solution. First, surely there is a citable source for LLM inadequacy for numeracy. Second, even if they were terrible at numeracy, the onus is on the authors to convince the reader that this a problem worth caring about, for at least two obvious reasons: 1) all of these tasks are already trivially done by a calculator or a python program, and 2) commercially available LLMs can probably do alright at numerical tasks indirectly via code-generation and execution. As it stands, it reads as if the authors are insisting that this is a problem deserving of attention --- I'm sure it could be, but this argument can be better made.

2. I am unconvinced that numeracy in LLMs is a problem in search of deeper understanding. Consider that "How many r's in strawberry" is a meme that normal people know about; the idea that tokenization impairs the ability of transformers to reason at a letter-, digit-, or word-unit level is well-digested. So when the authors claim that the weakness of LLMs at digit understanding is surprising (line 298), I find this a bit sensationalist and not grounded with respect to a broader and basic level of discourse around the capabilities of LLMs.

3. I am unconvinced of the normative rationales supporting the benchmarks, which seem ad-hoc. Here is an example: the authors claim that multiplication (not a particular algorithm) is O(L^2) in the length L counted in digits (of presumably both inputs; line 169), and take this as justification that models ought to find multiplication difficult. This doesn't hold up to armchair introspection: lookup is O(1), and we have all learned our times tables as children. The competent-human process of mental multiplication is mostly a sequence of lookups along with additive de/recomposition, so the real exponent is probably less than 2. This is a jarring oversight when contrasted with the effort with which the authors bring up cognitive and developmental psychology (lines 116 to 127), recasting what I assumed was a scholarly inclusion as a kind of checklisting exercise. There are other unsupported normative claims about what a good benchmark ought to encompass, such as "NUPAs are necessary and expected to learn by LLMs [sic]" (line 114), "any student who has completed primary and secondary education should be able to accomplish them" (line 160), and "for most practical tasks involving numbers (like arithmetic or math tests), all we care about is whether the answer is right, and there is no significant difference between being almost right and being completely wrong" (line 228). While I understand the importance of establishing normative standards to justify a benchmark with respect to, I find it hard to believe as a reader that the authors have the expertise and authority to do so.

4. I am unconvinced that this is worth "taking seriously"; let us suppose that everyone agrees that NUPA in LLMs is a problem worth solving and that improving on this benchmark is the way to solve the problem. We know beforehand that transformers have problems with NUPA due to tokenization, and now the finding of this paper is that fine-tuning and CoT as tricks don't solve the problem. There are many graphs in this paper that must have taken a fair amount of compute to obtain; is it worth it if everyone decided to run similar tests, or fine-tune with respect to the NUPA suite? I don't mean this in an unnecessarily antagonistic way, but one concerned with scientific interest and downstream impacts; if LLMs are just bad at numeracy for architectural or structural reasons as the current evidence suggests, could the introduction of a benchmark be inviting high-compute low-innovation approaches to the problem, and does the problem even merit this kind of resource investment?

Meta-review: The authors present a new benchmark for numerical reasoning and several experiments studying the effect of various manipulations on benchmark performance. This addresses an important problem, broadens the scope of previous benchmarks, and raises insights about model failures -- overall presents a clear and useful new contribution.

Decision: Accept (Poster)