Teaching Arithmetic to Small Transformers
Authors: Nayoung Lee
Kartik Sreenivasan
Jason D. Lee
Kangwook Lee
Dimitris Papailiopoulos
Venue: ICLR 2024 poster (2024)
Keywords: Machine Learning
Transformers
Large Language Models
Primary area: general machine learning (i.e., none of the above)

Abstract: Large language models like GPT-4 exhibit emergent capabilities across general-purpose tasks, such as basic arithmetic, when trained on extensive text data, even though these tasks are not explicitly encoded by the unsupervised, next-token prediction objective. This study investigates how even small transformers, trained from random initialization, can efficiently learn arithmetic operations such as addition, multiplication, and elementary functions like square root, using the next-token prediction objective. We first demonstrate that conventional training data is not the most effective for arithmetic learning, and simple formatting changes can significantly improve accuracy. This leads to sharp phase transitions as a function of training data scale, which, in some cases, can be explained through connections to low-rank matrix completion. Building on prior work, we then train on chain-of-thought style data that includes intermediate step results. Even in the complete absence of pretraining, this approach significantly and simultaneously improves accuracy, sample complexity, and convergence speed. We also study the interplay between arithmetic and text data during training and examine the effects of few-shot prompting, pretraining, and parameter scaling. Additionally, we discuss the challenges associated with length generalization. Our work highlights the importance of high-quality, instructive data that considers the particular characteristics of the next-word prediction loss for rapidly eliciting arithmetic capabilities.

Published as a conference paper at ICLR 2024
TEACHING ARITHMETIC TO SMALL TRANSFORMERS
Nayoung Lee"
Kartik Sreenivasan" University Wisconsin-Madison kszeenivas2@wiscc.em
University of Wisconsin-Madison nayoung. leeewisc......................s. edu
Jason D. Lee
Kangwook Lee University Wisconsin-Madison
Princeton University
jassnlseeprinneetonnedd
kangwook. lee@wisc. edu
Dimitris Papailiopoulos University of Wisconsin-Madison
dimitris@papai 1.
ABSTRACT
Large language models like GPT-4 exhibit emergent capabilities across general- purpose tasks, such basic arithmetic, when trained on extensive text data, even though these tasks are not explicitly encoded by the unsupervised, next-token prediction objective. This study investigates how even small transformers, trained from random initialization, can efficiently learn arithmetic operations such as addi- tion, multiplication, and elementary functions like square root, using the next-token prediction objective. We first demonstrate that conventional training data is not the most effective for arithmetic learning, and simple formatting changes can signifi- cantly improve accuracy. This leads to sharp transitions as function of training data scale, which, some cases, can be explained through connections to low-rank matrix completion. Building on prior work, then train on chain-of-thouse style data that includes intermediate step results. Even in the complete absence of pretraining, this approach significantly and simultaneously improves accuracy, sample complexity, and convergence speed. We also study the interplay between arithmetic and text data during training and examine the effects few-shot prompt- ing, pretraining, and parameter scaling. Additionally, we discuss the challenges associated with length generalization. Our work highlights the importance of high-quality, instructive data that considers the particular characteristics of the next-word prediction loss for rapidly eliciting arithmetic capabilities.
INTRODUCTION
Large language models like GPT-3/4, PaLM, LaMDA (Brown et al., 2020: Chowdhery al., 2022; Thoppilan et al., 2022) have demonstrated gemeral-purposs properties, often referred to as emergent abilities (Wei al., 2022a), for wide range of downstream tasks like language and code translation, compositional reasoning, and basic arithmetic operations (Webb et al., 2022; Nye et al., 2021; Wei al., 2022b; Shi al., 2022; Wang et al., 2022: Srivastava et al., 2022: Chen et al., 2023). What is perhaps surprising, is that these tasks are not explicitly encoded in the model's training objective, which typically is auto-regressive, wext-token-predietinn loss.
Prior research has delved into exploring these capabilities and how they emerge as the scale and of training compute, type of data, and model size vary (Wei et al., 2022a: Chung al., 2022: Tay et al., 2022). Untangling the factors, however, remains challenging due to the data complexity and the variety of tasks examined. Driven by the curiosity to understand the factors that elicit these capabilities in next-token predictors, we set out to pinpoint the key contributors that accelerate the
Authors contributed equally to this paper.
Our code is available at httpt:t///// // //g thub. com/ htwp:/eeeeeeheenneeeeeeeeeeeaeee http://tachinggart.. thmetic

---

Published as a conference paper at ICLR 2024
Data Formatting
Plain
Reverse
Detailed Seratchpad
128-367-495
5129-367-5945
128-367
Accuracy (%)
Tagget:
60
Simplified Seratchpad
[12.8] digis.
Target: A-5., GiS1
Imput: 128-367
[3667] 7 has digis,
[1.2.5] x [3677, A-]l, C-o, 847100185 Co]
1122)133 6) 0=1,,01,,,
Co0
40
reverse
plan
A-99.C.00 C.-0
[3] AB) [0.5] [0.5).C-n. 1--0, 14HH044 A-d,, Coll
0-0.A- [4,9,], CND
C--..
simplitee scratchpad
detailed screichped
695
</scmoocs
3000
49.5
Number examples
Figure 1: We investigate four data formatting approaches: (i) Plain: standard addition formatting (Section 4), (ii) Reverse: reversing the output (Section 4), (iii) Simplified Scratchpad: recording the digit-wise sum and carry-ons (Section 6), and (iv) Detailed Scratchpad: providing detailed intermediate steps (Section 6). We train small dncoder-only transformers from seratch on addition data in these formats. The results (right) highlight the crucial role of data formatting in accuracy and sample efficiency. Plain never reaches 100% accuracy and the sample complexity for the remaining methods steadily improves with the level of details in the data format.
emergence of such abilities. These contributors may include the format and scale of data, model scale, the presence of pre- pre-training, and the manner of prompting.
To provide a more precise examination of these factors, our study is conducted in a controlled setting: we first focus teaching arithmetic to small decoder-only transformer models, such as NanoGPT and GPT-2, when trained from random initialization. Starting with a model of 10.6M parameters and scaling up to 124M parameters, we use the standard autoregressive next-token prediction loss. Our objective is to understand if and to what degree these models can efficiently learn basic arithmetic operations like addition, subtraction, multiplication, square root, and sine, thereby providing clearer lens through which to view the elicitation of emergent abilities. Below, we summarize our findings.
Data format and sampling plays a significant role. We first observe that teaching a model addition (or any other operation) using standard addition samples, i.e., AAgA2A + B3B,B, = C'', is suboptimal, as requires the model to evaluate the most significant digit C3 of the result first, which depends globally on all the digits of the two summands. By training
samples with reversed results, i.e., 'A3A2A1 + B3BrB, = C C3''', we enable the model to learn a simpler function, significantly improving sample complexity. Additionally, balanced sampling of differrnnt "variints"" of addition, based on the number of carries and digits involved, further enhances learning. Even in this simple setting, we observe relatively sharp phase transitions from 0 to 100% accuracy function of the size of the training data. Although this may seem surprising, we observe that learning an addition map on digits from random samples is equivalent to completing low-rank matrix. This connection allows us to offer a reasonable explanation for such phase transitions.
thain-of-though data during training. Building on these findings, we then explore the potential benefits of thain-of-thougeee (CoT) data during training. This format includes step-by-step operations and intermediate results, allowing the model to learn the individual components of compositional tasks. This format is directly borrowed from related literature, e.g., Ling et al. (2017); Wei et al. (2022b); Zhou et al. (2022a:b). We find that CoT-type training data significantly improved learning in terms of both sample complexity and accuracy in agreement with CoT fine-tuning literature (Nye et al.. 2021; Chung et al., 2022). but even in the complete absence of pretraining. We conjecture that this is because breaking down the required compositional function be learned into individual components allows the model to learn a higher-dimensionall but easier-to-learn function map, in agreement with recent theoretical findings (Li et al., 2023; Malach, 2023). In Figure I, we provide examples of the data formatting methods explored in our work.
Training on text and arithmetic mixtures and the role of few-shot prompting. We also explore the interplay between arithmetic and text data during training, as LLMs are trained on massive amounts of data scraped from the internet (Bubeck et al., 2023: Peterson et al., 2019), where is impractical to carefully separate different types of data. We observe how the model's 's perplexity and accuracy vary with the ratio of text to arithmetic data. We find that jointly training on all the arithmetic operations discussed earlier can improve the individual performance of each task and that going from zero-shot to 1-shot prompting (showing one arithmetic example) yields a large accuracy improvement, but there is no significant improvement in accuracy by showing more examples.
The role of pre-training and model scale. We further investigate the role of pretraining by fine- tuning pretrained models like GPT-2 and GPT-3 (davi nci)) and observe that while the zero-shot
2

---

Published as a conference paper at ICLR 2024
performance arithmetic operations is poor, prior "skills" acquired during pretraining facilitate quick learning of some basic arithmetic tasks, even with a small number of finetuning samples. However, finetuning on non-standard data, such as those that result from reverse formatting. can interfere with the model's performance when pretrained, leading to decreased accuracy. We finally share our observations on how performance in arithmetic changes with scale, and although we find that scale does aid when finetuning for these tasks, it is not necessary trait.
Compositional and length generalization. One might question if our trained models truly grasp arithmetic. Our findings present nuanced answer. We find that length generalization beyond trained digit lengths is still challenging. For instance, if model is trained on all n-digit lengths, excluding a specific length, it still struggles to accurately calculate this missing digit length. Consequently, the models achieve high accuracy within trained digit lengths but struggle significantly beyond this range. This suggests that the models learn arithmetic not as a flexible algorithm, but as a mapping function constrained to trained digit lengths. While this significantly surpasses memorization, it falls short of comprehensive arithmetic "understanding".
Novelty over prior work. Our approach heavily builds upon prior work that uses reasoning- augmented data enhance model performance, and we do not purport originality in the types of training data used, nor in achieving the highest performance with the smallest model parameters possible. What sets our work apart is the primary focus on meticulously ablating our settings and extensive studies on various sampling techniques, training data formats, data source mixing ratios, and model scales. Our goal is to pinpoint the factors that contribute to the fast emergence of arithmetic capabilities. In the process, we also provide several straightforward yet novel and insightful theoretical explanations for some of the phase transition phenomena we observe. Our emphasis on arithmetic is not due to its intrinsic significance one can easily delegate calculations to external tools (Schick et al., 2023; Gao et al., 2023). Instead, arithmetic serves as an emergent skill, easy to isolate and test, facilitating a more precise exploration of emergent phenomena.
2 RELATED WORKS
Instructional data/chain-of-thoughe
Detailed reasoning in training data has roots predating
Transformers ( (Vaswani et al., 2017). Ling et al. (2017); Cobbe et al. (2021) use natural language to generate reasoning steps while Roy & Roth (2016); Reed & De Freitas (2015); Chen et al. (2017);
Cai et al. (2017); Nye et al. (2021) show that symbolic reasoning may suffice. Nogueira et al. (2021) stress the importance of large number of small-digit samples (Yuan et al., 2023). Razeghi (2022) observe correlation between the frequency of numbers in the dataset and the performance involving them. In contrast, we find that transformers can learn to add numbers that were not seen during training. Chain-of-thouge (Wei et al., 2022b) refers to the model's improved accuracy when prompted to produce intermediate reasoning steps. Zhou et al. (2022b) show that this can be achieved by providing sufficiently informative exemplars (2022a) showed that least-to poast-to-most prompting can help GPT-3 solve problems decomposable into simpler sub-problems, by sequentially solving these subproblems. We extend this notion to simple addition and show that asking the model to output the least significant bit first has a similar effect.
al.
few-shot prompt (Brown et al., 2020). Zhou et al.
Arithmetic using Transformer models. Our work focuses on decoder-only models they widely used in LLMs (Brown et al., 2020; Touvron et al., 2023; MosaieML., 2023). However, encoder-decoder models have also been extensively studied in the context of learning arithmetic operations (Kim et al., 2021; Wang et al., 2021). Wallace et al. (2019) on the other hand, focus on the impact of the learned embeddings. Ontanón al. (2021) extensively study the problem of compositional generalization on benchmark datasets, such as SCAN (Lake & Baroni, 2018; Drozdov al., 2022), and conclude that design choices. like relative position encoding (Shaw al., 2018), can improve performance. Charton (2022; 2021) show that Transformers can learn linear algebra operations with carefully chosen encodings. Hanna et al. (2023) use mechanistic interpretability techniques to explain the limited numerical reasoning capabilities of GPT-2. Dziri et al. (2023); Jelassi et al. (2023); Yang et al. (2023) focus on the challenges of length generalization. A recent line of work explores finetuning techniques to improve arithmetic capabilities in pretrained models (Qian et al., 2022; Lightman et al., 2023; Uesato et al., 2022).
Beyond Transformers. While we focus attention GPT-like models, there is rich literature studying other seq-to-seq models such as recurrent neural networks (RNNs) (Bowman, 2013; Bowman et al., 2014; Zaremba et al., 2014). Zaremba & Sutskever (2014) show that RNNs can learn how to
3

---

Published as a conference paper at ICLR 2024
execute simple programs with for-loops provided they are trained with curriculum learning. Sutskever et al. (2014) show that LSTMs show improved performance on text- based tasks such as translation when the source sentences
reversed, which is closely related to what we observe in addition. Kaiser & Sutskever (2015) propose Neural GPUs which outperform prior RNNs on binary arithmetic tasks and even show length generalization i.e., they can perform arithmetic on inputs of lengths that were unseen during training. This is yet to be seen even in modern pre-trained models (Bubeck et al., 2023) and therefore is interesting see if we can leverage some of these techniques and apply them to existing modern architectures. Dehghani et al. (2018) propose Universal Transformers (UTs) which introduce a recurrent transition function to apply recurrence over revisions of the vector representation at each position as opposed to the different positions in the input. They show that on the tasks from Zaremba & Sutskever (2014), UTs outperform traditional Transformers and RNNs.
3 PRELIMINARIES AND EXPERIMENTAL SETUP
In this section, we provide a detailed description of our experimental setup, including the model architecture and an overview of the various data formatting and sampling techniques used.
Model and Data. To examine the individual factors at play, we use NanoGPT (Karpathy, 2022), a lightweight implementation of the GPT family of models. NanoGPT is a decoder-only transformer with six self-attention layers, six heads, and an embedding dimension of 384, resulting in approxi- mately 10.6M parameters. Unless stated otherwise, position encoding. We train NanoGPT from random initialization, which we refer to as training from scratch, using the conventional next-token prediction objective. To study the effect of scale, we extend our experiments to GPT-2 and GPT-3 in Section 8. We investigate both training from scratch as well as tne-tuning using a pretrained GPT-2, whereas, for GPT-3, we only consider fine-tuning pretrained models. Refer to Appendix I for more details on the models and data used.
character-level tokenization and absolute
For arithmetic tasks like addition, subtraction, and multiplication, we define the training dataset for a binary operator f-) as Durin {aa, b;), ) where = f(a, b.). For unary operations like sine, the training dataset is formulated as Dinin {a,, 3i ) where y/ f(a)). The test dataset Diest is constructed by randomly sampling pairs of operands not included in Dunin- We then apply different data formatring techniques on each data sample from the training dataset, creating the final sequence that serves as the model's 's input. Note that while we view a; as a single integer. the model will see it as a sequence of digits after character-
tokenization.
100
100
Data Formatting. In the following sections, we will delve into the four data formatting approaches in our arithmetic experiments. See Figure I and Ap- pendix J for examples. In Section 4, we explore the limitations of the conventional plain-forma data and demonstrate how simple reversal of the output order can lead substantial performance improve- ments and enhanced sample efficiency. We introduce two Lemmas to support and explain these findings. Additionally, in Section 6, we present results on the simplified and detailed scratchpad formats, highlight- ing significant enhancements in sample efficiency for learning addition. We also emphasize the importance of carefully designing the intermediate steps in the detailed scratchpad method. Note that the scratch- pad formats are largely adopted from the literature of chain-of of-uuut (CoT) training (Nye et al., 2021; Zhou et al., 2022b).
Random
Test Accurscy (%)
80
99
Balanced
Delanced Digits
Balanced
40
96
20
Overall 1-digt 2-digt
03 Overall carr-11 carry22 cary-3
Figure 2: Performance of 3-digit addition on vari- ous data sampling methods used: (i) Random: uni- form sampling of operands; (ii) Balanced digits: assigning higher sampling weights to operations involving and 2-digit numbers: (iii) Balanced carry: balancing the dataset to contain an equal number of carry-on operations; (iv) Balanced both: balancing digits and carry-ons. We observe that balanced data improves accuracy compared to random sampling. Experiments on addition with the 'S' symbol wrapped for each sample.
Structured Data Sampling. While data formatting plays a crucial role, we also discover that choosing the samples carefully is also essential. When sampling operands for n-digit addition uniformly at random between to 10" - 1, the dataset inevitably becomes highly skewed in terms of the number of samples with (i) operands containing certain number of digits and (ii) operands
4

---

Published as a conference paper at ICLR 2024
resulting in certain number of carry-on" operations. For instance, in the of 3-digit addition, random sampling results in a meager 0.01% probability of selecting a 1-digit number. Additionally. 1 or 2 carry-on operations more likely to occur than 0 or 3. To address this imbalance, we employ structured sampling approach. Specifically, we aim to (i) balance digits by assigning higher weights to lower-digit numbers during the sampling process and (ii) balance carry-ons by ensuring an equal distribution of examples with 0, l....... n. carry-on operations. When sampling 10, 000 examples of 3-digit addition, we include all 100 1-digit additions, 900 2-digit samples and 9000 3-digit samples. Note that while the number of samples increases, the fraction of all possible k-digit additions that we sample for k "reasonable" fraction of all possible k diiit samples for all k. Similarly, we ensure that the number of samples with 0, 1, 2, or 3 carry-ons are all approximately 2500.
3 decreases due the inherent skew. The split was chosen ensure we saw a
Figure 2 reveals the importance of balancing. We observe improvements in accuracy across the board while using balanced data when compared to random sampling. Further, random sampling performs relatively poorly even for the simple task of 2-digit addition, possibly due to the fact that the model has not seen enough of these examples. For the remaining experiments, we set the default dataset for addition to be one that has both balanced digits and carry-ons.
4 DATA FORMAT CHALLENGES AND ARITHMETIC EMERGENCE
We start by examining integer addition. We first focus on 3-digit addition, i.e., where the two summads have at most 3 digits (< 999). Later, in Section 7, we extend our findings to numbers with up to 10 digits. Surprisingly, teaching addition can be more complex than expected.
Training on Conventional Data. We start by training NanoGPT on standard addition data represented as 'A3A2A1 + ByByB, C3C,Ci', : termed the plain format. However, shown in Figure I, this leads to fairly poor performance. We suspect that this is because the next-token prediction objective outputs the most significant digit (MSB) first. The following lemma clarifies the necessity to access all operand digits for outputting the MSB first
Lemma Let A and B be two n-digit numbers, and let A Suppose an algorithm A outputs the digits of C in decreasing order of significance, then A must have access to all digits of A and starting from the first digit that it outputs.
The lemma suggests that to train a model for addition and to output the MSB first, it is necessary to emulate a "global" algorithm. Unlike the standard "local" algorithm for addition, which consists of computing digit-wise sums and carry-ons, approximating the global algorithm would require learning a more complicated function than necessary. The increased complexity results in decreased accuracy, as observed in our experiments. Liu et al. (2023) refer to this phenomenon attention glitches.
Reversing the Output. We propose that the reverse format "SAgA2A1 + B3B2B; = C2 is more suitable for noxt prediction models. The rationale behind this is that when generating the sum by starting with the least significant digit (LSB), the model only needs to learn a local function of three inputs per digit- - the two relevant digits of the operands and the carry-on from the previous digit. This local operation simplifies the function to be learned. The following lemma formalizes this:
+ B for two n-digit numbers A and B and outputs its digits in increasing order of significance such that, at each position i, the algoritlum only requires access to the 1 digits of A and B, as well as the carry-on from the previous position.
Lemma 2. There exists an algorithm that computes C = A
Lemma 2 directly follows from the standard algorithm for addition, which performs the sum and carry-on operations digit by digit. The implications of these lemmata are evident in our experiments when comparing the accuracy of the plain and reverse formats. As shown in Figure 1, training on reversed outputs significantly enhances accuracy, with considerably fewer samples. What is particularly remarkable is the rapid emergence of addition occurring between Ik 4k samples for reverse. During that, the model rapidly transitions from being unable to add two numbers to being capable of perfectly adding. This leads us to ask:
PIn this paper, we adopt the definition that carry-on operation involves transferring information from one digit position to another position of higher significance. Therefore, we refer to the "borrow" operation in subtraction as a carry operation.
We use '8" symbol for data delimiter for the reverse format. Refer to Appendix B.I for details.
5

---

Published as a conference paper at ICLR 2024
Why does addition rapidly emerge as the number of training examples increases?
MATRIX COMPLETION: AN INCOMPLETE TALE OF EMERGENCE
Although the rapid transition observed in the previous section may initially seem surprising, closer examination reveals a fascinating equivalence learning an addition map on n digits from random samples be considered as completing a rank-2 matrix. Establishing this connection with low-rank matrix completion (LRMC) provides meaningful insights into the observed phenomenon. However as we explain in this Section, this connection does not tell the complete story, and Transformers possess generalization capabilities far beyond what LRMC would predict.
Learning addition tables Matrix Completion. Learning addition from samples + J can be formulated as rank-2 Matrix Completion (MC) problem, where we partially observe an n x matrix M, whose (i. 1-th entry represents + j. M can be decomposed into the sum of two rank-1 matrices, N1 1N where N is a vector with entries {l.... n). and 1 is the vector of ones, Recovering a rank-2 matrix, in the absence of noise, can be sample-optinally performed by a simple iterative algorithm from Király et al. (2015) (Algorithm I in Appendix B.2). As depicted in Figure 3a, a rapid transition occurs at O(n), a well-known matrix recovery phenomenon (Recht, 2011).
We notice a similar rapid transition in NanoGPT. To investigate it, we focus on 2- digit addition (i.e., n 100) and evaluate the performance of learning addition through NanoGPT and LRMC (Figure 3a) by con- structing train data as the revealed entries of the M matrix. Note that the dataset is no longer balanced, as the revealed en- tries are randomly sampled for LRMC exper- iments, to match the standard MC probabilis- tic settings Recht (2011). In Figure 3b, both NanoGPT and LRMC exhibit rapid transitions at approximately 1500 samples.
(a) MC of Addition Matrix
(b) Comparing LRMC NanoGPT
1.
100
Success Probabuiity
0.
80
Thxt Accuracy
n= 50
0.4
100
5
n 500
rewerss
matrix completion
101
10'
121
124
1630 2000 5050 40t0 5000 Number of train examples
Number of Reveaked Entries
Figure 3: (a) We run Algorithm 1 (Király et al., 2015) on the addition matrix for n = 20, 20,50, 100, 500 and report the success probability while varying the number of re- vealed entries. As expected, sharp transition occurs when approximately (n)) entries are revealed. (b) We compare the performance of NanoGPT trained on a dataset con- taining 100 samples (i.e., 2-digit addition) to that of the corresponding LRMC problem using the same sample set. Remarkably, at 1500 samples, both NanoGPT and Algorithm hegin learning addition almost flaw flwlessly.
While the observed rapid transition can be attributed to the principles of LRMC, shed- ding light the emergent arithmetic skill in NanoGPT, this connection falls short of cap- turing the full generalization capabilities displayed by NanoGPT.
GT generalizes better than Matrix Completion solutions. Upon further investigation, we find that NanoGPT exhibits capabilities beyond LRMC. Notably, LRMC is constrained by its inability generalize in the presence of missing rows or columns. In our context, this equates to certain numbers being omitted from the training data. To assess NanoGPT's ability to overcome this, we deliberately exclude specific numbers or digits from our training data and assess the model's ability in learning addition. Can the model still generalize to unseen numbers?
As shown in Table 1, the answer this question is resounding Yes! The model achieves almost per- fect accuracy even when excluding half of all possible 3 digit numbers. NanoGPT can successfully learn 3-digit addition even when numbers or digits are intentionally excluded from the training data, thereby exhibiting generalization capabilities that far exceed what standard LRMC would predict.
Table 1: Impact of excluding numbers on addition task: NanoGPT models trained with 100 100/200/550 200/500 excluded operands show no significant drop in accuracy and in some cases, the performance even improves.
Excluding 100 numbers
Excluding
Excluding 500 numbers
No Exclusion
200 numbers
Plain
Rev
Plain
Overall Acc.
(2.65%((2253 29.877
93. (3.369((2266) 99 99.8291420022) 93.61 99.78%128117 (3.47%(23.22) 100.0%(=0.00
Exclusion Ace.
14.43%(21110) 99.87%(1)024) 15.25%((11.60 99 88%)(10008) (4.39%(2225) 100.0%(00.00
Specifically, we randomly choose 100/200 200/200/500 numbers and exclude them from the training data. We then evaluate the trained models using two metrics: (i) Overall accuracy: which measures the

---

Published as a conference paper at ICLR 2024
accuracy over random set of 10, 000 examples and (ii) Exclusion accuracy: which measures the accuracy only over the excluded set (where either of the two operands is one of the excluded numbers). Remarkably, excluding numbers from the training data sometimes leads to improved performance. We conjecture that this may be due to a regularization effect, similar to random masking or cropping images in vision tasks. In Appendix B.2.1, we further find that NanoGPT models can even generalize to unseen digits, where a digit is absent from particular ordinal position.
6 TRAINING ON CHAIN-OF-
DATA EXPEDITES EMERGENCE
So far, observed that simply reversing the output can result in remarkable performance, exceeding that of LRMC in learning addition. Here, we investigate if it is possible to expedite the emergence of addition by further enhancing the data format. As addition the idea of incorporating additional information about each intermediate step. We adopt a CoT style approach, where we guide the model to learn addition step-by-step. We explore two levels of detail in the provided instruction steps, as shown in Figurel: (i) Simplified Scratchpad with minimal information the sum and carry information for each digit/step. (ii) Detailed Scratch- pad with comprehensive information on detailed traces of execution for each intermediate step.
multi-step process, we explore
The results in Figure 4a show that the model trained on simplified scratchpad achieves 100% accuracy with only 2000 samples, whereas reverse requires more than twice as many. Detailed scratchpad, which provides even more fine grained information, achieves perfect addition with just 1000 samples. This indicates clear message: incorporating more in- formation enables the model to learn addition with far fewer examples. We conjecture that this is because breaking down the required compositional function to be learned into individual, simpler components allows the model to learn higher-dimensional but easier-to-learn function map, in agreement with re- cent theoretical work (Li et al., 2023; Malach, 2023).
Sample Efficiency
(b) Efficiency
102
120
sddMtion
reverse
22
2
simplitied scratehpad.
dztalled scrochhoos
28 tr ecamples
1231
Number tokenss
Figure 4: (a) Comparison of sample efficiency: evaluating performance training dataset with different numbers of addition samples. While all variants other than plain achieve 100% accuracy. they differ in terms of sample complexity. (b) Num- her of tokens in train datasct required hy NanoGPT to learn addition. Reverse is the most efficient in terms of token usage for model training, as the scratchpad methods, although more sample- efficient, require more tokens per sample.
We note that while CoT-style training enhances sam- ple efficiency, it may not necessarily be the most "token-efficient" approach. To account for the cost associated with training and inference, we conduct cost analysis based on the number of tokens (number of training samples x number of tokens per sample- see Appendix G for details) in the train data. encountered during training. The result in Figure 4b shows that reverse is the most efficient in terms of token usage for model training. The scratchpad methods, although more sample-eficciet, require more tokens per sample.
In summary, incorporating scratchpad data and decom- posing the addition task into steps offer a promising strategy to improve the performance and efficiency of
100
Test Accuracy (%)
80.
small models in learning addition from scratch. Nev-
60
ertheless. for practical usage, it is crucial to evaluate both the number of samples for achieving the desired performance and the actual token requirements during training and inference.
plain
40
reverse
simplified scratchpad
20
detaiied scratchpad
digit
7:digit d0
50k
100k 150k
200k
Number of train examples
7 LONGER DIGITS AND BLENDING ARITHMETIC WITH SHAKESPEARE
Figure 5: Comparison of sample efficiency for
7, and 10-digit additions. Training on plain requires an increasing number of samples for higher digits, while the sample complexity for other data formats remains relatively consistent.
In this section, we go beyond 3-digit addition to compass a wider range of arithmetic tasks and longer digits to show that our insights data sampling and formatting hold true even in this regime. We also explore the effect of mixing arithmetic with text data, and few-shot prompting.
7

---

Published as a conference paper at ICLR 2024
Extending to longer digit addition. We repeat the experiment from Section 3 with up to 10 digit integers. Figure 5 shows that the behavior of all data formats remains similar across varying number of digits. In fact, the performance gap between the modified formats and plain grows with longer digits. While plain requires an increasing number of samples to learn higher-digit additions, the reverse and seratchpad formats maintain a consistent sample complexity. We also observe similar results in the fine-tuning setting, where we fine-tune a model initially trained on k-digits on k k+ 1-digit data. See Appendix C for details on the experimental setup and additional results.
Mixing Text with Arithmetic Data. While
(a) Plain
(b) Detailed scratchipud
the models far were trained exclusively arithmetic tasks, in practice, L.LMs utilize a com- bination of arithmetic and text data for training. How does that affect the emergence of arith- metic skills? To explore that we incorporate both addition samples and text into our train data and evaluate the models with few-shot prompt- ing (showing a few examples of addition in the prompt) to see if it is able to be effectively con- ditioned for the appropriate context (arithmetic/- text generation). As we see in Figure 6, we find that few-shot prompting improves the per- formance of the model, allowing it to perform addition accurately even in the plain format.
Test Acourac 19))
40
Accurnity (94)
40
Zerovshot
1-shot
Zero shot
2-shot
2-shot
shsht
40
text_prompt
shot
26
prompt
56 10k 15k 20k 35c 306: 43x Additioon Samples
1502 1300 2003 2000 5000
Scratchpo Samples
Figure 6: Performance of NanoGPT model trained with the Shakespeare dataset, addition datasct in plain, and detailed scratchpad format. The number of plain (left) and detailed scratchpad (right) formatted samples are varied. Performance is evaluated on zero-shot, few-shot, and text prompts, with the shaded area representing the standard deviation across various prompt exemplar sets.
Intriguingly, accuracy remains high using plain even with the inclusion of a text prompt preceding "A+B=". This is likely due to the structure of our mixed dataset where addition examples interspersed within Shakespeare text. With the incorporation of more addition examples, instances where addition follows Shakespeare text increases, leading when text content is present during addition test queries. We further analyze the effect of text on prompting for both cases with and without text in the training data in Appendix E.
decrease in potential inconsistencies
Teaching arithmetic operations beyond addition. We also note that the results on addition also hold for broader mathematical operations such as subtraction, multiplication, sine, and square root operations where each operation entails its unique challenges and intricacies. Refer to Appendix D for detailed settings and results.
8 FINE-TUNING, SCALING, AND PRETRAINING IN LARGER MODELS
We extend our study from NanoGPT to larger models like GPT-2 and GPT-3 to explore the impact of pretraining and model size. Initially, we compare the performance of NanoGPT and GPT-2, both trained from scratch. This high-
Test Accuracy (%)
NanoGPT, charl scratch
GPT-2, level scratch
GPT-2,
tiktaken, scratch
GPT:2
tiktoken, pretrained
GPT:2,
tiktoken scratch
lights the ad advantages of larger model scales, es-
GPT-2
titoken+ praepri-eece..peerrrieeed
pecially in zero-shot scenarios. Subsequently. we delve into the impact of tokenization meth- ods and model pretraining in GPT-2 models. We then fine-tune pretrained GPT-3 on var- ious arithmetic tasks using different data for- mats, reaffirming the importance of data for- matting for larger pretrained models.
1000
2000
3000
4000 5000
Train Iterations
Figure 7: Performance of various configurations of the GPT-2 model on the addition task. We compare the ef- fects of tokenization methods, specifically character-level tokenization versus Tiktoken (OpenAl's BPE tokenizer), training initialization (training from scratch versus train- ing from a pretrained GPT-2 model), and the inclusion or exclusion of spaces between numbers. The results highlight the significance of utilizing pretrained models and incorporating spaces for consistent tokenization of numbers when training model for arithmetic tasks.
Comparing NanoGPT and GPT-2: Tok- enizer and Training Pretrained Models We repeat our experiments on GPT-2 model with 85M parameters, with twice as many lay- ers, heads, and embedding size compared to NanoGPT. The transition to a GPT-2 setup ne- cessitates several modifications. Firstly, we shift OpenAT's Tiktoken BPE tokenizer. We also
8

---

Published as a conference paper at ICLR 2024
examined two different training approaches: training the model from random initialization (scratch) and fine-tuning the pretrained model sourced from Huggingface. To circumvent potential inconsistent tokenization of numbers, alterations were made in data formatting to include spaces between numbers.
Figure 7 shows that GPT-2 demonstrates high performance in addition tasks with both character-level tokenization and Tiktoken with spaces between digits. This aligns with the results by Wallace et al. (2019), suggesting that character-level tokenization exhibits stronger numeracy capabilities compared to word or sub-word methods. Furthermore, comparing the models trained from scratch and the models trained from the pretrained model, we observe that fine-tuning pretrained model results in better performance compared training a model from scratch.
GPT-3 experiments. We consider three GPT-3 variants: Ada, Curie, and Davinci (OpenAI). We tne-tune these models using the same four data formatting methods as our NanoGPT experiments.
Table 2: Evaluation of addition performance for fine-tuned GPT-3 models: Davinci, Curie, and Ada. In cach case, the model is finetuned on 1000 samples of addition in the corresponding format.
GPT-3 Model
Zero-shot
Plain
Reverse
Simplified Scratchpad Detailed Scratchpad
Davinei
2%
34%
80.9%
88.7%
99.5%
Curie
0.0%
1.4%
12.3%
10.7%
99.7 %
Ada 0.0%
0.3%
6.3%
0.6%
99.8 %
The results in Table show that starting with pretrained GPT-3 significantly improves performance compared to training NanoGPT or GPT-2 from scratch with only 1000 examples (Figure 4a). Similar to the result of training NanoGPT from scratch, the modified formats all outperform the plain format. Detailed scratchpad data achieves near-perfect accuracy, albeit with increased training and inference costs due to higher context length requirements. For detailed experimental setup and further experiments on larger models, including fine-tuning GPT-3 refer to Appendix F.
LIMITATIONS
Length generalization. In our experiments, we did not observe any instances where the model could predict beyond the number of digits it had been trained on (see Appendix H). Shaw et al. (2018); Sun et al. (2022) reported similar difficulties and proposed approaches such relative positional encodings. Anil et al. (2022) suggests that models can only perform out-of-distribution tasks by combining fine-tuning, prompting, and scratchpad techniques.
Model/Data scale. Due the smaller scale of our experiments, we were able to thoroughly examine the impact of individual components on the model's s arithmetic learning capabilities. Our model was limited to decoder-only architectures, primarily focusing on character-leve tokenization. Although we have some preliminary results on scaling up and incorporating BPE-based tokenization, is not clear if all our findings can be generalized to the scale of LLMs being used in practice.
Beyond elementary arithmetic. We choose to analyze simple arithmetic operations in order to carefully isolate factors that contribute to emergence. While the existing literature has already demonstrated the emergence complicated abilities practice, our work seeks to provide a better understanding of this behavior by extensive ablations in controlled setting.
10 CONCLUSION
In this study, investigate teaching arithmetic operations small randomly initialized transformers using the next-token prediction objective. We carefully ablate different aspects of the training setting so as isolate the factors that contribute to the emergence of arithmetic capabilities. Our results reveal that traditional training data is sub-optimal for learning arithmetic, and training on data with detailed intermediate steps or even simply reversing the output improves accuracy and sample complexity. We also study the effects of few-shot prompting, pretraining, and model scale. Despite improvements from detailed data, length generalization remains a challenge, highlighting the need for better-curated training data to ensure successful learning of specific algorithms opposed to just learning an approximate function map. The correct approach for learning multiple arithmetic operations, of different levels of complexity, is still unclear. We anticipate that this research will contribute to more nuanced understanding of the mechanisms by which transformers (approximately) acquire algorithmic skills.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the Institute for Foundations of Data Science (IFDS), ONR Grant No. N00014-21-1-2806, N00014-23-1-2845 and grant by FuriosaAI.
REFERENCES
Cem Anil, Yuhuai Wu. Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur, Exploring length generalization in large language models. arXiv preprint urXiv:2207.09901, 2022.
Samuel R Bowman. Can recursive neural tensor networks learn logical reasoning? arXiv preprint arXiv:1312.6199 2013.
Samuel R Bowman, Christopher Potts, and Christopher D Manning. Recursive neural networks for learning logical semantics. CoRR, abs/1406. 5, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2rXiv:2303.12712 2023.
Jonathon Cai, Richard Shin, and Dawn Song. Making neural programming architectures generalize via recursion. arXiv preprint arXiv:1704.06611, 2017.
François Charton. Linear algebra with transformers. arXiv preprint urXiv:21 2021.
François Charton. What is my math transformer doing?- three results on interpretability and general- ization. arXiv preprint arXiv:2211.00177 2022.
Xinyun Chen, Chang Liu, and Dawn Song. Towards synthesizing complex programs from input- output examples. arXiv preprint orXiv:1706.01284 2017.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05/28 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint urXiv:2204.02311 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, et al. Scaling instructionffieetuee language models. arXiv preprini arXiv:2210.1 /416, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprini
2018.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song. Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models. arXiv preprini arXiv: /5003, 2022.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality, 2023.
10

---

Published as a conference paper at ICLR 2024
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aides language models, 2023.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in pre-trained language model. arXiv preprint urXiv:2305.0000 005866 2023.
Samy Jelassi, Stéphane d' A'scooli, Carles Domingo-Eni Enrich, Yuhuai Wu, Yuanzhi Li, and François Charton. Length generalization in arithmetic transformers, 2023.
Eukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. arXiv preprint arXiv:1511.08223 2015.
Andrej Karpathy. char-ran. https: http:///hhhh..ccco///aaatttttaar 2015.
Andrej Karpathy. Andrej karpathy's lightweight implementation of medium-sized gpts. GitHub, 2022. URL https: : giigttt........... com/ wwt.ccooo/aaaaaaaaae karpathy/naaa /nanoGTT.
Jeonghwan Kim, Giwon Hong, Kyung-min Kim, Junmo Kang, and Sung-Hyon Myaeng. Have you seen that number? investigating extrapolation in question answering models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7031-7037, 2021.
Franz J Király, Louis Theran, and Ryota Tomioka. The algebraic combinatorial approach for low-rank matrix completion. J. Mach. Learn. Res., 16(1):1391-143) 2015.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sesquence-t-ssqquee recurrent networks. In International conference machine learning, pp. 2873-2882 PMLR, 2018.
Yingcong Li, Kartik Sreenivasan, Angeliki Giannou, Dimitris Papailiopoulos, and Samet Oymak. Dissecting chain-of- tf-toughhh A study on compositional in-context learning of mlps. arXiv preprint arXiv:2305. 18869, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:/705.04: 146, 2017.
Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Exposing attention glitches with flip-flop language modeling. arXiv preprint arXiv:2306.0080 00946, 2023.
Eran Malach, Auto-regressive next-token predictors universal learners, 2023.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.128837 2022.
MosaicMI.. Introducing mpt-7b: A new standard for open source, commercially usable llms, 2023. URL www . mosai cm1 com/b /bl og 1t/m//pt7777 Accessed: 2023-05-05.
Rodrigo Nogucira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations of transformers with simple arithmetic tasks. arXiv preprint 2rX.v:2102.13019 2021.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:21 2021.
Santiago Ontanón, Joshua Ainslie, Vaclav Cvicek, and Zachary Fisher. Making transformers solve compositional tasks. arXiv preprint 1rXiv:2108.04377 2021.

---

Published as a conference paper at ICLR 2024
OpenAI. OpenAl platform. URL attpss//ppaattmmm..pppa....oooo
gpt-3. Accessed: 2023-09-28.
Joshua Peterson, Stephan Meylan, and David Bourgin. Open clone of openai's unreleased webtext dataset scraper. GitHuu, 2019. URL https://githul. cot.b/ocoooooeooooooooooeoo jcpeterson
openwebtext.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 2023.
Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. Limitations of language models in arithmetic and symbolic induction. arXiv preprint arXiv:2208.05051, 2022.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:220..07266 2022.
Benjamin Recht. A simpler approach to matrix completion. Journal of Machine Learning Research, 12(12), 2011.
Scott Reed and Nando De Freitas. Neural programmerrinterrretern arXiv preprint drXiv:151100279,, 2015.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv:1608.014:3 2016.
arXiv preprint
Timo Schick, Jane Dwivedi-Wo Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXiv: 1203.0215, 2018.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models mul- tilingual chain-of-thouuhh reasoners. arXiv preprint arXiv:2210.03007, 2022.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alons et Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.00 04615. 2022.
Yutao Sun, Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrraolltable transformer. arXiv preprint arXiv:2212.10554, 2022.
Ilya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng. Jinfeng Rao, Aakanksha Chowdhery, et al. Transcending scaling laws with 0.1% extra compute. arXiv preprint arXiv:2210.11399, 2022.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201. 008399. 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
12

---

Published as a conference paper at ICLR 2024
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-basee feedback. arXiv preprint arXfv:2211.14275. 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. Do nlp models know numbers? probing numeracy in embeddings. arXiv preprint
2019.
Cunxiang Wang, Boyuan Zheng, Yuchen Niu, and Yue Zhang. Exploring generalization ability of pretrained language models on arithmetic and logical reasoning. In Natural Language Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2021, Qingdao, China, October 13-17, 2021, Proceedings, Purt I 10, pp. 758-769. Springer, 2021.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 111,2,
Taylor Webb, Keith Holyoak, and Hongjing Lu. Emergent analogical reasoning large language models. arXiv preprint arXiv:2212.091930 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 11903, 2022b.
Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Guo, Jinfeng Bai, and Jie Tang. Gpt solve mathematical problems without calculator, 2023.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02013. 2023.
Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv:1410.4615, 2014.
Wojciech Zaremba, Karol Kurach, and Rob Fergus. Learning to discover efficient mathematical identities. Advances in Neural Information Processing Systems, 27, 2014.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint 2rXiv:2205.1066 2022a.
Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint (tt.::211.090666. 2022b.
13

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Proofs
15
B Additional Experiments
B.1
Zero-Padding, Symbol Wrapping, and Testing
B.2
Low-Rank Matrix Completion
17
B.3 Supplementary Experiments Addition
18
B.4 The Importance of Intermediate Step Design
19
B.5 The Effect of Noisy Inputs on Accuracy
21
B.6
Analyzing the results on Sine/Sqrt
23
Extending to Longer Digit Addition
24
C.I Training from Random Initialization
:iiiaaa.........................
24
C.2 Fine-Tuning from Pretrained Models
Nec.............................
25
C.3 Impact of Formats on Fine-Tunde Fine-Tuning
27
D Teaching Arithmetic Operations Beyond Addition
28
D.1 Extended Arithmetic Operations Jointly Training on Five Arithmetic Tasks
28
E Mixing Text with Arithmetic Data
32
E.1 Few-Shot Prompting.
32
E.2 Disentangling the effect of text on prompting
32
E.3 Prompting with Text
33
F
Fine-tuning, Scaling, and Pretraining Larger Models
36
G Token Efficiency Across Data Formats
38
H Length Generalization
40
Experimental Setup
43
L1
Dataset
43
I.2
Model
46
1.3
Training
46
I.4
Hyperparameter Configurations
47
J
Prompt Examples
48
J.1
Addition
48
J.2
Subtraction
50
J.3
Multiplication
51
J.4
Sine
51
J.5
Square Root
52
J.6
Noisy Simple Scratchpad
52
J.7
Example data for GPT-3 fine-tuning
53
14

---

Published as a conference paper at ICLR 2024
A PROOFS
Here, we present the proofs of Lemma and 2.
Lemma Let A and B be two n-digit numbers, and les C= = A+ B. Suppose an algorithm A outputs the digits of C in decreasing order of significance, then A must have access all digits of A and starting from the first digit that it outputs.
Proof. We begin by assuming for contradiction that there does exist an algorithm Ao that does not have access to all digits of A and B and still outputs C A + B correctly for all n- digit numbers A, B. Without loss of generality, say Al go does not have access the k-th digit of A where k E [n] represents the position counting from the least significant digit. Then consider the example = (10" 1) and (A = 000 ... A0 ... where B is just the integer with n 9's and A just O's with Ag in the kth position. If A O, then Cn+1 O, but if A 1, then Cn-1 1. Therefore, without access to the k-th digit of A, there exist examples where the algorithm will surely make mistake. Therefore, by contradiction such Algo cannot exist.
Lemma 2.
There exists algorithu that computes C A + B for two n-digit numbers A and B and oupput its digits increasing order of significance such that, at each position i, the algorithm only requires access the it digits of A and B, as well as the carry-on from the previous position.
Proof. First note that the trivial algorithm for addition is exactly the proof this Lemma. However, we present more formal argument below for completeness. Let A, B be n-digit numbers and A+ B be at most an (n + 1) digit number. Define the digits of A, B, and C' as A,, B;, and C., respectively, for i [n]] counting from the least significant digit once again. Then, the addition can be performed using the following steps. First, C; (A; + B; + carry:) mod 10 where carry; is the there no carry from the previous position, A;+ B,+earry
carry-on from the addition of digits at position
then carry:
0. The carry for the next position is then calculated as carry;+1
10
Putting this together, the algorithm for addition can be described as follows:
Step 1: Set carry1 0. Repeat for i {1...., n): (Step 2: Compute C, C(AA; B, + carry:) mod 10 and carrys+
+coorry Step 3: Output C:) Step 4: Output C.+11 carrya+1.
is easy to see that this algorithm computes the digits of the sum C' correctly and requires only the individual digits at position and the carry from the previous position. Therefore, this algorithm satisfies the conditions of the lemma.
15

---

Published as a conference paper at ICLR 2024
B ADDITIONAL EXPERIMENTS
B.1 ZERO-PADDING, SYMBOL WRAPPING, AND TESTING
Zero-padding and Symbol wrapping As discussed briefly in Section 3, we found significant benefit to using padding for multi-digit addition. Throughout our experiments, we use the plain format without any such padding (denoted as "vanilla" below) as the default baseline representing the conventional data format used in training. Nonetheless, we explore modifications to this plain format to enhance performance; zero-pas pro-ppdding, and wrapping with a single symbol. Zero-padding ensures a fixed length for operands and the output. In the case of 3-digit addition, this means 3-digit operands and a 4-digit output. For example, '112 + 29 = 141' becomes '112 + 029 = 0141'. As shown in Table 3. this modification significantly improves model performance. Next, wrap each sample using the " symbol as '$112 + 29 = 141$'. We found this performs on par with zero-padding.
As result, adopt the 'S' symbol for efficient data delimiter, extending its Figure shows 'S' -wraping also enhances the performance of the reverse format. Despite the plain format being improved with the '$' delimiter, it remains short of the reverse format's s accuracy and sample efficiency. We continue to maintain the original plain format as baseline since it not only exemplifies conventional data but further emphasizes the need for improved data formatting to ensure efficient training. As such, for the reverse format, we have incorporated the 'S' delimiter in our formatting modifications.
the reverse format.
Table 3: Test accuracy NanoGPT model 3-digit addition trained 10, 000 samples of plain format data, comparing (i) vanilla format without modifications, (ii) Zero-padding format, and (iii) 'S' "Sr-wrapped format. The results show significant performance enhancement through sero-padding for fixed length and similar improvements when deploying single-symin wrapping.
Vanilla
Zero-pad
'S' ''-Wrappee
88. 38.17%
97.74%
97.76%
100
100
Test Accuracy (%)
80
Test Accuracy (%)
80
60
60
40
40
reverse, with wihout
reverse, with wihout plain, with plain, wioou $
20
20
plain, with
plain, wihout
0
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Number of train examples
Number of train examples
Figure 8: Performance of NanoGPT (left) and GPT-2 (right) models on 3-digit addition using plain and reverse format, both with and without 'S' delimiter. The addition of the ' symbol noticeably enhances performance in both formats. Nevertheless. the plain format underperforms compared to the reverse format, particularly terms of sample efficiency. While we maintain the original plain format as bascline emphasizing the necessity for improved data formatting for efficient emergence we incorporate the "š' wrapping in our modificd reverse format.
16

---

Published as a conference paper at ICLR 2024
B.2 LOW-RANK MATRIX COMPLETION
In our Low-Rank Matrix Completion experiment for the addition matrix (which of rank-2), we employ an iterative algorithm proposed by Király et al. (2015). This algorithm systematically searches for a 2 x 2 submatrix in which three entries are known and one entry is unknown, It then fills the unknown entry to ensure that the determinant of the 2 submatrix becomes zero, where the solution is known to be optimal. We present the full pseudo-code in Algorithm 1.
To assess the performance of the algorithm, we generate of n (e.g., 20, 50, 100, 500). We vary the number of revealed entries, randomly sampling sparse matrix where only a specified number of entries between and n known, while the remaining entries are set to zero. We repeat this process 100 times for each number of revealed entries, tracking algorithm's success or failure in finding the solution. We calculate average success rate across the trials and present the success probabilities in Figure 3a, where observe sharp phase transition when (n) entries are observed, as expected.
addition matrices for various values
Algorithm 1: Iterative 2 x Matrix Completion Algorithm
Data: Data Matrix M ==** with partially revealed entries, Assumed to be of Rank 2.
Result: M E R=*= Success/Fail
ni represents number of resolved submatrices.
v2 <- 0 represents number unresolved submatrices.
M < M
while n1 2 1 do
/* As long as resolved at least submatrix in the previous iteration
m1 4-0
n2 0-
for iori=1 to do
for = 1 to do
do something
if M.,, is not revealed and all its neighbors are revealed then
M. M,+1, M,54
M M:41.111
+1
if M revealed and all its neighbors are revealed then
M.11 M. xM
Meri
in < n1
16
if M is not revealed and all its neighbors are revealed then
M Met1.5 x M,,411
M...
+ m1 +1
if M. j. M.-1s, Misti. M are all revealed then
continue
21
else
+ +
> 0 then
return M, Fail
25
else
26
return M, Success
B.2.1
GENERALIZING UNSEEN DIGITS
Building upon the model's robustness to excluded numbers, we further investigate its ability to handle excluded digits, where digit is absent from particular ordinal position. Intuitively, this should
17

---

Published as a conference paper at ICLR 2024
be more challenging since excluding digit means the model cannot learn directly how to operate in that position. Instead, would have to generalize and infer that digits act similarly across all positions. We construct datasets with the number 5 excluded in 1st (LSB), 2nd, and 3rd (MSB) positions, and train separate models on each of these datasets. We compare the resulting models by evaluating overall accuracy accuracy specifically on samples with 5 in each position which we call exclusion accuracy.
test set of 10, 000 randomly sampled numbers, well their
The results presented in Table 4 indicate that the model not robust excluding digits compared to excluding numbers. However, it still achieves more than 66% accuracy on every test and maintains an overall accuracy above 85%. Moreover, it appears that excluding number in the least significant position yields the worst performance. This can be attributed to the fact that learning addition in this position is transferable to other positions since it is unaffected by carry-on operations. Failing to learn addition in this position, however, will have detrimental impact on other positions as well.
Table 4: Impact of excluding digits addition task: We investigate whether GPT-based models can infer addition on an excluded digit a specific position from training data on other positions. We compare NanoGPT models trained with and without an excluded digit and find that excluding digits is harder to learn hut not entirely impossible, with the worst performance observed when excluding the least significant digit.
Input
"5" in the 1st (L.SB) digit
"5" the 2nd digit
"5" in the 3rd (MSB) digit
Excluded position
format
Overall Acc
Plain
92.65%(+2.5)
92.58%(++29)
93.59%)(++257
95.16 15.116
No exclusion
Reverse
99. 99.87%(4-22 +-024)
99.89%(++0.0)
99.95%(++..)
99.97% 99.97% (.004)
Plain
93.87 93.87%ct114 1,64)
81.8 96 (1:7.11
94.04 (5))
94.22%ii111
(LSB) digit
Reverse
98.00% 1.50)
90.32 (116.44
98. 98.22%6c11. 1.77)
97.86 1366 1.54)
Plain
92.53%
90.46%cc1266
83.87 % (:1.3.7)
94.80%'
2nd digit
Reverse
97.91 1.47)
98. 18
91.4 (1b6.1))
08.93%r.o..cm
Plain
(0.85%%(42mm
89.23%
90,94%
85.88 % (=455
3rd (MSB) digit
Reverse
98.84%ct...0
98.7%
98.78%(tt
94.47 % 1:99)
B.3
SUPPLAMENTARY EXPERIMENTS ON ADDITION
B.3.1 COMMUTATIVITY IN ADDITION
We explore whether NanoGPT, trained with a plain data format on 10,000 training samples, captures the commutativity property of addition. This inquiry involves testing the equation (A+ B) = (B + A) across 9,900 instances.
As shown in Table 5, out of these 9,900 test cases, 8,799 samples exhibit commutative behavior. While majority (8,700/8,79) 7,90// of these instances are straightforward - the model correctly computes both (A + B) and (B + A) it is noteworthy that among 164 cases where both outcomes are incorrect, 99 (60.3%) still preserve commutativity. This suggests a considerable, though imperfect, understanding of commutativity post-full training.
Table 5: Commutativity analysis of NanoGPT on 3-digit addition, examining the validity of (A+ B) (B+ for 9,900 test examples.
P(A + B B A, both correct) P(A + B and B A both incorrect) P(A + B = B + A hoth incorrect)
0.878 (8,700 samples)
0.016 (164 samples)
0.603 (99/164 samples)
B.3.2 ADDITION INVOLVING SIGNED NUMBERS
While our primary focus has been on adding positive numbers, we posit that these results can extend to signed numbers, which effectively encompass both addition and subtraction operations. We conduct a preliminary experiment, training the model on various signed combinations of two 3-digit numbers.
Figure 9 shows that training on signed numbers presents more challenging task compared to training on addition or subtraction on only positive numbers. Notably, test accuracy for addition involving same-sign operands, (+, +) and (-, -), surpass those involving opposite-sign operands, (+,-) and
18

---

Published as a conference paper at ICLR 2024
(-,+). This observation aligns with the expectation that learning addition is relatively simpler than learning subtraction.
100
Signed Addition with 20k examples
100
Test Accuracy (%)
80
95
60-
Test Accuracy
40
addition
20
subtraction
70
signed addition
65
0 2.5k 5k 7.5k 10k 12.5k 15k 17.5k 20k Number of train examples
+,+
Sign pairs
Figure 9: Performance training NanoGPT addition of signed numbers. (Left) Training addition and subtraction tasks using only positive numbers up digits, contrasted with the signed addition task involving various sign combinations. (Right) Performance of the model trained on signed addition with 20,000 training examples, across different sign combinations in the test data.
B.3.3 PERFORMANCE BASED ON DIFFERENT DIGIT LENGTHS
We analyze the performance of NanoGPT models by segregating the test accuracy based on the digit count of operands. In Table 6 and Table 7, rows and columns correspond to the digit count in the first and second operands, respectively.
Notably, with the plain format, performance decreases with fewer digits in the operands, and an asymmetry based on operand lengths is observed. We speculate that this asymmetry arises because, despite efforts to ensure balanced sampling, combinations of small digits, such as (1, (),3)-u -igit or (2, digit pairs, are underrepresented in our training dataset.
Table 6: Performance by digit length in plain format addition.
I-digit
2-digit
3-digit
1-digit
72.8%
42.2%
77.0%
2-digit
27 27.8%
60.0%
91.4%
3 3-digit
38.6%
61.2%
97.0%
In contrast, the reverse format result exhibits notably consistent performance across various digit combinations, indicating near-perfect learning of addition by the model.
Table Performance digit length in format addition.
I-digit
2-digit
3-digit
1-digit
100.0%
99.8%
99.0%
2-digit
96.4%
100.0%
100.0%
3-digit
94.0%
99.4%
100.0%
B.4 THE IMPORTANCE OF INTERMEDIATE STEP DESIGN
In this section, we underscore the significance of meticulously designing the intermediate steps in Chain-of Thoughtt manner. Specifically, investigate whether the enhanced sample efficiency of NanoGPT in detailed scratchpad format arises from its longer length or from the breakdown of intermediate steps into simpler components.
19

---

Published as a conference paper at ICLR 2024
Randomizing the intermediate steps To discern the impact of length, modify the intermediate steps, replacing them with either uniform token "#" or random tokens within the vocabulary (see examples in Figure 10).
intermediate steps
intermediate steps random
100
Inpul:
Input:
128-367
Test Accuracy (%)
Target:
Target:
LVV 33d-o (BligggnnnSssiiilcg
60
<:04vi:-Wdk((ek: 9whhiied KE
&
pweINZ2P@@0
2
vanilia scratchpad
ntemedate staps mancom
<tcrrachh>
</scratch>
2.5x 5k 7.5x 10 12.5k 15k 17.5k 2ōk
495
Number of train examples
Figure 10: Replacing the intermediatc steps with random characters. We ablate whether the sample efficiency gain from the detailed scratchpad stems from the increased length the format.
The results (Figure 10) indicate that the sample efficiency of the vanilla detailed scratchpad surpasses the modified versions. This suggests that the advantage of detailed scratchpad format stems from the breakdown into simpler functions rather than its increased length.
Subtraction two different intermediate step design We further focus the subtraction task and conduct experiments to compare two different versions of the detailed scratchpad for this operation (see examples in Figure 11). These trials shed light on the importance of decomposing the subtraction task into simpler intermediate steps. Unlike addition, subtraction behaves differently depending on whether the first operand (a) is greater than the second operand (b) vice
The first strategy (Version in Figure 11) involves performing digit-wise subtraction starting from the least significant bit (LSB) and considering borrows when necessary. However, this strategy produces incorrect results when the first operand is smaller than the second operand. In such cases, we subtract the number in the most significant bit (MSB) position multiplied by 10 the power of (number of digits in the output- 1) from the remaining digits in the output. An example illustrating this approach shown Version 1, Case Alternatively, we can adopt a more familiar strategy. If the first operand is smaller than the second, we swap the operands and compute the negation of the subtraction of the swapped operands: a
- (b a) (referred to as Version 2).
The results in Figure 12 indicate that Version 2, which involves comparing two operands, performs considerably worse than Version In Version each intermediate step only requires the simpler 1-digit subtraction, along with addition in the final result processing step. Upon analyzing the failure cases of Version 2, we observe that the majority errors stem from incorrectly identifying which of the two operands is larger, while the intermediate steps handled correctly. This finding underscores the significance of breaking down arithmetic operations into simpler intermediate steps. Unless otherwise specified,
Version 1 in all detailed scratchpad experiments.
20

---

Published as a conference paper at ICLR 2024
Detailed scratchpad formatting for different arithmetic tasks
Examples of variations of detailed seratchpad formatting for subtraction, considering the scenario where the first operand a is greater than the second operand b, and vice versa. In Version 1, result processing step included the final stage handle negative outputs. In Version 2, the operands compared at the beginning, and if b is larger. their order is reversed.
Prompt (Case 1. - b2 0)
Input:
367-128
Target:
Version 1.
Version 2.
<scratch>
[3,6,7] has 3 digits.
[3,6,7] has 3 digits.
[1,2,8] has 3 digita.
[1,2,8] has 3 digits.
[3,6,7] -[2,2,8] a=l1 C=0 3-8-0+10=9
367>=128 # comparison of two operands
A->9 0->-1
[3,6,7) -1,2,8) A-() C-D 7-8-0+10-9
[3,6] [1,2] 2-19]
6-2-1-3
A-23 0->-1
C->0
[3,6] [1,2] A-19: cm-1 6-2-1-3 A-x3
A=[3,9]
3-1-0=2
C->0
C->0
[3] - [1] 2=13,9) Ce0 3-1-0=2 A-22
[1 U
3-12,,,31
0->0
200+39=239 END result processing
() (), a=(2,3,9) END
</acratch>
</soratch>
2
Prompt (Case 2. a b<
Input:
128-367
Target:
Version 1.
Version
...
<scratchs
<scratch>
[1,2,8] has digits.
[1,2,8] has digits.
[3,6,3] has 3 digits.
(3,6,7) has 3 digits.
[1,2,8] -13,677 B-ll C-0 8-7-0-1
1284367 128-336-- (367-129))1 # comparison
>1 C >0
[3,6,7] [1,2,8] A-(] C-0 7 0110-9
[1,2] -[3,6] Am[1] 0-0 2-6-0410-6
A->3 2->-1
>6 c
[3,6] [1,2]
A-19]
[1] --- [3] A-16,11 C--1 1-3-1--3, A->-3
0->0
C---11
[3] [-] [1] A=[3,9]
3-1-0=2 A->2
U] 2=1-3,6,1]
G->0
-300=61-- 230-681-239.00 END result processing
-- (], 2-12,3,9] BND
</scratch>
</scretch>
-2 3
2
Figure 11:
Two versions of detailed scratchpad formatting for subtraction.
B.5 THE EFFECT NOISY INPUTS ON ACCURACY
Noisy intermediate steps in the scratchpad data. We further investigate the significance of providing accurate intermediate steps in the scratchpad during the training process. While this was inspired by the findings of Min al. (2022), inherently different. Min et al. (2022) show that using random labels in ICL demonstrations caused minimal degradation when compared to the gold labels. However, those models were trained on gold labels and then evaluated multiple downstream tasks. In our setting, the model is trained and evaluated on single arithmetic task. Further, the final result(or label) is left untouched as the correct answer to the arithmetic operation. We only replace the intermediate steps. The goal of this study is to verify whether the model actually learns to reason using the given intermediate steps or merely uses the scratchpad to improve its expressivity. We compare the performance of training with simplified scratchpad formatting, which includes accurate A (digit sum) and C' (carry) information, with formatting that includes random A, random C, or random A and C for each intermediate step, as depicted in Figure 1.
The results in Figure 13, demonstrate that the inclusion of noisy labels can impede sample efficiency. However, with enough samples, the model ultimately achieves full accuracy. This suggests that while
21

---

Published as a conference paper at ICLR 2024
100.0
100.0%
97.5
Test Accuracy (%)
95.0
90.0
89.9%
87.5
85.0
82.5
80.5%
Piain
Reverse (DS]Ver (D5Wer:
Data Formatting Methods
Figure 12: Comparison of performance among various data formatting approaches (plain, reverse, and versions of detailed scratchpad (DS)) the subtraction task. experiments were conducted NanoGPT model trained on a dataset of 10,000 examples. Version 2, which incorporates operand comparison, exhibits significantly lower performance compared to Version 1. This observation highlights the substantial impact of the construction of intermediate steps on the model's performance.
(a) Test accuracy Addition
(b) Test accuracy Subtraction
100
100
Test Accuracy (%)
80
Accuracy (%)
80
60
60
correct A
correct A &
20
random A
20
random A
random
random
random
0
2000 4000 6000 8000 10000 12000
0 2000 4000 6000 8000 10000 12000 Number of train examples
Number train examples
Figure 13: Comparison of training with simplified scratchpad formatting using correct A and information with formatting using random A/C and their effect on sample efficiency and accuracy. Results show that noisy labels degrade sample efficiency, but with sufficient training data, the model eventually reaches full accuracy.
the model is capable of leveraging the information contained in the intermediate steps, it can also gradually learn how to perform addition while disregarding the presence of noisy intermediate steps.
Model robustness to noise in the auto-regressive output. In this analysis, explore the robustness models trained on plain or reverse formatted data (without noise) when exposed to noise during an auto-regressive generation process. In particular, aim unravel how much the learned mapping of the i-th output relies on the operands and preceding tokens in the addition result, given that transformer models generate tokens sequentially an autoregressive manner, making them prone to error propagation.
For this experiment, we focus on 3-digit addition. We train models either plain reverse format data and evaluate the accuracy of next-token predictions when the output sequence contains noise. Specifically, in the plain format setting, we expect a well-performing model generate the correct output tokens O3. Oz, O, sequentially, where O₃ C3. O2 = C2 O C, and C2C,C, represents the correct answer. We consider two types of perturbation: modify the first two output tokens O3O2 to random numbers different from C2C, and (ii) precise perturbation, where we perturb only the second output token O by The second case is particularly relevant since common error case is where the model misses digit by 1. We provide the model with an expression of the form "A3AzA1 ByBNB, = 03O2". where O₃O2 can be either (i) a random incorrect number, i.e., O302 C3 C2, (ii O2 C2 + 1 mod 10, and observe the next token generated by the model. A corresponding process is deployed for the reverse format, introducing a noisy sequence models trained reverse format data.
random perturbation, where we
22

---

Published as a conference paper at ICLR 2024
To evaluate the performance, define two accuracy criteria for O1: exact accuracy, reckoning O₂ as accurate only when O1 = C1 and relaxed accuracy, considering O2 correct if it deviates from the original output C₁ by at most 1. In other words, C O1, C 01 + mod 10 or mod 10.
G = 01 -
Table 8: Prediction accuracy for third digit output under different types of noise in the preceding output tokens. Random perturbation, applics random flips whereas precise perturbation shifts the preceding output tokens by 1. Relaxed accuracy, allows for 11 deviation from the true output whereas Exact accuracy is strict. Reverse consistently outputs number that at most 1 different from the true output, even in the presence of noise. The plain format has high exact accuracy the presence of precise perturbation, the noise in the oulput token lower impact on predicting the next token, which lower significance. However, with completely random noise, the plain format shows poor performance, suggesting a strong dependence on all digits. (See Lemma and 2).
Perturbation Type
Random
Precise
Plain
Reverse
Plain
Reverse
Exact Ace Relaxed Acc 61.55%
49.88%
81.26 %
99.85% 100%
90.47%
100 %
100%
The results presented in Table 8 reveal intriguing findings. We observe that the reverse format consistently outputs a result that deviates by no more than from the true answer, regardless of whether the preceding outputs O302 are subjected to random or precise perturbation. This consistency can be explained by Lemma indicating that the reverse format only requires learning straightforward function of digit-wise addition for each corresponding position, along with the carry-on (0 or 1). Therefore, even with noise the preceding tokens, the model accurately performs digit-wise addition, albeit with occasional carry-on prediction errors. With exact accuracy of 81.26% even in the presence of random perturbation, the reverse format demonstrates the model's ability to rely less on the preceding output tokens, indicating robust learned output mapping.
On the contrary, models using the plain format have to decipher a more intricate function drawing from all digits within the sequence, described by Lemma 1. Given that in addition, carry operations transition from right to left (i.e,, least to most significant digit), the introduction of precise perturbation on preceding output tokens, which possess higher significance, has minor impact the output (which has less significance). As result, models trained using the plain format attain an exact accuracy rate of 99.85% and relaxed accuracy of 100% for cases involving precise perturbation. Interestingly, under purely random perturbation, the plain format struggles, leading to reduced relaxed accuracy of 61.55% and exact accuracy of 49.88%. This suggests that the output mapping learned by the plain format is not merely function of the two operands but rather enmeshed in complex dependencies on preceding output tokens.
B.6 ANALYZING THE RESULTS ON SINE/SQRT
Since sine and sqrt arguably more complicated functions than the remaining arithmetic tasks, decided to more carefully analyze their performance. As shown in Figure 14, sin shows excellent performance across all data formats around sar( 0. We conjecture that this is because sin( for æ x 0, which is easy to learn. We also note that accuracy once again improves close to +1 potentially for similar reasons.
23

---

Published as a conference paper at ICLR 2024
(a) Test accuracy Sine
(b) Test accuracy on Square root
Sin
Squtt
100
Mu M
M
Test Accuracy (16)
M4
Test Accuracy
plain cmo emD
plain, 2-0
CoT,
plain, 5m50 So
plain, Sed GIT S4-4
11.00-0.75-000
0,250.00 true y value
0.75 1.00
1.0
2.0 truc y valuee
2.5
3.0
Figure 14: Error analysis of sine square root functions, considering varying error tolerance (eps) thresholds to determine correct output. The sine function demonstrates excellent performance across all data formats, particularly around siu(2) 0. where sin(æ) = for x z O. Additionally, we observe improved accuracy near 11.
C EXTENDING TO LONGER DIGIT ADDITION
In this section, we extend our experiments beyond 3-digit addition and explore longer-digit settings, ranging up to 10 digits. Our aim to investigate whether our previous findings regarding the sample efficiency of reverse and scratchpad formats hold true for larger numbers of digits.
We begin by observing that the phase transition behavior observed in previous sections also applies to longer-digit addition. Furthermore, discover that the advantages of using reverse and scratchpad formats become even more pronounced as the number of digits increases. Next, we examine the number of training samples required learn k digit addition when fine-tuning pretrained model trained on & digit addition. We find that while the number of samples needed to further learn k + 1 digit addition remains relatively consistent for reverse and scratchpad formats, the plain format requires an increasing number of samples.
Experimental setup and data generation. To explore the performance of the model higher-digit addition scenarios, we extend the experimental setup described in Section 3. We adopt balanced sampling approach for training data with D digits, ensuring an equal number d of all combinations of digits for both operands as follows:
begin by sampling 100-digit additions. For the remaining number of digits, ranging from 2 to D, we generate addition examples of the form "A + B C". The two operands, A and B, are randomly sampled d = [(V total number of training examples. Operand A is sampled between [10 10
+ 1)/2 1)J times for every D, where N the
and operand is sampled between [10 = - 1]. for all < k1 < li2 < D, excluding the case where k l2 = 1. After sampling the operands, randomly interchange them cover cases where has fewer digits than B and vice versa.
C.I TRAINING FROM RANDOM INITIALIZATION
We repeat the experiment from Section 3 on nanoGPT with longer digits. The results shown in Figure demonstrate similar behavior to the findings observed in Figure 4a for 3-digit addition. This indicates that our previous observations generalize to longer sequence lengths. Notably, the performance gap between the modified formats (reverse, simplified scratchpad, and detailed scratch- pad) and the plain format becomes plain format requires an increasing number of training examples to learn higher-digit additions, the reverse scratchpad formats exhibit more consistent requirement in terms of the number of training examples.
more significant in the context of higher digits. While the
This prompts to explore the differences between each format in fine-tuning setting. Specifically, we ask whether model trained on reverse or scratchpad-fonen formatted & digit addition data would find easier to learn is + digit addition compared model trained with plain format addition.
24

---

Published as a conference paper at ICLR 2024
(a) 5-digit Addition
(b) 7-digit Addition
(c) 10-digit Addition
100
1.00
100
Test Accuracy (85
80
Test Accuracy (%)
80
8G
Test Accuracy
60-
60
plain
piain
plain
reverse
reverse
reverse
simplified scratchpad
simplified scratchpad
simplified scratchpad
detalled scratchpad
detailed scratchpad
detalled scratchpad
0
25k
50k
75k
100k
50k
100k
150k
200k
100k
200k
300k
400k
Number of examples
Number train examples
Number train examples
Figure 15: Comparison of sample efficiency 5, 7 and 10-digit additions: performance of models trained with varying numbers of addition samples each data format. The plain format data requires an increasing number of training examples for higher digits, while the number of samples required for other methods remains relatively consistent.
C.2 FINE- TUNING FROM PRETRAINED MODELS
In this section, we investigate the generalization ability of transformer models, specifically focusing on their capacity to learn higher-digit additions based on their knowledge of lower-digit additions. Additionally, we explore how the choice of data format affects the number of samples required to learn higher-digit additions.
Forgetting of d-iigtt addition when trained on k + 1-digit addition.
We begin by fine-tuning model that was initially trained on 3-digit addition. We fine-tune this model using 4-digit addition training data, with each data format being used separately. To mitigate the "catastrophic forgetting" phenomenon, we experiment with different learning rates, gradually reducing the magnitude. We continue this process until the learning rate becomes too small for the model effectively learn 4-digit addition.
plain (Ir=le-4)
simple (I==le-5)
reverse (Ir=Se-6)
detailed (In=1e-6)
1-digit accuracy
2-digit accuracy
100
Test Accuracy (%)
3- 3-digit accuracy
4 4-digit accuracy
100
50
5000
10000
5000
10000
Number of Iterations
Figure 16: Accuracy of to 4-digit additions during fine-tuning of pretrained model 3-digit additions using different data formats. The model is fine-tuned using only 4-digit addition data with corresponding formats. We observe that the plain format 'Forgets' to 3-digit additions entirely when learning 4-digit addition. In contrast. the detailed scratchpad method successfully learns 4-digit addition while maintaining high performance on to 3-digit additions.
The results depicted in Figure 16 reveal interesting insights about the fine-tuning process. When training the model using the plain format with only 4-digit addition data, there an immediate drop in accuracy for 1 to 3 digit additions. This indicates that the model experiences significant forgetting of previously learned additions. In contrast, the reverse and scratchpad methods exhibit a more favorable behavior. The model trained with these methods does not completely forget or 2 digit additions while learning 4-digit addition. Remarkably, the detailed scratchpad method stands out by enabling the model to learn 4-digit addition without compromising its performance I to 3 digit
25

---

Published as a conference paper at ICLR 2024
additions. Although there is slight decrease in performance for 3-digit additions initially, the model quickly recovers and picks up the knowledge again as it trains on 4-digit additions.
This result can be explained by the hypothesis that learning a model an incremental process for the detailed scratchpad method. The model already has solid only needs to adapt to
digit addition from k-digit
foundation in understanding the intermediate steps involved in addition, longer sequences. In contrast, for the plain format, learning higher-digit additions requires the model establish new mappings generate correct outputs, which is more challenging task.
Sample efficiency fine-tuning k-digit models with + 1-digit examples. Building upon our previous findings that fine-tuning model solely k 1-digit addition leads loss in performance for k-digit addition, we modify our approach to prevent the loss of performance in the k-digit addition task. Instead of training solely on k 1-digit examples, addition tasks from 1-digit to k + 1-digit, with the method described in the previous section. By doing so, we aim to maintain the performance of k-digit addition while enabling the model to learn k + 1-digit addition during fine-tuning.
construct dataset that includes all
In this experiment, we investigate the number of k 1-digit training examples required for the model to effectively learn k + 1-digit addition when fine-tuning a pretrained model on k-digit addition. It is important to note that this setting differs from the previous section (Section C.I), where we focused on training models from random initialization. Here, we specifically focus on the fine-tuning process. We fine-tune individual models pretrained on each data format (using k-digit addition) and further train them using the same data format new dataset that includes all addition examples from 1 I-digit to & + d-digit.
(a) Plain
(b) Reverse
Test Accuracy (%)
80
Test Accuracy (%)
80
60
60
40
40
3 digit
3-digit
20
4-digit
digit
diiiit
8-digit
8-digit
10000 20000 30000 40000 50000 60000 Number digit train examples
0
1000
2000 3000 4000
5000
6000
Number k+1 digit train examples
(c) Simplified Scratchpad
(d) Detailed Scratchpad
100
100
Test Accuracy (%)
80
Test Accuracy (%)
80
60
60
40
40
3 3-digit
3-digit
20
digit
20
4-digit
6-digit
6-digit
8-digit
didigt
1000
2000
3000
4000
5000
0
500
1000
1500 2000
2500
3000
Number of k+1 digit train examples
Number k+1 digit examples
Figure 17: Fine-tuning performance of pretrained k-digit models using varying numbers of & 1-digit examples, with corresponding data formats. The plain format requires an increasing number of + 1-digit examples as number of digits (A + 1) increases. In contrast, the modified formats (reverse, scratchpad) exhibit consistent performance across different numbers of digits, requiring a relatively consistent number of examples to learn the additional digit.
The results in Figure 17 demonstrate the number of k + 1-digit addition samples required for a pretrained model capable of performing k-digit addition to leam the addition of k: digits. The findings reveal that modified formats (reverse, scratchpad) require a relatively small number of samples (between 1000 and 5000) to learn the addition of an extra digit. In contrast, the plain format
26

---

Published as a conference paper at ICLR 2024
necessitates significantly larger number of training examples, with the requirement increasing as the number of digits grows.
This observation aligns with our previously established Lemma and Lemma 1, which suggest that learning higher-digt addition in the reverse format involves processing the i-th digit of the operands and carrying from the previous position. This operation remains consistent regardless of the number of digits being added. As a result, the model primarily needs to learn how to handle longer digits to perform addition effectively.
In contrast, the plain addition format requires the model to learn a more complex function that incorporates all digits from both operands. As the number of digits increases, the complexity of this function grows as well. This highlights the greater difficulty faced by the plain format in accommodating additions with larger number of digits.
C.3 IMPACT OF FORMATS ON TNE-TUNING
We delve deeper into the impact of different formats the fine-tuning process. Specifically, investigate whether training a model in one format helps in learning addition in another format, and vice versa. To conduct this analysis, we begin with model trained each data format using 3-digit addition examples. We then individually fine-tune these pretrained models using different data formats, 4-digit addition examples.
plain
simple
random init
The results depicted in Figure 18 highlight some interesting findings. Firstly, we observe that model trained with the same format as the fine- tuning format exhibits faster learning in terms of the number of iterations. For instance, train- ing a model with the plain format outperforms training a model pretrained with scratchpad for- mats. This suggests that the model benefits from
reverse
detailed
finetuning with plain
finetuning with reverse
100
Test Accuracy (%)
50
finetuning with simple
finetuning with detailed
the consistency and familiarity provided by the same format throughout the training process.
100
Additionally, we notice that fine-tuning de- tailed scratchpad pretrained model on other for-
50
mats proves be more challenging. This ob- servation can be attributed to the need for the model to "unlearn" the intricacies of the ver- bose detailed scratchpad format and adapt to the new format. For example, the plain format does not involve the use of alphabet characters in the data, so a model pretrained with the plain format would have a low probability of gener- ating alphabetic outputs. contrast, detailed scratchpad pretrained model would have encoun- tered various alphabets and may have tendency to output them. Therefore, adjusting to new format requires additional effort for the model to "unleam" the patterns specific to the previous format and effectively learn the new format it is being trained on.
0
5000 10000 0
5000 10000
Number of Iterations
Figure 18: Performance fine-tuning a 3-digit model trained on diffcrent data formats (plain, reverse, simple scratchpad, detailed scratchpad, and random initializa- tion) individually with different data formats of 4-digit addition. The results demonstrate that fine-tuning yields the best performance when the pretrained model and the fine-tuning format are consistent. Notably, fine-tuning a detailed scratchpad format model shows suboptimal performance. We hypothesize that this is due to the need for the model to "unlearn" " the rigid and verbose format and adapt to the new format.
These findings highlight the importance of considering format consistency during the fine-tuning process, as it can impact the efficiency and effectiveness of the learning process. We will delve further into this topic in the upcoming section 8. where we fine-tune pretrained GPT-3 models. Notably, observe that fine-tuning with reverse or simplified scratchpad formats actually yields worse results compared to fine-tuning with plain formats. For detailed exploration of these observations, please refer to the forthcoming section.
27

---

Published as a conference paper at ICLR 2024
D TEACHING ARITHMETIC OPERATIONS BEYOND ADDITION
While this study has a primary focus on the addition operation and aims to comprehend the signifi- cance of data sampling and formatting, its findings are applicable beyond the realm of addition alone. In this section, we expand our examination to include other arithmetic operations, thus demonstrating the broader applicability of our insights. We consider a mix of arithmetic tasks, including binary operations like subtraction and multiplication, and unary operations such as sine and square root. Each operation entails its unique challenges and intricacies. For instance, subtraction introduces the concept of negative numbers, multiplication can generate significantly longer outputs, and sine and square root functions entail computations involving floating point numbers, which are considered up to four digits of precision in work.
We acknowledge that while our examination is detailed, it does not encompass all the fundamental arithmetic operations the entire scope of floating point arithmetic. Specifically, our focus is primar- ily on integer arithmetic for binary operations, considering limited length of digits. Additionally, for unary operations, we confine ourselves to a restricted number of digits below the decimal point.
In Section D.1, we delve into each arithmetic operation individually, exploring the impact of data formatting and determining the relevancy of our insights across disparate tasks. Further, in Section D.2, we perform an analysis of joint training across all five tasks, investigating the potential performance implications for each individual task.
D.1 EXTENDED ARITHMETIC OPERATIONS
In order to extend our analysis to arithmetic operations beyond addition, we consider the following tasks:
Subtraction (-). We consider subtraction of positive numbers up to 3 digits, written as AsAzA1 ByByB, C2C2C, in (i) plain formatting, and SA3A2A1 B3ByBr = C2CC in (ii) reverse formatting. As with addition, scratchpad-baat based methods (iii, iv), present the intermediate steps of disit-wise subtraction and handling of carry-ons. These steps proceed from the least significant bit (LSB) to the most significant bit (MSB). If the final result after computing all the digit-wise subtractions is negative, by 10 to the power of (number of digits in the output- 1) from the remaining digits in the output. In Section B.4, we present an alternative version of the detailed scratchpad formatting for subtraction.
subtract the number in the most significant bit (MSB) position multiplied
Multiplication (x). We consider multiplication of positive numbers up to 2-digits. (i) Plain formatting examples are formatted as AAA1 B2B; = C2C,C,C,, while (ii) reverse formatting is formatted as $AzA1 B2B, = C3CC The (iv) detailed scratchpad method simplifies each intermediate step by conducting series of multiplications between the first operand and cach digit of the second operand, starting from the least significant bit (LSB) and moving toward the most significant bit (MSB). For each step, we multiply the result by an exponentiation of 10 corresponding to the relative digit position.
Sine (sin). We consider decimal numbers within the range |-T/2, π 4/2). truncated to 4-digit precision. (i) Plain formatting examples are formatted as Ain = Bo By For (iv) detailed scratchpad method, we include the Taylor series expansion steps for sine, which is represented as sin(x) x _ + 1 # + .... These intermediate steps involve exponentiation, which may not be any easier to compute than the sine operation itself.
Square Root (V). We consider decimal numbers within [1, 10), truncated 4-digits of precision with the format, written as (1.AAAAAAAAAAA B, ByBBB for (i) plain formatting. For (iv) detailed scratchpad method, we enumerate each step of Newton's 's method to compute the square root function. The iterative formula given by Jn, = 1 (#n-1 + x ). where xo initialized as the floor of the square root value of the operand x. These intermediate steps involve a division operation, which can be complex as the square root operation itself.
For evaluation of sine and square root, we classify the result S as correct if the absolute difference between yi and the ground truth value y is less than or equal to predefined threshold 2 0.
28

---

Published as a conference paper at ICLR 2024
For each arithmetic task, explore both the plain format and the detailed scratchpad format. The detailed scratchpad formatting for each task is illustrated in Figure 19 and Appendix J. For subtraction, the process involves breaking down the operation into intermediate steps of digit-wise subtraction, including carry-ons when necessary. Unlike addition, subtraction requires an additional step to handle cases where the first operand is smaller than the second. Further details the detailed scratchpad for subtraction can be found in Section B.4. For multiplication, each intermediate step carries out 2-digit 1-digit multiplication between the first operand and each separate digit of the second operand. For sine and square root, we utilize sequence of iterative approximations instead of algorithmic explanations. Specifically, Taylor's series expansion steps for sine and Newton's method steps for square root are used. It is important to note that while addition, subtraction, and multiplication are broken down into simpler operations at each step, CoT for sine square root functions requires intermediate steps involving operations like exponentiation or division, which might not be inherently simpler.
Detailed scratchpad formatting for different arithmetic tasks
Examples of detailed scratchpad formatting for different arithmetic tasks:
(1) Subtraction - includes borrows for intermediate steps, (2) Multiplication decomposes the second operand for 2-digit x 1-digit multiplication at each step, (3) Sine utilizes Taylor series expansion, and (4) Square root employs Newton's method.
Subtraction
Sine
Input:
Iaput:
128-367
sin(1.5707)
Torget:
Target:
<scratch>
<scratch>
[1,2,8] digits.
x_0=1.5707
[3,6,7] has digita.
X_1: X_D
1/31
(x"3)
_.140.9247
[1,2,8] [3,6,7] A-ll C-0 0-1
x_ x_1
1/51
(x"5)
x_2=1.0043
A->1 C->0
x_3: x_2
1/71
(x*7)
x_3=0_9996
[1,2] [3,6) A=(11] c=0 2-6-0+10=6
x_4: x_
1/91
(x"9)
x_4=0.9997
END
A-26 C->-1
</scratch>>
[1] [1]-13] A-16,11
C--1, 1-3-1-33
0.9997
->-3 -33--
I1 A-1-3,6,11
500-61--239. ENC
</scratch> -2 3.9
Multiplication
Sqrt
Ingut:
Input:
12+36
(2.
Target:
Target:
<scratch>
<scratch>
[1,2] has digits.
x_0=1
[3,6] digita.
x 1/22/112..11//11444..838 1_1418888811118887
[1,2] A=[7,2] x=1 8=(7,2)
=0+72=72
1..603
[1,2] A-13,61 x-10 D-(3,6,0]
~72-360-132 BND
-1.6485
</scratch>
x_4: 1122(1.666552..1175//
a 3 2
=1.6484 RND
</scratch>
0.6484
Figure 19; Examples of the detailed scratchpad format for different arithmetic tasks such subtraction, sine, multiplication, and square root.
The results depicted in Figure 20 indicate that similar to the findings of addition, the detailed scratchpad format significantly improves performance over plain or reverse formats and yields efficient results with few samples for subtraction and multiplication tasks. Interestingly, find reverse is not particularly effective in multiplication. On the other hand, the detailed scratchpad format exhibits reduced efficiency for sin and / compared other operations -,,, This
29

---

Published as a conference paper at ICLR 2024
(a) Subtraction
(b) Multiplication
100
Test Accuracy (%)
Test Accuracy (%)
80
60
plain
detailed scretchpad
plain
detailed scratchpad
2.5k 5k 7.5k 10k 12.5k 15k 17.5k 20k Number of train examples
1000
2000
3000
4000
5000
600D
7000
Number train examples
(c) Sine
(d) Square Root
Test Accuracy (%)
80
Test Accuracy (%)
80
60
60
plain, c-0
plain,
40
plain, r=Se-4
40-
plain, c=
plain, c-Se-3
plain, r-Se-3
detailed, e=0 detailed,
detailed, e=0
20
detailed, c-Se-3
detailed, 5-
5k 10k 15k 20k 25k 30k 35k 40k
Sk 10k 15k 20k 25k 30k 35k 40k Number of train examples
Number of train examples
Figure 20: Performance of 3-digit subtraction, 2-digit multiplication, 4-digit precision sine and square root with varying data formats. As with addition, reverse always produces improved sample complexity and performance for all operations. For sine and square root, scratchpad formatting provides limited improvement. This discrepancy can be attributed to the complexity of the intermediate steps involved in the detailed scratchpad.
discrepancy can be traced back to the complexity of the intermediate steps involved in the detailed scratchpad. While addition, subtraction, and multiplication are decomposed into simpler functions, sine and square root operations involve more intricate operations. For a broader analysis the error profile, Appendix B.6.
D.2 JOINTLY TRAINING ON ALL FIVE ARITHMETIC TASKS
So far, we only considered the problem of learning different arithmetic operations individually. In this section, we study the effect of jointly training on all five arithmetic tasks addition, subtrac- tion, multiplication, sine, and square root. We construct a single train dataset incorporating all
task Durin = {D D Diriin Dix Dsin DV Di'nn- and randomize the sequence of tasks in our train samples. For example, randomly chosen segment of the training data may exhibit task order such as (+, sin
x, ...... We consider 10, (000 training examples for each task of addition, subtraction, sine, and square root and 3, 000 for multiplication.
The model's performance, after training on our joint dataset Dusin- evaluated in both zero-shot and few-shot settings. These results are also compared with the performance of models that were trained
separately on each dataset (D itin D Priiin Piiin Dsin DV' ). identical to those used to construct Drrai- In the few-shot setting, each task is given examples from any of the five arithmetic tasks (not necessarily related to the test task under consideration) or prompt texts, followed by test queries specific to the task of interest. For further details the few-shot prompting methods used, please refer to Section
Table 9 shows that joint training significantly enhances the zero-shot performance for multiplication and square root tasks, yet it slightly reduces the performance for subtraction. Generally, few-shot prompting exhibits improved performance. Notably, the performance of few-shot prompting remains consistent regardless of whether the exemplars provided are from unrelated tasks or are taee-spec We propose that this consistency is due to randomized task sequence during training, which
30

---

Published as a conference paper at ICLR 2024
presents the model with numerous instances where task directly follows another, thus simulating few-shot prompting with different tasks. Furthermore, we observe that text prompting performs similar to zero-shot. We conjecture that this is because the training data does not include text data and the model has never encountered text and therefore, text prompting serves as a random prefix attached to test query.
Table Performance of models trained individually and jointly five arithmetic tasks. The threshold for sin / functions set For models trained jointl on five tasks, we evaluate their performance both zero-shot setting and few-shot setting. In the few-shot setting, each task is presented with exemplars from one of the five arithmetic tasks prompted with text, followed by task-specith test queries. The results show that few-shot prompting with any arithmetic operators (even unrelated the test task) generally improves performance. However, text prompting shows performance similar to the zero-shot setting.
Trained jointly on all 5 tasks
Trained on individual task
Few-shot exemplar format sin
Zero-shot
sqrt
text
84.06
87.96
96.45
96.90
96.92
97 97.06
97.01
88.71
79.97
72.83s
81.28
79.59
81.39
81.84
81.74
68.91
x
4.58
14.28
18.86
18.96
15.43
19.20
19.59
15.48
sin
35.03
34.74
34.35
34.31
34.34
32.64
33.42
33.96
sqrt
19.85
27.37
26.65
26.74
26.70
25.60
25.61
26.02
31

---

Published as a conference paper at ICLR 2024
E MIXING TEXT WITH ARITHMETIC DATA
Until now, our focus was primarily on models trained exclusively on arithmetic tasks. However, in practice, large language models (LLMs) utilize a combination of arithmetic and text data for training. In this section, we broaden our scope by incorporating both addition samples and text into our pretraining data. We then evaluate the trained models with various few-shot prompts to analyze the model is able to effectively identify the correct context.
Experimental Setup. We mix addition and text data in our experiment using the Shakespeare dataset (Karpathy. 2015) that includes 1, 115, 394 tokens of text, 10, 000 plain addition examples (120, 027 tokens) without the $ delimiter, and 3, 000 detailed scratchpad formatted addition examples (813, 510 tokens). We fix the number of detailed scratchpad examples and plain addition examples (3, 000 and 10, 000 respectively) while varying the number of each example type in the training pro- cess. The Shakespeare text is segmented into dialogue chunks, with random number of consecutive plain addition data and detailed seratchpad data inserted between them. We use character- tokenizer with vocabulary size of 80, containing all characters present in the dataset, including alphahets, digits, and certain symbols like +, and Vr.
E. FEW-SHOT PROMPTING
Given the mixed nature (arithmetic and text) of our dataset, introducing relevant examples seems an effective strategy to prime the model to generate the desired type of output. To assess the performance of such few-shot (1/2/3-shot) prompting, we provide task-specific exemplars as illustrated in Figure 21. Plain addition formatted exemplars are used for testing plain addition inputs, while detailed scratchpad formatted exemplars are utilized for assessing performance on detailed scratchpad formatted inputs. Additionally, we experiment with demonstrating text (sce Appendix E.3. for details) before querying addition (which we denote, fixed set of exemplars. Standard deviations of these prompts are denoted by shaded areas in the plots. The term "few-shot" refers to the reported mean of all 1/2/3-shot prompting results.
For each 1/2/3-shot and text prompting.
average performance is reported
k-shot evaluation
Set-i exemplars + b y1
4 b y
Set-i exemplars
input to model
4: + Y2
Set-i exemplars
Set-i accuracy
4g + b y
a2 + bg 3/2
Mean Accuracy
Set-i exemplurs
set i-accuracy
Test dataset
dg + b = y/3
Figure 21: Few-shot prompting method. Few-shot prompting performance is evaluated by presenting relevant exemplars of addition and detailed scratchpad formatted inputs. Each 1/2/3-shot prompting is tested fixed five set of exemplars, and the accuracy is averaged these evaluations.
Figure 22 shows that few-shot prompting directs the enhancement of performance, thereby allowing plain addition to perform almost perfectly with 40,000 train samples. Intriguingly, performance remains high on plain addition even with the inclusion of a text prompt, given substantial number of addition examples. We hypothesize that this is due to the structure of our mixed dataset where addition examples are interspersed within Shakespeare data. With the incorporation of more addition examples, instances where addition examples directly follow Shakespeare text increase, leading to decrease in potential inconsistencies when text content is present during addition test queries.
E.2 DISENTANGLING THE EFFECT OF TEXT ON PROMPTING
To disentangle the effects of the textual content in the training data, we train a model strictly on plain addition, utilizing enlarged vocabulary that also includes alphahet characters, thereby enabling text
32

---

Published as a conference paper at ICLR 2024
(a) Test accuracy on plain addition
(b)
Test accuracy on detailed scratchpad
100
100
Zerosshot
Accuracy (%)
Test Accuracy (%)
80
1-shot
2-shot
60
3-shat
text_promig
40
Zero-shot
40
1-shot
2-shot
20
3-shot
text prompt
10k 15k
20k 25k
30k
35k
40k
1000
2000 3000
4000 5000
Number of Addition Samples
Number of Detailed Scratchpad Samples
Figure 22: Performance of NanoGPT model trained with the Shakespeare dataset, addition dataset in plain, and detailed scratchpad format. The number of plain (left) and detailed scratchpad (right) formatted addition samples are varied. Performance evaluated on zero-shot, few-shot, and text prompts, with the shaded area representing the standard deviation across various prompt exemplar sets. The results indicate a consistent enhancement in model performance using few-shot prompting.
prompting. (Note that previous experimental settings on plain formatted additions used a vocabulary size of 13, which only includes 10 numerals and symbols "+","= "=", "nn"). We introduce variant of few-shot prompting, termed as noisy-prompe which prompts the model with erroneous addition exemplars, ie., == C, where * B.
100
Test Accuracy (%%
Zero-shot
40
1-shat
2-shot
3-shot
Noisy-prompt
Text Text-prompt
0 5k 10k 15k 20k 25k 30k 35k 40k
Number Addition Samples
Figure 23: Performance of NanoGPT model trained exclusively plain addition, but with extended vocabulary including both addition and alphahets (vocabulary size 80). Few-shot prompting. using both correct addition examples (1, 2, 3-shot) and incorrect addition examples (noisy-prompt) leads to enhanced performance, while the use of text prompts results in degradation of performance when the model is trained solely on addition.
Figure 23 shows that few-shot prompting contributes to performance enhancement even when the model is confined to training on a single plain addition task. Even the presence of noisy prompting. simply providing the model with the A B C format yields performance nearly identical to few- shot prompting. aligning with the result observed by Min al. (2022). Conversely, we notice that text prompts negatively influence performance when the model is trained only addition. This finding reinforces our earlier observation in Figure 6 that the advantageous impact of text prompts originates from the combined text and addition data.
E.3 PROMPTING WITH TEXT
To extend on the fhw-shot prompting experiments from Section D.2, we also evaluate the effect of prompting the model with pure-text prompts. If few-shot prompting with addition samples improves accuracy through in-context learning, we expect few-shot prompting with text to hurt accuracy since the text exemplars are contee- We use five different types of text exemplars: short text prompt that is not present in the Shakespeare dataset, (ii) Prompt2: short text prompt extracted from within Shakespeare dataset, (iii) Prompt3: longer form text prompt extracted from within the Shakespeare dataset, (iv) Prompt4: prompt that includes numbers, and (v) Prompt5:
(i) Promptl: a
33

---

Published as a conference paper at ICLR 2024
long text prompt that is not present in the Shakespeare dataset. More details on the text prompts be found in Figure 24.
Text prompts for few-shot experiments
Examples of the different text prompts used in the few-shot experiment. Each exemplar is separated by >--=!.
Prompt 1. Short, & Shakespeare
Prompt 3. Long, Shakespeare
tu bruse
JULIET:
Xomeo!
hello, world
ROMED:
My dear?
ace you doing?
MENENTUS:
agi is coming
This is good news:
will meet the ladies. This Volumnia
bocm! stability
is worth of consuls, senators, patriciens,
Prompt 2. Short, E Shakespeare
LADY ANNE:
devil, God's seke, hence, trouble
JULIET:
not;
Romeo!
For thou hast made the happy earth thy hell,
2ill'd it with cursing cries and deep exclaims.
A11: Resolved. resolved.
SUCKTNGHAM:
I fear he will.
VOLUMNLA:
Now now, Catesby,
says your Lerd?
why, I proy you?
CATES2Y:
CORIOLANUS;
and my lord: Bly fled Kichmend; And Rackingham, back't with the hardy Nelahmen, power increaseth.
Nay! prithes, woman,
the field, and
thy general.
Prompt 4. Has number, & Shakespeare
1 16-12 That the code my heart, ah 1-6-1-2 Star
Like a river flows 17-23 Surely the 15-22
parling, so it goes 46-92 Some things are neant
I got my first real 6-string Bought it the five dime #layed it 'til my fingers bled '69
the
I think someday I might just 5-3-2-1 get real job half of life 1-2-3 in I'm getting off 17-36-8-2 the road and in a real job
4light
every time that 27-67-29 1 lock in the mirror these lines my 1-3-92-5 face getting clearer The past 45-5-3 1a gone
Prompt 5. Long, & Shakespeare
Is this the real life? is this just fantasy? Caught in landside, no escape from reality. Open your eyes, I'm just poor boy, need sympathy, Becauae I'm easy come, Little high, little low, way the wind blows doesn't reclly matter
up to the
see.
life
And it's now nevez
ain't gonna live forever
I just want to live while S'a alive My heert is like an open highway Like Frankie said, did my way
Destruction leads very rough
34

---

Published as a conference paper at ICLR 2024
girl's guitat, Lhey're jusz another vibratios And tidal waves couldn't 't save the world from Californication
earthquakea
I want to stay But need to go
be the best for you But just don't know what to do baby, say
you
The time we have spent together Kiding through this English
Loren ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum mattis in lea vel gravido.
Pellentesque Libero elit, scelerisque varius vehicula hendrerit et tellus. Prcin cenveilis neque nisl, lobortis est scelerisque tineidunt. bune venenatis auctor urns.
optent taciti saciosqu
torquent per nostro.
Figure 24: Text prompt exemplars for few-shot experiments.
(a) NanoGPT, Tost accuracy on plain addition
(b) GPT-2, Test accuracy on plain addition
100
100
Test Test Accuracy (%)
Test Accuracy (%)
Zera-shot
Zera-shat
Prompt]
Prompti
Prompt2
Prompt2
Prompt3
Prompt3
20
Prompt4
20
Prompt4
Prompt5
Prompt5
0
10k 5k
15k
20k
25k
30k
35k
40k
0
5k
10k Number Addition Samples
15k
20k
25k
30k
35k 40k
Numbee Addition Samples
NanoGPT, accuracy on scratchpud
(d)
accuracy detailed scratchpad
100
Zero Shat shat
Test Accuracy (%6
Promptt
Prompt1
Test Accuracy (%6
80
Prompt3
60
Promptd
Prompt5
Zero Zero-shot
Promati's
40
Prompt2
Prompt3
Promgt5
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Number of Detailed Scratchpad Samples
Number of Detailed Scratchpad Samples
Figure 25: Experiments on few-shot prompting with different text prompts: (i) Promptl: short text not in Shakespeare dataset (ii) Prompt2: short text within Shakespeare dataset (iii) Prompt3: long text within Shakespeare dataset (iv) Prompt4: text with numbers (v) Prompt5: long not the Shakespeare dataset. Each prompt (Prompt 1-5) consists five distinct exemplars. The solid lines represent mean performance across the five exemplars, while the shaded area indicates the standard deviation. We observe that the effectiveness of text prompts varies greatly depending on the exemplars used.
The results presented in Figure 25 show notable variations in evaluation accuracy for addition, depending on the chosen text prompts. Longer text prompts (Prompt 5) typically result in a more significant decline performance. With the exception of NanoGPT trained on plain addition, the result in Figure 26 indicates that employing text prompts followed by test addition queries tends to have an adverse impact on the overall model performance, whereas incorporating relevant few-shot exemplars (1/2/3-shot) is beneficial. This aligns well with our intuition on the benefits on in-context learning.
35

---

Published as a conference paper at ICLR 2024
(a) N.noGPT. Test accuracy on plain addition
(b) GPT-2, Test accuracy on plain addition
100
100
Test Accuracy (%)
Test Accuracy (%)
80
40
Zero-shot 1-shot
Zero-shot
1-shot
2 shot
2-shot
20
text promot
text text_prompt
5k
10k 15k 20k
25k 30k 35k
40k
5k 10k 15k 20k 25k 30k 35k
40k
Number of Addition Samples
Number of Addition Samples
(c) NanoGPT, Test necuracy detailed scratchpad
(d) GPT-2, 2, Test accurncy detailed scratchpad
100
Zero-shot
Test Accuracy (%)
1-shot
Tost Accuracy (%)
2-thot
3-shot
60
Zero-shot
50-
1-shot
20
2-shot
40
3-shot
30
text prompt
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Number of Detailed Scratchpad Samples
Number of Detailed Scratchpad Samples
Figure 26: Performance of NanoGPT and GPT-2 model trained with entire Shakespeare dataset and a varying numher of samples of plain addition, and addition with detailed scratchpad dataset. Performance is evaluated on test prompts formatted plain addition and detailed scratchpad. Few-shot experiments are based on average of 5 exemplars, while text prompts involve average of 25 exemplars. The shaded area represents the standard deviation. Our observations indicate few-shot prompting consistently improves performance, whereas test prompts generally have a negative impact.
F FINE-TUNING, SCALING, AND PRETRAINING IN LARGER MODELS
This section focuses on bridging the gap between our experiments on NanoGPT and the more realistic setting of larger language models like GPT-2 and GPT-3. We begin by comparing the performance of NanoGPT and GPT-2 models when trained from random initialization. This comparison highlights the improved performance achieved with the larger model scale, especially in the zero-shot setting. Subsequently, we delve into the impact of tokenization methods and model pretraining in GPT-2 models. Our exploration reveals the crucial role of pretrained models and the consistent tokenization of numbers (achieved by introducing spaces) during the training phase for arithmetic tasks. Building on these findings, we proceed to fine-tune a pretrained GPT-3 model on various arithmetic tasks, employing different data formats.
Comparing NanoGPT and GPT-2. To examine the impact of scale on arithmetic performance, explore larger GPT-2 model with 85 million parameters, featuring twice many self-attention layers, heads, and embedding size compared to the previously used NanoGPT model. We train the GPT-2 model from seratch using character-level tokenization, jointly adopting both plain and detailed scratchpad formats; an approach mirroring the setting in Section 7. The results depicted in Figure 27 demonstrate that the larger model outperforms both plain and detailed scratchpad evaluations. For comprehensive analysis of GPT-2, including few-shot learning and the influence of text prompts, refer Figure and Figure
and addition tasks,
Going from character-level tokenization to BPE. The transition to GPT-2 setup necessitates several modifications. Firstly, we shift to OpenAl's Tiktoken BPE tokenizer, which is the default tokenizer for the pretrained GPT-2 model. featuring vocabulary size of 50.257. We also examined two different training approaches: training the model from random initialization (scratch) and fine-tuning the pretrained model sourced from Huggingface. To ensure uniform digit tokenization,
36

---

Published as a conference paper at ICLR 2024
(a) Test acc. on plain addition
(b) Test acc. on detailed scratchpad
100
100.0
Test Accuracy (%)
Test Accuracy (%)
99.5
80
99.0
98.5
60
NanoGPT, Zero-shot
98.0
NanoGPT, Zero-shot
40
NanoGPT, Few-shot GPT2, Zero-shot GPT2, Few-Shot
97.5
NanoGPT, Few-shot
97.0
GPT2, Zero-shot
20
96.5
GPT2, Few-Shot
96.0
5k 10k 15k 20k 25k 30k 35k 40k Plain Addition Samples
1000
2000 3000 4000 5000
Detailed Scratchpad Samples
Figure 27: Performance NanoGPT and GPT-2 model trained with entire Shakespeare dataset and a varying numher of samples of plain addition, and addition with detailed scratchpad dataset. Performance is evaluated on test prompts formatted plain addition and detailed scratchpad. Few-shot experiments are based on average of exemplars, while text prompts involve average of 25 exemplars. The shaded area represents the standard deviation. Our observations indicate few-shot prompting consistently improves performance, whereas test prompts generally have a negative impact.
alterations were made in data formatting to include spaces between numbers. This change aims to circumvent potential inconsistent tokenization of numbers while utilizing the Tiktoken tokenizer.
Figure 7 shows that GPT-2 demonstrates high performance in addition tasks with both character-level tokenization and Tiktoken with spaces between digits. This aligns with the results by Wallace et al. (2019), suggesting that character-level tokenization exhibits stronger numeracy capabilities compared to a word or sub-word methods. Furthermore, comparing the models trained from scratch and the models trained from the pretrained model, observe that fine-tuning pretrained model results in better performance compared to training a model from scratch.
GPT-3 experiments: Supervised fine-tuning, We extend our experiments to verify if our observa- tions hold while fine-tuning larger pre-trained models. In the following, we consider three GPT-3 variants: Ada, Curie, and Davinei. Note that since we perform fine-tuning using the OpenAl APIs, by default only the completions are loss generating tokens. Therefore, these experiments are slightly different when compared to the previous settings. We fine-tune these models using the same four data formatting methods as NanoGPT experiments: (i) plain formatting (ii) reverse formatting, (iii) simplified scratchpad, and (iv) detailed scratchpad. These formats are identical to those from our NanoGPT experiments except for one aspect. We introduce spaces between numbers in plain and reverse formatting to ensure consistent tokenization.
Due budget constraints, all experiments examples, and models were trained for 4 epochs. Performance evaluation was carried out on 1, 0.0 examples that were disjoint from the training dataset. Note that this training scale is significantly smaller than our experiments on NanoGPT, which employed 10, 000 training examples for 5, 000 iterations, with evaluations conducted on 10, 000 test examples. However, given these models' extensive pretraining on large data corpora, this scale can be deemed rational.
conducted using fine-tuning dataset of 1, 000
The results for addition and subtraction tasks are presented in Table 2 and Table 10, respectively. We observed that initiating with pretrained GPT-3 model significantly improves performance compared to training NanoGPT or GPT-2 models from random initialization with only 1000 samples. This indicates the utility of leveraging pretrained models for improved arithmetic performance. Interest- ingly, while reverse formatting and simplified scratchpad formats improve addition performance, they adversely affect subtraction performance. This observation is consistent with earlier finding depicted Figure 18, wherein transitioning from one data format to another often results in lower performance compared initiating training from random initialization. We postulate that this discrep- ancy may be due to the pretrained GPT-3 model's 's requirement to adapt to the reversed approach and "unlearn" its knowledge of plain formatting arithmetic, thereby introducing additional complexity. On
37

---

Published as a conference paper at ICLR 2024
the other hand, the detailed scratchpad method achieves excellent performance, albeit with increased training and inference costs due to higher token requirements.
Table 10:
Evaluation of subtraction performance for fine-tunes GPT-3 models: Davinci, Curie, and Ada. In each case, the model is finetuned on 1000 samples of addition in the corresponding format.
GPT-3 Model
Zero-shot
Plain
Reverse
Simplified Scratchpad
Detailed Scratchpad
Davinci
0.1%
84.8%
66.0%
15.4%
99.5%
Curie
0.1%
24.1%
6%
3.8%
92.5 %
Ada
0.0%
3.7%
2.6%
3.4%
81.5 %
Table 11: Evaluation of sine and square root performance for fine-tuned GPT-3 models: Davinci, Curie, and Ada. each case, the model is finetuned 1000 samples of addition in the corresponding format.
Sine
Square Root
GPT-3 Model
eps
Zero-shot
Plain
Detailed Scratchpad
Zero-shot
Plain
Detailed Scratchpad
0%
11.0%
10.3%
0%
0.7%
4.6 %
Davinci
5c-4
0%
35.9%
29.7%
0%
7.5%
17.2%
5c-3
0.4%
85.5% %
72.8%
0%
59%
60.5% %
0.0%
8.6%
1.2%
0.0%
0.7%
2.1 %
Curie
5e-4
0.4%
32.7 %
5.4%
0.1%
6.5%
6.0%
5e-3
0.9%
80.8 %
15%
0%
52.7 %
30.2%
0.0%
5.8%
4.3%
0.0%
0.3%
2.7
Ada
5c-4
0.0%
21.4%
9.1%
0.0%
3.8%
11.9% %
Se-3
0.3%
67.8%
25.2%
0.0%
32.2%
45.8%
For the more complex sine and square root tasks as shown in Table 11, we found that training with only 1000 samples is insufficient generate exact answers (eps=0). The GPT-3 model, fine-tuned with 1,000 samples, performs worse than the NanoGPT model trained with 10,000 samples. Further experiments with larger training datasets necessary for deeper insights and improved performance on these tasks.
It is worth mentioning that while few-shot prompting notably improves the performance of all three GPT-3 models, their zero-shot performance is quite poor (as shown in the lefumost column of the tables). However, post-training, few-shot prompting becomes less effective as OpenAl's fine-tuning process trains the model on individual prompts and desired completions serially, rather than in concatenation with multiple examples like our NanoGPT experiments. Consequently, our comparisons primarily focus on the zero-shot performances of each task.
G TOKEN EFFICIENCY ACROSS DATA FORMATS
Figure 4a demonstrates that more detailed training data leads to improved sample efficiency. However, this comparison does not account for the cost associated with training and inference. To address this, conduct
analysis based (i) the number of tokens within the train dataset (measuring the efficiency of the training dataset), and (ii) the number of tokens encountered during training. For (i) loken-efficiency of train dataset, we calculate the number of tokens within the dataset which can be derived by number of samples x number of tokens per sample. For instance, the mean token count for single training example reverse format, 64 for simplified scratchpad format, and 281 for detailed scratchpad format. For (ii) token-efficiency for training, we calculate the number of tokens encountered by the model which is number of iterations x context length of the model batch size. This approach ensures our cost calculation accounts for a vanilla implementation of attention with no additional optimizations (Pope al., 2023). Table 12 presents the number of tokens required for prompting and completion in each data format, per example. Evidently, the detailed scratchpad method uses considerably more tokens compared to other techniques.
3-digit addition task is 13 for plain format, 15 for
38

---

Published as a conference paper at ICLR 2024
The result in Figure 4b indicates that reverse formatting is the most token-efficient approach for train- ing dataset construction. While detailed scratchpad training is more sample efficient, necessitates a larger number of tokens per sample, both during training and inference. Given that the inference cost for commercial models is determined by the number of tokens utilized per inference call (sum of prompting and completion tokens), the of models trained escalate overall costs. Furthermore, since the cost of a single forward pass is quadratic in the number of tokens, this important consider. On the other hand, the result in Figure 28 shows that for the same number of tokens input to the model to be trained on, the model is trained faster with detailed scratchpad data. Therefore, for practical usage, it is crucial to evaluate both the number of samples needed for achieving the desired performance and the actual token demands during training and inference.
detailed scratchpad formats may
Table 12: Token requirements for prompting and completion per single example of 3-digit addition.
Plain
Reverse
Simplified Scratchpad
Detailed Scratchpad
Prompt
8
23
23
Completion
258
Total
13
15
64
281
100
Test Accuracy (%)
95
90
85
80
plain
reverse
75
simple scratchpad detalled scratchpad
70
1.5
2.0
2.5
3.0
Number of Tokens (1e8)
Figure 28: Model performance by the number of tokens input to the model during training.
39

---

Published as a conference paper at ICLR 2024
H LENGTH GENERALIZATION
In this section, we present results from experiments conducted assess the model's ability to gener- alize across different digit lengths. Initially, we exclude training examples featuring 2-digit operands from the s0,00e-sample addition dataset, yielding a reduced dataset of 7,655 samples, consisting solely of I or 3-digit operands. The model is trained with reverse format and its performance is evalu- ated on test dataset containing 100 random samples of I-iigit, 2-digit, 3-digit, and 4-digit additions. The results in Figure 29 demonstrate that the NanoGPT model is incapable of performing 2-digit and 4-digit additions. This suggests an inherent necessity for exposure to all digit combinations to perform accurate calculations and lacks generalization capabilities for unseen digit lengths.
Additionally, we investigate the model's s ability to extrapolate over larger digit lengths. The model is trained on 7-digit plain-formatted additions (each digit addition comprises 16650 samples, except 1 -diigit addition, which is trained on 100 samples). Its ability to add add 8-digit numbers is then put to test. The results in Figure 29 show that the model is unable to generalize beyond what it has been trained on. Similarly, when training the model on 10-digit binary numbers, it fails to generalize to I digit binary additions, further confirming its limited ability handle unseen digit combinations.
greater number of digits
(a) Trained on and digit addition
(b) Trained on
digit addition
100
100
Test Accuracy (%)
80
Test Accuracy (%)
80
digit
digit
digit
digit
60
60
digit
digit
digit
digit
digit
digit
digit
digit
0
5000
10000
15000
20000
0
5000
10000
15000
20000
Iterations
Iterations
Figure 29: Generalization experiments testing NanoGPT's performance on unseen numbers of digits in addition tasks. (Left): NanoGPT trained on reverse formatted addition with I and 3 digits, and tested on additions ranging from to 4 digits. (Right): NanoGPT trained up 7-digit plain formatted addition and tested additions ranging from
digits. both cases, NanoGPT exhibits inability to perform addition digits has not been exposed to.
We further explore the impact of detailed scratchpad formatting. The model trained on additions of up to digits, struggles to generalize 4-digit additions. Notably, it randomly drops single digit from the 4-digit number, erroneously perceiving it as a 3-digit number. We illustrate this difficulty in Figure 30 through multiple detailed error cases, ranging from instances in which only the test query is provided (Case 1) to scenarios where all intermediate steps are provided except only the final answer (Case 5). The prompts are highlighted in light grey and the responses generated by our trained NanoGPT model are highlighted in light green. These cases emphasize the model's 's shortcomings in accurately managing larger digit lengths.
Examples for length generalization prompts
Results obtained by prompting the NanoGPT model with larger digits than those it was trained on. The model is trained using detailed scratchpad formats with 3-digit numbers. We evaluate its performance 4-digit numbers, with varying levels of provided information. The prompt input is highlighted in a light blue box, while the model's output is highlighted in light green box.
Case 1: Just asking the question
40

---

Published as a conference paper at ICLR 2024
Input:
3165+3541
Target:
kscrasch> [,,4, has digits.
[3,5,1] has digits.
[8,4,5] + 13,5, A=ll C-0 6+1+0=7 A->7 C-20
[8,4] [3,5] , A-[7]
41310-9
[8] + [3] A-13,7] C-0 8+3-0-11 A-51 2-11
11 11 A=11,3,0]
</scratch>
199
Case 2: Giving number of digits
Input:
8465+3541
Torget:
<acratch>
18,4,6,5] aas 4 digits.
[3,5,4,1] hag digita.
[8,4,6] 13,5,41 A-1l C-0 6+4+0-10 X-20 0-91
[8,4] [3,5] 2-[0] C-1 41311-10 3 0.1
[E] [3] A-10,0] C-1 8+3-1-12 A-32 8-1
[1 + [1 1=12,0,0] C=1 END
s/acratch>
1200
Case 3: Giving intermediate steps
Inpul:
8465+3541
Target:
<acratch>
18,4,6,5] has 4 digits.
[3,5, has 4 digits.
[8,4, 6,51 4,,5, A= 0=0 5+1+0=6 A-26
C->0
[8,4] [3,5] 2-(6)
A-[6] 0-0 4:510-9 A-59
C-50
[31 A=[3,6]
8+3-0=11
11 + [1 A= (,1,6, C=1
END
k/acratch>
119.
Case 4: Giving all but intermediate steps
Ingut:
8465+3541
Target:
<scratch>
[8,4,6,5] has digits.
[3,5,4,1] has 4 digits.
[6,4,6,5] [3,5,4,11 A=ll , 5+1+0=6 a-26 C->0
[8,4,6] + [3,5,4] 4] A=[6] 0e0 6+440-10
A-50 G->1
[8,4] [3,5] A-10,6] C-1 41511=10
A-50 0-91
0= c= 8-3+1=12 A-22 G->1
II + II 2-12,0,6] C=l END
k/scratch>
1006
Case 5: Giving all intermediate steps, all it has to do is aggregate the answer
Input: 346543541
Target:
escratcho
6, has 4 digits.
[3,5,4,1] has 4 digits.
18,4,6,5] 13,5,4,11 A-:]
5+1+0=6 A-26 C->0
[8,4,6] [3,5,4] a-16] 0-0 61410-10 à >0 C >1
[8,4] [3,5] A-(0,6]
4+5+1=10
2->0
[8] [3] A=12,0,6] e= 8-3+1=12 X-22 d-xl
I1 [ 8-12,2,0,61 C-1 END
</scratch>
1206
Case 6: Giving in-context example
Input: 8465+3541
Target:
<scratch>
41

---

Published as a conference paper at ICLR 2024
18,4,6,51 aaa digits.
[3,5,4,1] haz 1 digits.
[8,4,6,5] 1,,,5,4,1] , a=ll
S+1+0m6 a-26 C->0
[8,4,6] [3,5,4] B=[6] C-0 61410-10 A 20 C.
[6,4] [3,5] Am[0,6] g-l 4+5+1=10 4-30 g->1
[8]
0=(2,,06] c= 8-3+1=12 A-22 C->1
+ UU-. 2-12,3,0, c-1 END
</scratch>
12006
Input:
194643598
Target:
kscratcc>>
[1,9,4] has digits.
[3,5,81
digits.
[4,9,4] ,13,5,81 A=ll USO 4+8-0=12 A->2
[1,9] [3,5] A=[2]
9-5+1=13
[3] A-15,21 C-l 1+3+1-5 4->5
C->0
[1 + 11 A=15,5,2] C=0 END
s/acratch>
5
Case 7: Giving In-context example, and all intermediate steps
Inpul:
2465+3541
Target:
<acratch>
18,4, has 4 digits.
[3,5,4,1] has 4 digits.
18,4,8,8] 13,5, A-:l
5+1+0-6 C->0
[8,4,6] [3,5,4] [ 2-[6] c-0 61410-10 a0>0 C->1 [8, [3,5] Am[0,
4+5+1=10
[8) [3] x=(3,0,6] c=t 8-3+1=12 X-22 6-21
II I 8-12,3,0,6] C-1 END
</scratch>
12006
Input:
1946+3598
Target:
<scratch>
[1,9,1,6] has digits.
[3,5,9,8] has 4 digits.
[1,9,4,6] + 13,5,,,81 A=l) 0-0 6+8+0-14 A->4 c->2
[1,9,4] [3,5,9] a-[4]
41911-14
A4 C:s1
[1,9] + [3,5] A](4,4] c=l s+5+1=15 4->5 0->1
[3] 4=(5,4,4] c= 1-3+1=5 A-25
U + 2,15,,,444] C-O
END
s/scratch>
5
Figure Example results model's output when prompted with a larger number of digits those was trained on.
42

---

Published as a conference paper at ICLR 2024
1 EXPERIMENTAL SETUP
In this section, summarize the datasets, models and hyperparameters used for experiments. All of our experiments on NanoGPT and GPT-2 models are run using PyTorch 2.1 and CUDA 11.7 on Nvidia 2808 Tls and NVIDIA 3090s. Detailed dependencies are provided our github repository".
I.1 DATASET
In this section, we explain the details of the datasets used for our experiments. For arithmetic tasks, we construct our own datasets as described below while we use the standard shakespeare (Karpathy, 2015) dataset for text.
Arithmetic Tasks As mentioned above, for all arithmetic tasks, we prepare our own datasets. We refer to the training dataset for a binary operator f()) as Dirain = {(x),, xf), y4 ) where y: = ,f)) Similarly, the test dataset Diest is constructed by randomly sampling pairs of operands that do not appear in Dinn- During both training and inference, we then apply different formatting techniques (see Section 3), construct the final sequence that is input the model. We
would like to repeat that both the careful choice of samples in the training dataset as well as their formatting play a crucial role in the final performance of the model.
For text data, we use the Shakespeare dataset which was introduced by Karpathy (2015) originally featured in the blog post "The Unreasonable Effectiveness of Recurrent Neural Networks". It consists of 40,000 lines of dialogue carefully curated from William Shakespeare's 's plays. The dataset comprises of a total of 1,115,394 characters and 64 unique tokens(when using the character-level tokenizer that we employed in all NanoGPT experiments).
Text
1.1.1 DATA BALANCING
As mentioned in Section 3, we carefully sample our data to ensure that they are "balanced" with respect to the number of carries and number of digits. As mentioned earlier, sampling the operands uniformly at random would lead to an extremely skewed dataset. To avoid this, we try to (i) Balance digits by sampling lower-digit numbers with higher weights and (ii) Balance carry-ons by sampling such that we have equal number of examples with 0, 1, 2 and carry-on operations.
Specifically, create balanced dataset of 10, 000 samples. This dataset includes all 100 1-digit additions and a random sampling of 900 2-digit additions (including both (2 + 1) and (1 + 2) digit additions) and 9, 000 3-digit additions. For the 3-digit addition samples,
employ rejection sampling to ensure an equal distribution of carry-ons (0, 1, 2, or 3). For the test dataset, we uniformly sample 10, 000 addition examples that do not overlap with the train dataset. Results in Figure 2 and Table 13 demonstrate clear advantage of the employed data balancing methods.
For the train dataset, we follow a specific approach based on the number of examples. For sample sizes smaller than 10, 000 (e.g., 500, 1, 000, 2, 000, 3, 000, 4, 000, 5, 000), we include all 1-digit additions and a proportionate number of 2-digit samples (e.g., for total of 5, 000 samples, we include 900 x 5, 000/10, 10, 000 450 two-digit additions). The remaining samples are filled with 3-digit additions from the constructed train dataset of 10,000 samples. For sample sizes larger than 10,000 (e.g., 20,000, 40,000), we include all examples from the 10,000-sample train dataset and then add additional samples as needed. Similar to before, we perform rejection sampling to maintain an equal number of carry operations. Table 14. provides detailed information on the number of samples with I-digit, 2-digit, and 3-digit additions, as well as the number of carry-ons.
For the other arithmetic operations (subtraction, multiplication, sine, and square root), we construct the train dataset using the following approach: (i) For subtraction, we use the same pairs of operands that were used for addition. (ii) For multiplication, we include all 100 cases of a 1-digit number multiplied by a I I-digit number. Additionally, we randomly sample multiplications involving operands of up to 2 digits. (iii) For sine, we sample random number in [7/2, /,, π r/2) and truncate it to 4 decimal places. (iv) For square root, we sample random number between [1, 10] and truncate it to 4 decimal
hhttps://gii wtt.:tthh...cmemmm/eee http://eeeeee/etteennngeggg/ chinnninnngg thmetic
43

---

Published as a conference paper at ICLR 2024
places. For the test dataset, sample 10, 000 data points (7, 000 for multiplication) that do not overlap with the train dataset.
Table 13: Performance of addition on various data sampling methods used: (i) Random uniform sampling of operands; (ii) Balanced digits sampling more and 2-digit operations (iii) Balanced carry halancing the dataset contain equal number of carry-on operations. Experiments addition with zero-padding each operand oulput have and digits, respectively. We observe balancing the dataset can significantly improve the performance or arithmetic operations.
Data Sampling
Overall
I-digit
2-digit
Carry-O
Carry-1
Carry-2
Carry-3
Random
97.74
98.00
96.20
95.88
98.61
98.74
94.98
Balanced Digits
98.13
100.00
99.70
98.87
98.64
98.13
95.93
Balanced Carry-Ons
98.29
100.00
99.70
98.38
97.56
99.02
98.22
Table 14: Number of examples digit 1/2/3 0/1/2/3 carry-ons NanoGPT experiments on addition for different number of samples varying from 500 to 40, 000.
Total number
1-digit
2-digit
3-digit
O-carry-ons
I-carry-ons
2-carry-ons
3-carry-ons
500
100
45
355
163
141
97
99
1000
100
90
810
283
268
236
213
2000
100
180
1720
535
502
481
482
3000
100
270
2630
781
782
748
689
4000
100
360
3540
1020
1016
958
1006
5000
100
450
4450
1279
1271
1229
1221
10000
100
900
9000
2500
2500
2500
2500
20000
121
1937
17942
5000
5000
5000
40000
132
3939
35929
10000
10000
10000
10000
I.1.2 DATA FORMATTING
For each of the four formatting techniques, as applied to each arithmetic operation we provide the details below. (i) Plain refers to the simplest formatting where we simply create sequence as the mathematical representation of the corresponding operation (e.g., A3A2A1 + ByBNB, = C2C2C). For (ii) Reverse, we simply reverse the digits of the output so that they appear in increasing order from LSB to MSB (e.g., $AgAzA + B3B2B, = C, C2CS), (iii) Simplified Scratchpad and (iv) Detailed Scratchpad provide algorithmic reasoning steps like (Nye al., 2021: Zhou et al., 2022b) so as to help the model get more "information" " per sample. Our intuition is that this approach nudges the model towards actually learning the algorithm of addition or subtraction rather than merely trying to fit the training examples. Refer to Appendix J for detailed examples of data formatting for each arithmetic operation.
44

---

Published as a conference paper at ICLR 2024
Different data formatting methods for addition
Four input formatting methods used for the addition task:
(i) Plain: standard formatting of addition
(ii) Reverse: flips the order the output and encapsulates each data sample with the' symbol at the start and end.
(iii) Simplified Scratchpad: provides carry and digit-sum information for each step addition, from the LSB to the MSB
(iv) Detailed Scratchpad: provides explicit details intermediate steps of addition.
Plain
Detailed Scratchpad
128+367=495
Input::
128+367
Reverse
Target:
<scratcch>
9128-367-5949
[1,2,8] has 3 digils.
[3,8,7] has 3 digits.
Simplified Scratchpad
[1,2,8] + 13,6,
x=1] c=0 8-7+0=15 A-25 c->1
[1,2] + [3,6] A=(5)
2+6+1=9 A-29 0-20
Inpul:
[1] + [3] A=(9,5] C=O 1+340=4 4-24 C->0
128+367
[] (1 2=(4,9,5) c=0 END
Target:
</soratch>
A-25 C->1
9
C-20
A->4 C->0.
Figure 31: The four input formatting methods used for the addition task. We progressively increase the amount of detail with each format.
Note that we wrap each data sample in the reverse format with the 'S' symbol at the beginning and end as delimiter. We originally observed improved performance in both the plain and reverse formats when the operands and outputs were zero-padded to fixed length (e.g., and digits, respectively, for 3-digit addition). But later realized that a single symbol can effectively replace zero- aero-padding. While maintain the original plain format without padding baseline emphasizing the necessity for improved data formatting for efficient emergence we incorporate the 'S" e'eneassullation in our modified reverse format. For further details, refer to Appendix B.I.
Addition (+). We focus additions of positive numbers up to 3-digits, in which the plain formatting would look like A3A2A1 + ByB2B, = C3C2C,. For experiments on comparing data sampling presented in Figure 2, pad the two operands and the output with zero, be of length and respectively. For all other experiments, we do not utilize zero-padding. For Scratchpad- based methods (iñi, iv), provide the digit-wise addition (denoted A) and carry-on (denoted C) information for intermediate steps from the least significant bit (LSB) to the most significant bit (MSB).
Subtraction (-). We consider subtraction of positive numbers up 3 digits, written as = A C,CCC, , in (i) plain formatting, and SA3A2A1 - ByB,B, = CCC.3S in (ii) reverse formatting. As with addition, bcratchpad-based methods (iii, iv), present the intermediate steps of digit-wise subtraction and handling of carry-ons. These steps proceed from the least significant bit (LSB) the most sigmificant bit (MSB). the final result after computing all the digit-wise subtractions is negative, we subtract the number in the most significant bit (MSB) position multiplied by 10 the power of (number of digits in the output 1) from the remaining digits in the output. In Section B.4, we present an alternative version of the detailed scratchpad formatting for subtraction.
Multiplication (x). We consider multiplication of positive numbers up to 2-digits. (i) Plain formatting examples are formatted AzA1 BNB, C2 while (ii) reverse formatting is
'We deviate from the strict definition of "most significant bit" (MSB) and "least significant bit" (LSB). typically associated with binary numbers, and reinterpret them for the purpose of this paper the most significant. "digit" and least significant "digit", respectively.
45

---

Published as a conference paper at ICLR 2024
formatted $A2A1 ByB1 = C The (iv) detailed seratchpad method simplifies each intermediate step by conducting a series of multiplications between the first operand and each digit of the second operand, starting from the least significant bit (LSB) and moving toward the most significant bit (MSB). For each step, we multiply the result by an exponentiation of 10 corresponding to the relative digit position.
Sine (sin). We consider decimal numbers within the range |- /2 x/2], truncated to 4-digit precision. (i) Plain formatting examples are formatted as sin(Ag.AAzAAAAu) = Bg B For (iv) detailed scratchpad method, we include the Taylor series expansion steps for sine, which is represented as sin(æ) x ~ + J.² # + .... These intermediate steps involve exponentiation, which may not be any easier to compute than the sine operation itself.
Square Root (v). We consider decimal numbers within [1, truncated to 4-digits of precision for (i) plain formatting. For (iv)
with the format, written as Aa detailed scratchpad method, we enumerate each step of Newton's ' method to compute the square root function. The iterative formula given by In = (#n-1 z-=1): where zo initialized as the floor the square root value of the operand z. These intermediate steps involve a division operation, which can be complex as the square root operation itself.
For evaluation of sine and square root, we classify the result y as correct if the absolute difference between yi and the ground truth value y: is less than or equal to predefined threshold 2 0.
1.2 MODEL
For all experiments, we use a Decoder-only Transformer architecture. Specifically, we primarily use the NanoGPT model, scaled-down variant of the GPT-2 model with half the number of self-attention layers, heads, and embedding dimension. Note that we use character-level tokenization instead of using the OpenAl's BPE tokenizer (Tiktoken) of vocabulary size 50257, making the vocabulary size significantly smaller. We use a learnable absolute positional embedding initialized randomly, following the GPT-2 model. Are results are generated using a temperature of 0.8.
In the case of arithmetic tasks performed on plain and reverse formatting, we set a context length of 256 for NanoGPT experiments. The length of single train example falls within the range of 13 to 15, approximately. However, when conducting experiments the context length 1024. This adjustment allows us to accommodate more examples per batch. In the case of simplified scratchpad, the length of each train example is approximately 64, while the detailed scratchpad has length of approximately 281. For GPT-2 experiments we fix the context length to 1024 for all experiments. See Table 15 for details on model configuration.
scratchpad formatting, we increase
For experiments on fine-tuning a pretrained large language model, we use OpenAl's GPT-3 model Ada, Curie, and Davinci.
Table 15: NanoGPT and GPT-2 model configuration
Model
Input Formatting
Context Length
Self-Attn Layers
Num Heads
Embedding Dim
NanoGPT
Plain, Reverse
256
6
384
Scratchpad
1024
384
GPT-2
Plain, Reverse
1024
12
768
Scratchpad
1024
12
12
768
I.3 TRAINING
Our overall experimental setup closely follows the standard training procedures for language mod- els Karpathy (2022); Brown et al. (2020). We train models using the autoregressive loss for all tokens, including the prompts, rather than limiting the training to only the answers. In addition, unlike popular approaches for synthetic tasks, we pack multiple training examples to fill up the entire context window. We also randomly sample the starting position, allowing the model to process shifted inputs.
46

---

Published as a conference paper at ICLR 2024
Text Predictin
Task Classiter
Layer Norm
Fonward
12x
Layer
Masiked Apention
Text Poetiion Embed
Figure 32: The GPT-2 Architecture. Image from (Radford & Narasimhan, 2018). NanoGPT model is a smaller model with half the number of self-attention layers, multi-heads, and embedding dimensions.
1.4 HYPERPARAMETER CONFIGURATIONS
In this section, provide a detailed overview of the hyperparameter configuration used in experiments in Table 16 and 17. To enhance memory efficiency and training speed, we employ flash attention. For most experiments, Nvidia 2080 GPUs, which do not support biloat 16, we switch to float16. It is worth noting that we did observe significant differences training and evaluation performance between the two data types.
utilize the 6 data type. However, when working with
The learning rate is chosen from (le-3, 5e-4, le-4, 5e-5) based validation loss. For the scratchpad format, NanoGPT is trained longer since the number of tokens per sample is higher and it requires more iterations to converge.
For the GPT-2 experimentation, we reduced the batch size to to accommodate the GPU memory limitations. However, to mitigate the impact of the smaller batch size, we employed gradient aceu- mulation steps. This approach involves taking multiple steps between gradient updates, effectively increasing the effective batch size to 64. For specific hyperparameter details, please refer to Table 17.
Table 16: Hyper Parameters used for NanoGPT experiments arithmetic tasks
Input Format
Batch Size
Optimizer
Betas
Iterations
Warmup Iter
Wt decay
Dropout
Plain, Reverse
256
AdamW
0.001
(0.9, 0.99)
5000 50000
100
0.1
0.2
Scratchpad
16
AdamW W
0.001
(0.9, 0.99)
0.1
0.2
Table 17: Hyper Parameters used for GPT-2 experiments on arithmetic tasks
Input Format
Batch Size
Optimizer
LR
Betas
Iterations
Warmup Iter
Wt decay
Dropout
Plain, Reverse
64
AdamW
0.0005
(0.9, 0.99)
5000
100
0.1
0.2
Scratchpad
64
AdamW
0.0005
(0.9, 0.99)
20000
0.1
0.2
47

---

Published as a conference paper at ICLR 2024
Table 18: Hyper Parameters used for tandem training experiments in Section 7.
Model
Batch Size
Optimizer
LR
Betas
Iterations
Warmup Iter
Wt decay
Dropout
NamoGPT
16
AdamW
0.001
(0.9, 0.99)
5000
0.1
0.2
GPT-2
40
AdamW
0.0006
(0.9, 0.95)
50000
2000
0.1
0.2
0.5k, Add: 10k
DS: 1k, Add: 10k
2k, Add:
Add:
Add: 5k
DS: 3k, Add: 10k
DS: 3k, Add: 20k
DS: 3k, Add: 40k
DS: 5k, Add: 10k
(a) NanoGPT, plain addition
(b) NanoGPT, scrntchpad
(c) NanoGPT, Perplexity
80-
Test Accuracy (%)
Accuracy (%)
1.30
Perplexity
1.25
60-
80
20
0
10000
20000
20000
40000
50000
30000
20000
20000
40000
50000
0
10009
20000
30000
40000
50000
Itera tions
(d) GPT-2, plain addition
(c) GPT-2, detailed scratchpad addition
(f) GPT-2, Perplexity
80-
2.0
Accuracy 1993
Accuracy (99)
Perplesity
2.8
80
20
1.4-
1.2
30000
20003
Iterations
Figure 33: Training loss curves for NanoGPT and GPT-2 trained with varying numbers of plain (Add) and detailed scratchpad (DS) samples well the shakespeare dataset described in Section 7. As the model continues to improve in addition accuracy as number iterations increases. However, the training perplexity on Shakespeare also tends to increase, which indicates some overfitting. However, we note that model still outputs "reasonable" text when prompted with shakespeare text.
J PROMPT EXAMPLES
In this section, we provide three examples of each formatting (plain, reverse, simplified scratchpad,
detailed scratchpad) of arithmetic operations (+, - sin, v)-
J.1 ADDITION
48

---

Published as a conference paper at ICLR 2024
Addition Examples
Plain
Detailed Scratchpad
266+738-1004
980-143-1723
396+262
11:34-75
Tazgel:
<scratch> [3,9, has digits.
Reverse
[2,6,21 has digits.
[3, 9,,966 112,6,21 A= C=0 6+240=a A->a
2-50
9913-524-14375
[3,9] [2, A=[8]
9-6+0=15
A-25
6->1
5226-598-8245
[3] + [2] A-15,81 C-1 342-1-6 A-26 0->0
935-58-938
A-16,5,81
s/acratch>
8
Simplified Scratchpad
Input:
796+890
9221244
kscraLch>
Target:
[7,9,8]
digits.
A-26 0->0
[8,9,0] has digita.
A-56 0->0
[7,9,8] 18,9,01 , A-11 C-0 640-0-6 A-55 0-50
A->1 C->1,
[7,9] [8,9] A=(6] 0=0 9-940-18
A->8
t->1
1166
[7] + [8] 188..20 -=- [8,,61 C=l 348-1816 A-36
6->1
A-16,8,6] C-1
285143
</scratch>>
Target: A 28 0 >0
Inpu::
A-52 C->1
788-389
A->3 C-X0.
Target:
328
<scratch>>
Inpul:
17,8,8]
digits.
993+849
[9, 19,8,91 has 3 digits.
Target:
[7,8,8] [9,8, A=ll 64940=17 , A-57
A-32 0-51
[7,8] [9,8] A-[7] C= 8-8-1-17 A-37 0:1
A->4 g->1
[7] +
[9] A-127,7 C=l 749-1=17 A->7 8-1
A= C=1
1842
s/acratchs
49

---

Published as a conference paper at ICLR 2024
J.2 SUBTRACTION
Subtraction Examples
Plain
Detailed Scratchpad
266-738=-472
980-743-2371
396-262
31-7
<acratch>
[3, has digita.
Reverse
[2,6,21 has digits.
[3,9,8]
-12,6,21 A= C-0 6-2-0m4 A-24 2->0
9913-524=9838
[3,9] [2,6] A=[4] C=0 9-6-0=3 A-x3
C-20
52266558--27--
[3] [2]
121.2-11 --113,41 [= C-0 3-2-0=1 A-21 0-50
[1 1-11,3,41
100+34-134
END
k/scratch>
Simplified Scratchpad
334
Input:
Toguo:
796-890
396-262
Target:
Target:
kscratch>
A-24 0->0
17,9,6] has 3 digita.
>3 C >0
[,,9,0] has digits.
A->1 C->0
[7,9,8] -1,,9,31 A-ll C-o
6-0-0-6 A-36 2-50
120+34-134.
[7,9] -8,9) A=[6] C 9-9-0=0
A-30
C-50
[]]
---13,6
7-8-0=-1 A-3-1
Inpul:
[1 11 8-1 1,0,6]
796-890
Targel:
9
A-36 0-50
Inpu::
A-50 C->0
788-369
A-2-1 C-9-1
Target:
100+5--54
994
[7, 8, has 3 digits.
Input:
[9,8,9]
digits.
788 989
[7,833] 19,8,9] A-1l C-0 8 0
A 051
Target:
[7,8]
[9,8] A=19] Cm-1 8-8-1-10-9 , A-39 0-5-1
A->9 0->-1
[7] [9]
A=19,31 C=-1 7-9-1=-3 A-3-3 6->-1
A-ンカ 0->-1
U|
11 1-1-3,9,91
201
-300+39 201.
</scratch>
50

---

Published as a conference paper at ICLR 2024
J.3 MULTIPLICATION
Multiplication Examples
Plain
Detailed Scratchpad
5+32=160
Imput:
56-76=5016
22+52
67+74-4958
Target:
[2,2] has 2 digits.
Reverse
[5,2] has 2 digits.
[2,2] + A-[4,4] k-1 3-14, C-0+40-4
35-32-0618
[2,2] +3, A= k=10 (= -=4411000-114 END
366-76-61053
</scratch>
357x74-85949
1141
Input:
8x69
Target:
<soratch>
[8] has digita.
[6,3]
[3] 1 9 A-(7,2] k-1 B-[7,2] C-0-72-72
[3] A= k-10 (=[4,,00 2=724480-552
</scratch>
2.
Input:
52-34
TargeL:
<scratch>
[5,2] has digits.
[3,4] has digits.
[5,2] A-12,0,8 k-1 B-[2,0,8] C-01208-208
[5,2] +3. A=(1,5,6) k=10 6, 0=208+1560=1 768
s/scratch>
SINE
Sine Examples
Plain
Detailed Scratchpad
sin (011L..031)-00877
Input: sin (1.0333
atn -41151
Target::
<scraLeh>
x_0=1.0313
x_1: x_0 1/3!
(x"31
x 11048484
x_2: x_1 1/5!
[x"51 x_2=0.8581
x_3: X_2 1/7!
(x"71
3-0-88878
x X.3 179!
[x"91 450.8578
FND
0.3578
Input:
s1a1-0.6909)
Target:
kscraLch>
x_0=-0.6909
x_1:
[x"31 1-
x x + 1/5!
(x"51
2-2--0.000 03.377
x
(x'7) x_3=-0.637
4: x + 1/9!
(x"9) 4=4066337
END
</acratch>
00.6375
Input:
0.5719)
Target:
Ksoratch>
x_0=00.37"
x_1: x_0 1/31
(x"31 1-
0,5408
x_2: x.1 1/51
(x"51
x_3: X 1/71
[x"
X-3-
3_3-00.541 00.414
x_4: x
</scratch>
51

---

Published as a conference paper at ICLR 2024
J.5
SQUARE RooT
Square Root Examples
Plain
Detailed Scratchpad
sget (7.2726)-2.6961
input:
sqrt (3. (3.6224)=1.3032
sqrt: (7.1042)
aqrt (1.0895)-
Target:
<sciatea>
X_0=2
x_1: 1/2-1211..1022/2222.1116 7.1
x.2: 1 x_2-2.6675
WF212..9957..0222./755222...333 X_3=2.6653
x_4:
x_4=2.6653
FNN
2.
Input:
(GEL(6.26EE)
Yarget:
Kacratch>
X_0=2
x_1: 1/22/2115.2668/2222288
x. 1/2 222122588744226882228877112112.. x_2=2.5041
x_3: 1/2112550044.22222/000022222.22
x_3=2.5033
x_4: 172222.91366,266222000322.2....0
x_4-2.5033
2ND
</scratch>
2.5033
Input:
(q.3(6.3226))
Target:
Kecratch>
x_0=2
x_1: 1/2=(228.2211/2))33.33 X_133.088
1/2-0350804488331166
x_2=2.8939
X_3: 1/2-112.833918.3225588 8291) x_3-2.8817
x_4: 1/221 -.58877, x_4=2.8847
k/soratch>
2.8847
J.6 NOISY SIMPLE SCRATCHPAD
We provide one example for each case of adding noise in the simplified scratchpad experiments discussed in Section B.5.
Noisy Simple Scratchpad Examples
We provide one example for each scratchpad experiments discussed in Section B.5. The input prompt is highlighted light blue, while remaining part is highlighted in 1ight green. We construct the dataset to have either correct or random digit-sum. and carry information C. For all cases, the final answer remains accurate.
adding noise in the simplified
Prompt:
Tnpub:
6861886
Target:
Correct A & C
Random C
Random A
Random A & C
0:>1
2- C-00
C-51
>
0-51
A-55
t->1.
k->5 C-11.
A-59
C->1.
A->2 C->1.
1572
52

---

Published as a conference paper at ICLR 2024
J.7 EXAMPLE DATA FOR GPT-3 FINE-TUNING
We provide an example from the training dataset consisting of one prompt-completion pair used for fine-tuning the GPT-3 model using OpenAI's API. The prompt highlighted in light grey, while the completion is highlighted light green. Note that for plain and reverse formatting, we include spacing between digits ensure consistent tokenization of numbers. "###" used as the stop sequence for generation.
J.7.1 ADDITION
Addition Examples
Plain
Detailed Scratchpad
S#EE
356+787
TargeL:
Reverse
<scrarch>
[3,5, has digits.
1849
[7,8,7] has digits.
[3,5,6] [7,8,7] A= C=0 6+7+0=13 A-33 C-11
[2,8] A=[3] C= 5-8+1-14
4-24
Simplified Scratchpad
[3] + [7] A-1,,111311 C-1 317-1-11 A-1 031
b-11,4,31
Ingut: 321981
</scratch>
4 3444
Target:
A->3 C->0
A 50 C >1.
J.7.2 SUBTRACTION
Subtraction Examples
Plain
Detailed Scratchpad
Iopoo: :
848 367
Target:
Reverse
sscratch>
[8,4,8] has 3 digita. (3,6,) has digits.
13,6,71
8-7-0-1
[8,4] [3,6] A=[1] C=0 4-6-0+10-8 A-28 (->-1
[8] [3] A-18,11 C=-1 8-3-1-4 A-24 C-00
Simplified Scratchpad
I II 8-14,8,11
400+81-481 END
Ingut:
</soratch>
595-489
8 1+94
Target:
X-26 0-3-1
>0 >0
A-32 c->0
200+6=206.
206
53

---

Published as a conference paper at ICLR 2024
J.7.3 SINE
Sine Examples
Plain
Detailed Scratchpad
(1n(-0.8649) -0.7511ffF
Inpet: :
sin (-1.35166)
Target:
X x_0--1335 35516
x_1: 1.3516
1/31
(xxxxx)
0.3101
x.2: -0.3101
175!
[x<2*29**8]
x_2--0.9777
-0.9777
1/71
3_33-00.761
X 4: -0.9761
1/91
X x_4=-0..78 9766
RND
-0.9762#8f
J.7.4 SQUARE RoOT
Square Root Examples
Plain
Detailed Scratchpad
sqrt (1.2179)
Input:
1. 103555588
sqrt (8.5808)
Targel:
<scratch>>
X_0=2
172-:2255.5000/222233322 x_1-2.3952
x_2: 581.1/2885522..888.222222 26262,, x_2=2.3625
x_3: 122. 580.2255555588222266
x_3=2.3623
8 (2.35233335 1772223333335 5808/2 36231 2623)-2.3623,
x_4=2.3623
END
k/scratch>
2.3623444
54


Reviews:
[{"review_id": "t3rZizGiAd", "reviewer": "Reviewer_9wwh", "summary": "The paper investigates the performance of Transformers (small scale) on a set of arithmetic tasks when trained from scratch with next token prediction objective. The study is motivated by the emergent ability of LLMs in solving arithmetic problems even though they are not directly trained on these tasks. The main task is multi digit arithmetic (-,+,*), most of the paper is focused on setting where the task is only multi digit addition. \n\nIn the paper,\n1. They study the impact of input representation (reversing the output digits helps the models learn the task faster).\n   - In the setting where they reverse the output digits, the model learns the task but there is a sharp phase transition as they increase the number of training samples. They try to explain this by the argument that learning a map on `n` digits from random samples is equivalent to completing a low-rank matrix (but they mention that this doesn't explain the generalisation behaviour of the models). \n  - For the multiplication operation, reversing the output does not have a positive effect.\n2. They study the impact of data distribution (balanced vs non-balanced)\n3. They show that using chain of thought during training helps (the more detailed the better).\n  - In this context, they compare the models in terms of sample efficiency and token efficiency, and show that models trained with CoT are more sample efficient but in total they require more number of tokens.\n  - They show that a detailed scratch-pad doesn't help with operations like sine and square-root.\n4. They show that the techniques of reversing and using CoT during training, stay as sample efficient when the complexity of the task grows in terms of the number of digits, where is training the models on the plain format of the task becomes harder (requires more samples) as the number of digits increases.\n5. They investigate the effect of training the model on a mixture of text and arithmetic data. \n6. They investigate the effect of models size (comparing nano-gpt and gpt-2, and pre-trained/fine-tuned GPT-3).", "questions": "1. When chain of thought is applied during training, does the model also generate the chain of thought during inference? Is there any correlation between the validity of the chain of thought (during inference) and the correctness of the answer? \n2. Could the reason for better performance of the detailed chain of thought model simply be its length? \n3. In Table 1, could you report a confidence interval? In the caption, you say in some cases it improves (when some numbers are excluded from the training data)? If the difference here is significant (it actually is a an improvement), what is the intuitive explanation? You mention a regularisation effect, could you elaborate on that?\n4. In Figure 4, are the models with scratch pad also with reversed output?\n5. The paper argues that the sudden jump in accuracy can be explained if addition is formulated as 2-rank matrix completion, but the generalization behaviour of the model can not be explained by this. Could you elaborate how this explanation holds even though it is not fully consistent with the behaviour of the model?\n6. When comparing models with different sizes, is the smaller model trained for longer?\n7. Is there a reason why the number of digits for experiments in Figure 6 are different for different operations?\n8. What is the number of digits for experiments presented in Figures 7 and 8 and Table 2.\n9. Do you have any experiments where the different operations are not split into different tasks (where the task contains the mixture of operation)?\n10. Is there a failure point as you increase the number of digits?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Lot's of interesting analysis. \n- Maybe a difference with some of the previous work on this topic is that here the objective, similar to language models, is the next token prediction, as opposed to modelling the task as a classification task. This is basically training language models on arithmetic data. The paper aims to reveal the factors that lead to emergence of arithmetic capabilities in a minimal setting.", "weaknesses": "While it is very interesting to understand if, how and under which settings language models learn simple arithmatics, It's not clear to me how the findings in the paper can be generalised. \n- For example, even in case of the ability of models to learn arithmetics, as mentioned in the paper, these results are based on using character level tokeniser which simplifies things a lot when it comes to such tasks and is potentially one of the biggest challenges for LLMs to perform well on these tasks. (A parallel work that looks into this: https://openreview.net/forum?id=OinvjdvPjp).\n- Methods like reversing the output seem a bit tacky. I agree it is interesting to see that these types of modification to the input/output impacts the results, making it easier for the model to learn the task, but I am not sure if they can have any value beyond analysis/understanding purposes. \n- While some of the experiments presented in the paper are conceptually very interesting they seem to be exploring the space a bit sparsely which makes it harder to make any firm conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zQ3wvr8Ve4", "reviewer": "Reviewer_UmA7", "summary": "In this study, the researchers explore the ability of small transformers, to grasp basic arithmetic tasks using the next-token prediction objective. The research reveals that simple alterations in the format of the training data, such as inverting the results or incorporating step-by-step breakdowns, can markedly enhance model accuracy. Furthermore, the paper delves into the intricate relationship between arithmetic and text data in training, as well as the length generalization challenges encountered by these models. This research underscores the significance of refined and targeted data, keeping in mind the unique traits of the next-token prediction objective, in swiftly fostering arithmetic capabilities.", "questions": "How do the authors envision scaling their methodology beyond GPT-3? Additionally, do they believe their approach is compatible with subquadratic transformer variants?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Well-written, technically sound paper", "weaknesses": "The study's main shortcoming in terms of novelty stems from its reliance on previously established methods and datasets, particularly the use of reasoning-augmented data, which has been prevalent in enhancing model performance. The authors explicitly acknowledge that their work doesn't break new ground in terms of the types of training data or in achieving peak performance with minimal model parameters. However, the research sets itself apart through its meticulous ablation studies and in-depth exploration of various sampling techniques, training data formats, data source mixing ratios, and model scales. Additionally, while they provide certain novel theoretical explanations for observed phenomena, their primary emphasis on arithmetic isn't for its intrinsic importance but as an easily testable emergent skill to better understand emergent phenomena in models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "03bHoimYcA", "reviewer": "Reviewer_ofJx", "summary": "The authors present an analysis of the performance of a small decoder-only transformer (NanoGPT), trained from scratch on arithmetic tasks (2 or 3-digit positive integer addition, mainly). They show that 3 digit addition can be learned to very high accuracy, from less than 5000 training examples, if the output digits are represented in \"reverse order\" (i.e. representing 256 as the sequence [6, 5, 2]), and less than 2000 if the model is provided chain of thought information (intermediary steps in the calculation) during training.\n\nOn two-digit addition, the authors observe that learning addition from a small training set amounts to completion of a low rank matrix (the addition table). They show that NanoGPT has the same data efficiency as classical completion algorithms, both needing about 2000 examples to \"fill the table\", but that NanoGPT overcomes one of the main limitations of classical algorithms: the need to have at least one example on every line and column.\n\nThe authors also present extensions of their approach to longer operands, and other mathematical operations (on integers and decimals). Finally, they present experiments with larger models (pre-trained or not), showing that whereas larger models, like GPT-3, achieve better performance with few-shot learning, their observations on the role of reversed digits in the output, and chain-of-thought prompting, remain valid.", "questions": "* To which extent is your finding on reversing digit order in the output specific to the decoder-only architecture you use? I believe output order would be irrelevant in an encoder-only setting (possible here because the output sequence is guaranteed to be shorter than the input sequence), what about an encoder-decoder architecture?\n* In the paragraph on balancing digit, you say : \"For instance, in the case of 3-digit addition, random sampling results in a meager 0.01% probability of selecting a 1-digit number.\" The probability of selecting a 1-digit operand should be 1%, right? \n* What are the performances of the model when operands have different lengths (e.g. 2 + 312, or 546 + 7)?\n* Does the model learn the properties of addition, e.g. commutativity? This could be done by testing whether model predictions for A+B and B+A are the same, even early during training.\n* Can those results generalize to decimal numbers (e.g. 1.21+13.12)? \n* Can those results generalize to signed numbers, by adding a sign token to all three integers?\n* There is a tension between your results from section 3 and 4, which suggest that models are learning the same algorithm as us (quickly memorizing the 10x10 table, then adding successive digits and propagating carries), and the results from section 5, which frame addition as a memorization+interpolation problem (learning to interpolate a large but low rank matrix). Is there a way to decide what algorithm the model is actually learning? \n * Is there a chain-of-thought approach suitable to low-rank matrix completion? If so, it would greatly improve its data efficiency, and perhaps allow it to scale to larger operands.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The paper demonstrates that a basic arithmetic operation, like integer addition, can be learned by small transformers from a limited number of examples. They also show that techniques like chain-of-thought prompting, introduced as a method for finetuning pre-trained models on arithmetic tasks, also benefit small transformers, trained from scratch. This is an important result.", "weaknesses": "The submission is a compressed version of a very long paper. As a result, some of the claims in the introduction, e.g. those relative to length generalization and compositionality, are not discussed in the main paper. The results on other operators, and the impact of pre-training on text, are very hard to assess, because the paper provides almost no description of the experimental setting. Finally, some of the figures (e.g. fig. 5) are compressed beyond legibility. This makes the main paper difficult to read, especially starting with section 7 (unless one is ready to read 25 additional pages). \n\nI would recommend that the authors either submit the full version of their paper to a journal, or limit it to their results on addition, which are significant enough, and deserve a longer discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t9S9W60kuA", "reviewer": "Reviewer_sewW", "summary": "This paper focuses on an important problem of trying to understand how emergent capabilities like being able to solve arithmetic tasks arise while training transformer-based language models. Since it is difficult to decouple the various factors like compute, data, and model size to understand emergent capabilities, this paper conducts extensive experiments focusing on arithmetic tasks like addition, subtraction, multiplication, and unary operations like sine/sqrt on small models like GPT-2/nanoGPT. The paper presents various key findings:\n\n1. Training data format and sampling is important to make the decoder models learn arithmetic tasks properly. Models can learn addition in reverse better compared to the standard addition (with most significant digit being predicted first auto-regressively). The training data should have a good distribution over the number of carry operations (for addition/subtraction) as random sampling leads to an imbalance and reduced performance.\n2. Chain-of-thought style prompting / scratchpad based training leads to improved performance. Although there's a tradeoff, as the number of tokens to be generated to get the final answer have significantly increased.\n3. Mixing text data with arithmetic task data during training does not lead to performance degradation, just that the number of samples required to achieve emergence for addition becomes higher.\n4. Generalization beyond the digit lengths present during training is hard. If the model is not trained on a specific digit length, the model is not able to perform well even when it is trained on various digit lengths excluding this specific length.", "questions": "I have asked most of my questions in the weakness section, but here are a few more:\n\n- Figure 1, $A$ is built incrementally, like $A = [5]$, then $A = [9,5]$, and so on. What is the effect of starting with $A = []$ (empty set) in the prompt, as the first line just starts with a carry.\n- There's a slight confusion on the experiment setting in Figure 2. Is it plain addition with \\\\$ symbol and not the actual baseline? Also it seems that even with structured sampling, the overall performance on 2-digit addition is low when $n = 3$.\n- Do you have an equivalent diagram for Figure 9 for GPT-2 and GPT-3?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "I really liked reading this paper and going over the various results and ablations presented in the paper. I believe the paper has various strengths as listed below:\n\n - The paper tackles an important problem of understanding emergence in language models wrt arithmetic operations. Data with arithmetic operations is not inherently present in the pre-training corpus, but still the models can do some rudimentary level of arithmetic tasks with few-shot prompting. The paper has some good ideas on adding arithmetic task data to the pre-training, or adapting pre-trained models with supervised fine-tuning on arithmetic data with some caveats (adding spaces to ensure better tokenization).\n- One of the important contributions of the paper is the structured data sampling for making models learn addition - ensuring good distribution over $n - 1$ digit operations for $n$ digit learning, and also ensuring equitable samples of the number of carry operations.\n- The section on the equivalence of learning addition and low-rank matrix completion is insightful, and how transformers have generalization capabilities beyond matrix completion.\n- The paper is well written and builds the story coherently, with lots and lots of ablation studies in the appendix and additional results.", "weaknesses": "- I was curious about the \\\\$ symbol bit present here and there in the paper, but it became clear after reading appendix B. In my opinion, the baseline used for making the strong claims of models being able to handle reverse addition better in the paper is wrong. The authors should have either done the reverse methodology without \\\\$ or used \\\\$ for the baseline too. This is a bit concerning as data formatting techniques highlighted in the paper are touted as one of the important contributions, and this finding makes that invalid.\n- Expanding on the previous point, Figure 9 specifically highlights that even plain addition is able to reach almost 100% test accuracy with the addition of the \\\\$ symbol, and reverse without \\\\$ is almost at 90% (only 2/3% difference with the baseline).\n- I believe some parts of the original manuscript is appendix material and some important bits in the appendix should be moved to the main paper. Specifically, the lemmas in section 4 seem a bit irrelevant given that difference of \\\\$ symbol in the baseline and reverse addition. Appendix section B.1 should be present in the main paper.\n- Not a weakness since the paper is fairly well written, but here are some typos in the paper:\n    1. I believe a latex shortcut is used to represent $A_{3}A_{2}A_{1} + B_{3}B_{2}B_{1} = C_{3}C_{2}C_{1}$ in the paper, because all occurrences of $B_{3}B_{2}B_{1}$ are represented as $B_{3}B_{1}B_{1}$.\n    2. Figure 1, in the detailed scratchpad solution, I think a carry ($C = 1$) has been missed after $[1, 2] + [3, 6], A = [5]$.\n    3. Figure 4b, it should be just Number of tokens on the x axis instead of Number of unique tokens?\n    4. Figure 8: Larger model scale instead of sacale.\n    5. No space between $\\textit{text}$ and $\\textit{and}$ in the last line on page 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4HI4xfBi0E", "reviewer": "Reviewer_twgj", "summary": "This paper investigates why arithmetic capabilities emerge from next-word prediction. The authors do several experiments with Transformer models between 10.6M and 124M parameters trained from scratch, and GPT-3 fine-tuned. The findings for the models trained from scratch is that for arithmetic they can generalise from relatively few examples (~6k), but generalise better when the output is reversed because this allows learning a simpler algorithm for arithmetic. The authors also find that holding out entire numbers from training doesn't decrease performance. Using scratchpads during training makes the models more sample efficient (even when accounting for the extra tokens this costs). The findings are similar for arithmetic with up to 10 digits and for different functions like subtraction. The authors also mix in text during training, and the produced model gets 100% arithmetic performance with few-shot prompting. In the third part the authors fine-tune GPT-3 and find further improved performance and sample-efficiency.", "questions": "- I don't understand why you would call the learning curve between 1k and 4k examples a phase transition? You're learning a task with a deterministic output, and 100% accuracy is possible and this model happens to learn it between 1 and 4k examples. This doesn't seem to be a fast phase transition when the model is anyway only trained for 6k examples.\n\n- It seems like the main reason for the model to learn arithmetic in this setup is because every example it sees is an entirely new one (new pair of operands), so the only strategy to achieve low training loss is actual arithmetic. It would be interesting to learn whether the generalisation behaviour is different when you train on the same set of examples for multiple epochs, which in turn could in fact explain some of the emergence of arithmetic in LLMs who are often trained for only a single epoch\n\n- I'm not sure I understand the exclusion experiments; For example for 1st (LSD) digit exclusion, do you just exclude 5 in the 1st digit? meaning that something like 543 cannot occur, but 453 can? If yes, how is that exclusion? The numbers are represented as a sequence over absolute embeddings, so 5 is not excluded at all but simply held-out at a particular position in the number no?\n\n- Typo end of page 8 (textand should be text and)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper is written well, a lot of experiments are done with some interesting findings (e.g. scratchpad training from scratch is more efficient even when accounting for extra tokens, holding out digits doesn't degrade performance).", "weaknesses": "The introduction says that the novelty of this work over prior work is to pinpoint the factors that contribute to the fast emergence of arithmetic capabilities through careful ablations, but I'm not sure how well the findings from this study would transfer to LLMs at scale. Additionally, I feel like some of the findings are not properly backed by the experiments:\n\n- The Transformers in this work are almost exclusively trained on arithmetic data, and it's unclear whether the findings would transfer to models trained primarily on language with some arithmetic data weaved in. Indeed, you show that arithmetic can be learned from next-digit prediction, but that is unsurprising since the setup is simply a cross-entropy loss on the next \"token\"(=digit).\n\n- The finding that reversing the output helps also most likely doesn't explain emergent abilities in LLMs, as this doesn't happen in natural text.\n\n- You show that few-shot prompting can be used to improve performance when text is mixed in, but you don't try few-shot prompting when text is not mixed in, so it's unclear whether the increase in performance is due to text being mixed in, or simply because of few-shot prompting. Few-shot prompting should also work when you train the model without text, or am I missing something?\n\n- The finding that sampling strategy is important is interesting, but on its own it's again unclear whether that transfers to emergent capabilities of larger models. For example, does the distribution of numbers in pretraining dataset follow a similar one as the one produced by your sampling strategy?\n\n- You claim you try to disentangle the factor of pretraining, but compare NanoGPT and GPT-2 from scratch to a fine-tuned GPT-3, which doesn't disentangle pretraining from scale. Why not compare to GPT-2 fine-tuned?\n\nIn summary, I feel like the motivation of the study (to explain emergent arithmetic abilities of LLMs) does not match with the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)