Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems
Authors: Tian Ye
Zicheng Xu
Yuanzhi Li
Zeyuan Allen-Zhu
Venue: ICLR 2025 Poster (2025)
Keywords: pretraining
language model
error correction
error detection
Primary area: interpretability and explainable AI

Abstract: Language models have demonstrated remarkable performance in solving reasoning tasks; however, even the strongest models still occasionally make reasoning mistakes. Recently, there has been active research aimed at improving reasoning accuracy, particularly by using pretrained language models to "self-correct'' their mistakes via multi-round prompting. In this paper, we follow this line of work but focus on understanding the usefulness of incorporating ``error-correction'' data directly into the pretraining stage. This data consists of erroneous solution steps immediately followed by their corrections. Using a synthetic math dataset, we show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy directly (i.e., through simple auto-regression, without multi-round prompting) compared to pretraining on the same amount of error-free data. We also delve into many details, such as (1) how this approach differs from beam search, (2) how such data can be prepared, (3) whether masking is needed on the erroneous tokens, (4) the amount of error required, (5) whether such data can be deferred to the fine-tuning stage, and many others.

Published as a conference paper at ICLR 2025
PHYSICS OF LANGUAGE MODELS: PART 2.2, HoW TO LEARN FROM MISTAKES ON GRADE-SCHOOL MATH PROBLEMS
LEXTENDDD ABSTRACT]*
Tian Ye
FAIR Meta and Carnegie Mellon University sigma 64 a com and
cmu. edu
Zicheng Xu
FAIR Meta (now Google Research) zi sichhnnxxu@meta com (now zi zichengxxu@google.com
Yuanzhi Li
Mohamed bin Zayed University of AI Yuanzhi Li@mmbzai........m..m ac.
Zeyuan Allen-Zhu
FAIR at Meta
zeyuanal lenzhu com
ABSTRACT
Language models have demonstrated remarkable performance in solving reason- ing tasks; however, even the strongest models still occasionally make reasoning mistakes. Recently, there has been active research aimed at improving reasoning accuracy, particularly by using pretrained language models to "self-correct" their mistakes via multi-round prompting. In this paper, we follow this line of work but focus understanding the usefulness of incorporating "error-correction" data di- rectly into the pretraining stage. This data consists erroncous solution steps im- mediately followed by their corrections. Using synthetic math dataset, we show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy directly (i.e., through simple auto-regression, without multi-round prompting) compared to pretraining on the same amount crror-free data. We also delve into many details, such as (1) how this approach differs from beam search, (2) how such data can be prepared, (3) whether masking is needed the erroneous tokens, (4) the amount of error required, (5) whether such data can be deferred to the fine-tuning stage, and many others.
INTRODUCTION
Language models have achieved near-human-leved performance in various tasks, including math solving, coding, and natural language understanding (Allen-Zhu & Li, 2023 Olsson et al. 2022] Ahn et al., 2024; Zhou al. 2023, Raghu al., 2021), and have been argued to have reached an L2 or L3-1evel of intelligence (Allen-Zhu & Xu, 2025). However, their problem-solving skills are still imperfect, sometimes resulting in logical errors. Recently, there have been numerous attempts to improve the reasoning accuracy of language models.
"The first six papers in the Physics of Language Models series were presented as a two-hour tutorial at ICML 2024 in Austria (youtu. be be/yyL/ JOK A 50-min deep dive into Part 2.2 is available at youtu.bee/ytee wwwpbbbbyyyxxxxxx02..3 Full and future editions of Part 2.2, including code release, are available physics. a alen- com g github.e. com com and s. com tboo

---

Published as a conference paper at ICLR 2025
User
User
User
(Linddd it. Each hox It. Cach box has box inside Each box has a box
Fach
invide
Go over the problem statement sentence by sentence to verify your Line 1.
Verify your answer. This is not correct.
weightt ghts pounds
Inside It. Each box weights pounds on its own. Each box
own. box has Each box weights 6 pounds inside Each
GPT-4a 4o (2024-08- d6 version):
GPT 4o 12024 08 C6 version):
has inside it. Each box weights 1 pounds on its owm. Each box has box Inside Each box has it kinside Cach box
Certalnlyl Let's go through the problem statoment sentence by sentence to verify the contents and sweight of each box, starting
go through the problem again and recalculate the total weight box X, ensuring we account for all the contents and their weights correctlv.
weights pounds
own. Each box O weights pounds on its own. Each box.A has box inside it. Each box K has box inside it. Each box Y inside box invide Each bos weightt O pounds
1. *** Each box Yf has box insicle it. This talls us that box contains box
L2
"*Box Weighs pounds own and contains a box
weightss pounds
own. Each
"*Box kt*; Weight 0 pounds on OWN
Fach box
Each box L has box inside it
contains box
on its own. Each box A has box inside it, Each box has a bos K. insidee Each has box inside
- This tells us that box contains box K.
***ox R**; Weighs 8 pounds on its own.
Total weight of box K 00+ 8 = pounds.
What the total weight box
14. "Eac box weights 2 pounds
-ottal weight
pounds..
GPT-4o (2024 08 066 versi
multi-round prompting, why can't it
It model can correct mistakes via
own.
This tells the weight of box
5. **Box 4**; Weights 9 pounds on its own
To determine the total we
correct the mistake immediately after?
and contains box Q, box and box Total weight
weight of each box inside
[box Q)- 10 [box + 8 [box K] 114 pounds.
working
way
24. **"ach box x has a box inside it,
1. "Box 1** Weighs 2 pounds.
Amissing box K)
tells box contains
6. "*Box x** Weighs pounds on its own
2. "Box gt*: Weighh pounds en win, but contains box
From broakdoww,, we can confirm that
contains box
hox L weighs 2 pounds on its own, as stated the problem. My Initial calculation for weight box was X correct.
Total weight of box X 124 [box V)+
8. x Weighs pounds on ownn, contains
120 pounds.
Therefore, the total weight of box pounds
Total weight of hox X 6 74 (box VI 80 pountee
X
Therefore, the total weight box X
120 pounds.
Figure 1: An illustration of how GPT-4o self-verifies and corrects own mistakes. Observation:
Correcting mistakes during generation, rather than after, can save inference tokens (the model avoids generating useless tokens following an erroneous step) and simplify the inference process (eliminating the need for multiple rounds of verification and correction).
One promising approach is to use verifier to check the correctness of the language model's out- put (Li et al., 2023a, Cobbe et al. 2021; /Zhang et al., 2024, Skreta et al., 2023, Yang et al., 2002)). Interestingly, some studies show that language models can "self-verify" (Weng et al., 2022, Madaan et al., 2024): They can be prompted to verify the correctness of their own generation, thereby im- proving overall accuracy. An illustrative example is shown in Figure 1
This leads to the following fundamental questions:
If a language model can correct its own mistakes after generation, (1) Why does it make those mistakes to begin with? (2) Why doesn't 't it correct the mistakes immediately during generation. instead of waiting until after?
There are many works that attempt to understand question (1). Notably, studies such as (Shu & Yu, 2024| Miller al., 2020; Anil et al., 2022) show that many mistakes arise due to "distribution shift" the training distribution of the language model differs from the prompts used during testing. Thus, even if the training data is error-free, language models can still make mistakes during generation.
Much less work focuses on question (2). While correcting mistakes after generation is a valid ap- proach to improve a language model's accuracy, is more desirable to correct mistakes immediately as they occur, such as "A- + B, oh I made a mistake, actually A + C." Doing so during genera- tion can save inference tokens (the model does not need continue generating based on an erroneous step) and simplify the inference framework (eliminating the need to call the model multiple times to verify and correct the answer), see Figure Thus, can we train language models to directly perform (2)?
A natural conjecture for why current language models cannot perform (2) is an alignment issue: There might be little to no training data in the language model's corpus to teach it to correct errors immediately. After all, people are unlikely to write an erroneous sentence followed by immediate correction in internet articles! However, even if we are given enough such "retry" training data (i.e., error + immediate correction such as "A + B, oh made a mistake, actually A = C"), is it clear that the language model can improve its overall reasoning correctness?
Potential harm of next-token prediction on data with mistakes: Training on data with mis- takes can be dangerous. By minimizing the next-token prediction objective, the model might learn to generate mistakes (such as "A + B²) instead of writing the correct steps (i.e., A + C).
IIf they make error when writing an article and realize it afterward, they will simply delete the error line in the final article.
2

---

Published as a conference paper at ICLR 2025
One could try to mitigate this by masking out the erroneous steps. However, unlike the "error correction after entire generation" type of training data where we can simply mask out the entire generation, 2 here, the error and correction steps are intertwined, making much harder to mask out the error only.
Unclear benefits over training on "perfectly error-free" data: Intuitively, if our final goal is to have model output correct reasoning steps to improve the reasoning accuracy, then why would training on "A + B, oh made a mistake, actually A = C" be better than training directly on the correct step "A = C Introducing errors is a distribution shift compared to what we want the model to generate during test time (i.e. a solution without error). Moreover, if we mask out the error part [A + B, oh made a mistake, actually], would it just reduce to training on perfectly correct data, with the model simply learning + C?
In this paper, we provide initial results towards understanding the usefulness of including training data in the language model's corpus that teaches it to correct errors immediately. Specifically, ask the following question:
Can training on retry data (errors and immediate corrections) successfully teach the language model to perform error correction? Can language model achieve higher reasoning accuracy compared to training on the same amount of perfectly error-free data?
To initiate our study and perform a controlled experiment, we need a setting where we can reliably collect data with errors and corrections. While it is tempting to use language models such as GPT-4 to synthesize such data, there is no evidence that GPT-4 can robustly generate errors or make those corrections consistently." 3 To avoid complicating our conclusions with the success rate and reliability of GPT-4, we need a different setting where we can 100% reliably generate errors and corrections.
In this work, we choose to use the iGSM dataset (Ye et al. 2025), which consists of a large number of program- discuss the dataset and provide examples in Section 2 to make this paper self-contained. Due to the generation procedure, we can also easily construct erroneous steps (simulating the reasoning mistakes that GPT-4 can make on such problems) and make 100% correct corrections to those steps. We choose this setting because mathematical reasoning errors are among the most widely observed errors made by language models. We outline our results in the following section.
(not LLM-generated elementary-schoo level math reasoning problems. We
Results 0-1: retry upon regret. We begin with warmup result. After pretraining the model on perfectly error-free math data, one can finetune it with a small number of trainable parameters (e.g., rank-8 update on its embedding layer) to detect errors in its solution steps. This finetuning process is remarkably lightweight and essentially indicates that the model "already knows" it has made a mistake (i.e., its internal states exhibit regretful behavior).!
Next, we consider a "retry upon regret" generation process. During the inference phase, the model detects an error in its generated solution step, it will "regenerate" from the end of its previous sen- tence. We demonstrate that this process leads to improved accuracy on this math dataset, surpassing what can be achieved using beam search.
Remark 1.1. This should not be confused with the "self-correction" works (Pan et al. 2023, Huang et al., 2023, Madaan et al. 2024), where the model evaluates its own output through careful prompt- ing. Here, we finetune the pretrained model for error detection (> 99% accuracy), and this error detection accuracy is significantly higher than that achieved through prompting.
Foor example, we can have training data like [MaskStart] A wrong math solution [MaskEnd] What is the error in the above solution?...
AAthoough GPT-4 can sometimes "verify" the correctness a generated solution, its verification accuracy is far from 100%. In the example of Figure 1. using prompts such as "Please verify carefully and correct your solution." "Please check your solution line by line carefully, matching it to the problem description." or even "You made mistake in Line I, please correct it." or correct it by checking against the problem statement carefully," GPT-4o can insist that its solution correct and makes no correction (using the 2024-08-06 version with temperature 0).
very you made mistake in Line 1, please
"Similar observations have heen made in (Liu et al., 2023, Li et al. 2023b Wang et al. 2024), where pretrained transformer can be easily finetuned effectively verify its own solutions (true false).
3

---

Published as a conference paper at ICLR 2025
While this provides strong evidence that language models "regret" their mistakes, improving final accuracy relies solely on randomness (for re- re-generation) to correct errors - similar to beam search," S This can take many rounds to re-generate, which is inefficient and requires the error detector to be highly accurate. It also alters the generation process, which might not align with the "general intelligence" framework, where the goal is typically for one model to perform all tasks using the same (autoregressive) decoding algorithm.
Results 2-6: pretrain with retry data. We now turn our attention to retry data. If the pretraining data includes errors and their immediate corrections (which we call "retry data"), the model can learn to not only detect errors but also correct them, achieving much higher accuracy compared to "retry upon regret" in Section 3 This differs from the "self-correction" line of work: the model learns to automatically retry after detecting an error, without requiring additional prompting or multi-stage generation.
Furthermore, we discover that even when the model is pretrained entirely on retry data with high error rate (e.g., for the iGSM data, p = 20% or even p 50%, meaning roughly half of the solution steps have an inserted error), it does not tend to produce erroneous steps during generation. The model still strives to generate "perfectly correct" solutions most of the time and only corrects itself on the rare occasions when it makes a mistake, leading to an overall improvement in accuracy. In fact, within a reasonable range. the higher the p value in the pretraining data, the better the model performs in reasoning accuracy.0 6 We also observe that it is not necessary to perform label masking on the errors, so the vanilla autoregressive (causal masking) training simply works.
Result 7: finetune with retry data. In contrast, if a model is already pretrained with error-free data, even using sufficiently many (additional) retry data, with a wide range of LoRA finetuning configurations, label masking or not, the model's accuracy does not significantly improve. This indicates that the skill of error correction can be very different from the original error- free reasoning, and thus requires major weight changes. potentially beyond what parameter-efficien fine-tuning (PEFT) can handle. For comparison, we show that full finetuning is effective when sufficiently many retry data is available, though this resembles continued pretraining.
Thus, unlike error detection (see Section where even a rank-8 adaptation on the embedding layer suffices), error correction is not a skill that can be easily adapted from model pretrained with only error-free data. This implies that retry data should be included in the pretraining stage for practical LLM training, rather than in the finetuning (alignment) stage.
Result 8: prepare fake retry data. Retry data can be difficult to obtain, so we explore practical methods to automatically augment correct math solutions with "fake" mistakes, ideally without needing to semantically parse or understand the correct solution. The most effective method we found is to introduce random future step B as "fake error" at each step A in the solution, followed by A as its "correction." " This approach encourages the model not to skip steps, even though some solution steps may be interchangeable in order so these may not be true errors. In the synthetic iGSM setting, this method is nearly as effective as retry data with perfectly designed errors and corrections, and it could potentially be adapted to real-life math problems.
By utilizing fully-controllable synthetic data (e.g., controlling error rates or label masking), conducting controlled experiments (e.g.. beam search vs. retry vs. error-free; pretrain vs. finetune), and performing fair comparisons (e.g., same number of training tokens), the goal of this paper to try to predict the needs of future LLMs. We do not claim that the synthetic data used here can directly aid in building future LLMs. However, given that commercial LLMs already employ synthetic data (Team 2024; Marah Abdin et al., 2024) and future LLMS are rumored to use Q*, perhaps crucial to understand how to best prepare and use such data effectively to teach models to learn from the mistakes. We give more detailed conclusion section in Section 6
Conclusion.
'Beam search does not have the "error detection" subroutine but uses the model's next-token prediction probability distribution to simulate a re-generation pracess.
'To provide strong comparison, for different we pretrain over the same number of tokens; higher p means we pretrain over smaller number of problems, because solutions with a larger p are longer.
4

---

Published as a conference paper at ICLR 2025
(Problem) The number of each Riverview Hlgh's Film Studio equals times much the each Film Studio's Backpack und each Dance Studio's School Daypack. The number of each Film Studin's School Daypack equals 12 more than the sum of each Film Studio's Messenger Backpack
each Central High's Studio's is School Daypack and each Film Studio's 's Messenger Backpack, The number of each Riverview High's Dance Studio equals the sum each Film Studio's Backpack, each Film Studio's s Messenger Backpack, each Film Studio's School Daypack and each Central High's Backpack. The number of each Dance Studio's School Daypack equals 17. The number of each Film Studio's ' Messenger Backpack equals 13. How many Backpack does Central High have?
number each Central High's
equals
of Dance
(Solution) Deline Dance Studio's School Daypack as so 17. Define Film Studio's Messenger Backpack W; so W = 13. Deline Hlgh'ss 's Studio
B = 17 + R = 12 20 9. Define Film Studin's Backpack as so W 9 13 22, Define Central High's Backpack as so B
Studio's is School Daypack g: R= 13 sa g
=7 Answer: 16.
Structure Graph
Dependency Graph
Central High
RivervIEW High
Clessroom
Pince Studio
Ilim Studio
RNG
Backpack
Daypa
Messenger Backpack
Tiversi Skido
neccesary paramter
unused paramter dependency
question paramter
dependency
Figure 2: An example math problem with op = 7 operations needed to compute its solution. A much harder example with op 20 is in Figure 9
2 SYNTHETIC MATH DATA FROM PRIOR WORK
Yc et al. (2025) introduced a family of controllable, synthetic datasets of math problems with step- by-step solutions. These data simulate GSM8K (Cobbe al., 2021), while removing arithmetic difficulties (by restricting computations to integers modulo 23) and common sense knowledge (e.g., candle burns and its length decreases). What remains is "logic reasoning." " The dataset has much larger diversity (over 90 trillion solution templates), and the solutions are fully verifiable. We briefly summarize to make the paper celf-contained, emphasizing some important aspects.
The structure graph describes the set of instance pa- rameters, such as "the number of school daypacks in each film studio." They also allow for abstract parameters, such as "the (total) number of backpacks in central high," which requires hierarchical computation
An example from their dataset is in Figure 2)
The exact data construction is not important for this paper. What matters is that the parameters form a dependency graph, as shown in Figure 2) where a parameter can be computed only when its predecessors have all been computed. To remove arithmetic difficulty, the computations are broken into binary operations - such as + 13 + 7 broken into (12 + 13) + 7 ensuring that failure to solve the problems is not due arithmetic difficulties. They op to denote the number of operations needed in the solution and prepared four families of data:
iGSM-med pg/gp uses op < 15 for train; op E {20, 21, 22, 23} for OOD (out-of-distributiin) test.
iGSM-hard uses op < 21 for training and op {28, 29, 30, 31, 32) for OOD testing.
Here, pq denotes the problem description comes before the question, and gp otherwise. They also introduce reask data for evaluation purposes: for instance, iGSM-med -00ee00ppp0eee re,reask a dataset con- structed by first creating (GSM-med pq and then re-sampling a parameter to query - this greatly changes their math data distribution, making good candidate for OOD evaluation. (After reask, the problem's 's op value may change.)
They showed that GPT-4/4o cannot solve such problems for op 10 (even with few-shot learning and their best efforts to remove English and arithmetic difficulties), indicating that the datasets are of some non-trivial difficulty. For completeness, we include in Figure 9 on Page 16 an example with op 21 to illustrate that these problems require non-trivial reasoning even for humans.
In this example, it equals A x (Bi + B2) for A central high's number of film studios, B1, B2 each film studio's 's number of school daypacks/ messenger backpacks.
5

---

Published as a conference paper at ICLR 2025
3 RESULT 0-1: LANGUAGE MODELS CAN RETRY UPON REGRET
Generative models solve math problems step by step a chain-of-thought (CoT) manner. Each step in our math problems is single sentence formatted as "Define [param] as X; so ...", as shown in Figure How do generative models make mistakes in this CoT process?
The most common reasoning mistake occurs when the model generates [param] that is not yet ready for computation (i.c., the model has not determined the values of all parameters that [param] depends on, also known "skipping steps", frequent error even in GPT-4 (Bubeck al. 2023)).
For example, in Figure 2, the model generates "Define Central High's s Film Studio" as the first few words in its solution, it cannot go back and erase this sentence, leading to mistake. Ye et al. (2025) confirmed that not only do language models pretrained using the iGSM datasets make mistakes in this manner, but even GPT-4/GPT-40 make such mistakes (when using few-shot). It's worth noting that the failure example in Figure 1 is also in this spirit.
3.1 RESULT O: MODELS CAN BE "REGRETFUL" AFTER MAKING MISTAKES
Interestingly, their same paper also implies the following:
Result 0. [corollary of (e et al. F0r models pretrained on iGSM (with correct solutions only!), during their solution generation process, after writing "Define [param] as" for a wrong [param], they often "realize" such a mistake, showing a regretful pattern in their internal states.
To see this, one apply their probing technique (illustrated in Figure 3(a)) extract information from the model's s last hidden layer after "Define [param A] as" to see if the model knows A can truly be computed next. This probing task denoted as can.next (AA) {true, false]. They found:
When A ranges over all possible parameters, the probing 99% accurately predicts can_next meaning the model knows if A can be computed next, even for the hardest op = 32 problems.
When the model makes mistake, the first sentence with mistake usually has (A)) false. Probing shows the model has ~60% chance of knowing can_next(A) = false, indicat- ing it often knows it has made mistake, right after stating the parameter name in full
This indicate that the model's internal states exhibit some "regretful" pattern, which can be detected via probing such as can_next, which almost just a linear classifier on top its hidden states In other words, error detection is easy and is a skill almost already embedded within the model's internal states, even when pretrained on correct math problems only.
3.2 RESULT 1: LET MODELS RETRY UPON REGRET
If a model knows it is mistake, why does it generate the wrong [param A] in the first place? The issue lies in the generation process. Before explicitly stating "Define [param A] as", the model might falsely think A is ready to compute among all the parameters it can focus on. After stating it, the model shifts its focus to the actual computation of A, and this the moment it can better realize that A is not ready for computation (using its attention mechanism) [1 Now that we know the model exhibits some "regret" towards the mistake, can we use this to improve accuracy?
Retry upon regret. We conducted an experiment using the probing result to guide the model's generation process. After generating each solution sentence, we use the can next probing to
*In practice, language models can make other mistakes such as arithmetic or common sense (see Section 2); however, the design of the iGSM datasets has removed such difficulties, allowing us to focus solely on the reasoning aspect.
"Nte that 60% accuracy is significant. If were a random guess. 50% accuracy would be trivial. However, the probing method is 99% accurate in predicting true or false, and only on a small set of examples (i.c., when making mistakes), it has a 60% chance of correctly predicting can_next( (A) 4)= false.
This aligns with observations that detecting mistakes is usually casy: works like (Liu et al. 2023, Li et al. (2023b) Wang et al. 2024), they show pretrained transformers can he casily fine-tuned to effectively verify their own solutions (truc or false).
Similar phenomenon also occurs knowledge partial retrieval (Allen-Zhu & Li. 2025a).

---

Published as a conference paper at ICLR 2025
Predict next(A] E (True, False)
jesedis Mil)
[304] The number BARRSIRN
Now may Seclpett (Sat] Delne
Deline Dentral
Film Studoo
p- 17-37-122 13 Delins Film Studio's School Deypuck
a prefix of solution (a) can next probing Ye et al. 2025). After pretraining, V-probing technique can detect if the model's internal states exhibit regretful pattern: right after "Define param A as," the model no longer thinks A is ready for compute.
problem
Question
some parameter
EAC RST of SOl
T:04 Fard
Pour
2I BOOD!
1F:
16:40
Hvg
orginal boant
55.0 50.3
100
833 868
I00 73
9 B6L
100 932
95-233
15.6
enpeed bourt
28.2
81.3
86.5
89.1
erginal buanc
sward
-132 50.4
13.2
100
79
60.6
23.
92.6 26.
24
03.2
22.5
[asen] zears
[0] 0m.
11
08.4
166
16 02.
00
100
06.
160
95 16
67.0
any wss meve) wenee2] beard
-232 50.2
22.9
23.7
86-1
723552
200
29-3 022
845 22.
200 22-
25.1942
00.8
26- 23.3
12.5
mny oat meter] verce2] beams
102
72.822
23.40.9932
85-1
233
25.1
5-2
86.8
222-264
12.
iory uze rgesl seard
mey
66.2
92.7
00.3
953
66.5
$2.6
repeet) (ersenP50) soand
66.
96.
91.9
64.06.000
100 98-3001
020
942 00.3 953
100 101
66.
2 100
100 66.1
96.2
27
93.6
P45
(b) Retry upon regret vs. original accuracies. Version 1/2 uses the can_next probing to detect regret (see Scction 3.2 they can detect errors to 99% accuracy), and versionP uses perfect error detector to decide when to retry.
Figure 3: The can_next probing and using it assist the model's generation process.
determine the model knows it has made a mistake. If so, we revert to the end of the previous sentence and regenerate. We use multinomial sampling (i.e., beam=l and dosample=true) during this regeneration process, with a maximum of total retries for generating each solution 12 We call this method "retry upon regret", and we carefully compare few different versions (details are in Appendix G.2).
The original can_next probing (Ye et al. 2025) freezes the pretrained model and fine-tunes rank-8 update on the embedding layer we call this "versionl", and it can 99% accurately detect errors."! 13
We additionally consider different can_next probing that fine-tunes only 0.25% data we call this "version2", and it can also 99% accurately detect errors.
parameters but with
For comparison, we also consider the perfect, ground-truth can_next error detector and call this "versionP"; and we use "versionP50" to indicate using a maximum of 50 total retries.
The goal of version/ check we can use extremely small parameter changes from the pretrained model for eтror detection and use this towards higher accuracies. The goal of version2 is to check if we can use a small number of fine-tuning samples for error detection. The goal of versionP/P50 for comparison purposes; after all, allowing perfect error detection can enable the model to correct nearly all mistakes retried infinitely.
We compared this with the original generation (with beam=1, 4, 16, and 32); and used beam=l or 4 for all versions of "retry upon regret." " The results give rise to the following:
Result 1 (Figure 3(b)). The idea of "retry upon regret" can improve reasoning accuracy, to an extent better than beam search. However, the accuracy gain can be limited unless the error detector is highly accurate. This also alters the inference process and increases the inference complexity.
We explain the three sentences in detail below:
Comparing version 1/2 with beam32, we see "retry upon regret" improves upon beam search. However, even though error detection is 99% accurate, this improvement is still marginal: about 2% for the op 23 (resp. op 32) case for iGSM-med (resp. (GSM-hard).
It is necessary to limit the maximum number retries avoid infinite loops.
See their reported probing accuracy table, where the can_next (A) probing accuracies can be 99% even for op 23 (GSM-med for op 32 iGSM-hard.
7

---

Published as a conference paper at ICLR 2025
(Solution retry rale 0.5) Define Dunce Studio's School Daypack us = 17. Define Film Studio'ss Schhool Seyaeh-Daypcccceeeeeeee [BACK]. Define Film Studio's is Messenger Backpack as W; so W 13. Define Central High's Classroom as [BACK]. Define Central High's Backpack as [BACK], Deline Central High's Film Studio B; B =p+
17 13 13=7, Deline Film Studio's School Daypuck as g. R W +7 20; so g 12 R 12 20 9. Befine Riverview Hirhrieeeettt----e Highh---eete--eeee-eeee Penee--eeeeeeeeeeeeee
[BACK] Define Film Studia's Backpack as w; so W + 22. Refine-ivveriew- Higheiiiii---ii---ee Highre Stndi-ttii-----i [BACK]. Define Central High's 's Backpack as c; s w 22
(a) A solution example identical to Figure but with retry-rate 0.5. The strikethrough like "Define Central High's Backpack as" is for illustration purpose,and the actual data is normal English text without strikethrough symbols.
KER ZE
IOPTER &
E-69
F-36a
S2F- ded----iii TOOO
P-GR:
arginl
15 18th muali
100 06.
M8.1
16.5
100 966
MG.
Mi.1
58.1
160
852
106 666
N1.
62.0
65.1
ney necaoe..
100 00-1
97.1
35.0
333
92.6
2
22
83.
F1.
100 22
02.
56.2
53.
25. 872
163
100 593
26.1 97.4
23.3
332 93
94.3
m7y neyyoee (ith Paai
100 10.1
08.1
94.2
9-5
18
109
22
26.0
20.3
19-3
825
21.
96.
54.
22. 87
100 999
56-2
94.8
11.
822-224
1ey
100 10.
35.1 950
87
13
100
214
96.2
56.4
100
18.
912
99.3
95.0
100 999
26.
97.3
16.1
95.3
4
100 06-1
97.4
15-4
91.
926 27
07000 100
98
93.1
96.2
92
100
06.1
96-
16.1
97.
96.7
248 973
106 PA6
06-1
24.3
93.0
93
1977 HETYEOO
06-1
16.1
12.
13.694
56.1
16.2
24.3
91.0
06.1
16.1
83-1
15.6
18.
106
01.2
1C.
107/ IETWOOD
29 50
(b) Comparison of models pretrained using iGSM data with retry.rate .rtee 0. For a stronger comparison, the model pretrained the retry vs original (error-free) data using the same number of tokens (i.e., retry data has fewer problems than original data) and identical training parameters, see Appendix G.I.
Figure 4: Pretrain language models on error-free vs retry data. Observation: especially on the hardest tasks (op = 23 or 32), models pretrained from retry data exhibit the greatest improvements for larger retry rate.
Comparing version1/2 1/2 with versionP, we see that the success of "retry upon regret" largely de- pends on an extremely accurate error detector - increasing the error detection success rate from 99% to 100% can significantly improve the final reasoning accuracy, but this is too ideal.
The idea of "retry upon regret" increases the inference complexity because one needs to keep an error detector model alongside and keep checking the correctness of the generated solution steps. In the event of error, the model needs to regenerate using randomness (possibly multiple times) until it passes the error detector. Ideally, one wishes to have just one model to achieve "general intelligence" using the simplest autoregressive decoding algorithm, without multi-round error corrections.
4 RESULT 2-6: PRETRAIN WITH RETRY DATA
In this section, we prepare pretrain data to teach the model to directly correct mistakes.
Math data with retry. Since use controllable, synthetic math dataset, can, at the beginning of each solution sentence, with probability retry. rrate E [0, 1). insert a wrong parameter that cannot be computed next, followed by a special token [BACK] |5 We repeat this process, so with probability (retry (retry_rate)² it may generate another wrong parameter at the same location, and so on. We provide an extreme example with retry_rate = 0.5 in Figure 4(a)) and a more complex example with op- 21 and retry_rate = 0.2 in Figure 9 We call this the retry data (with error and correction), to distinguish that from the original (error-free) data.
* Can a language model be pretrained using retry data to improve its reasoning accuracy com- pared with language models pretrained on the error-free data?
We conduct experiments on the (GSM-medpa//g and iGSM-hard p datasets for retry .rate rang- ing from 0.01 0.5. For a controlled comparison, we compare models when they are pretrained using the same number of tokens (along with other parameters) 16 This means, when pretrained with retry data, the model sees fewer math problems compared to the original training; when pretrained with larger retry_rate, the model also sees fewer math problems.
Another important question when pretraining with retry data is that:
After all, false negative in error detection results in a wrong answer, and having a false positive can result in the model regenerating too many times.
This parameter is uniformly randomly chosen from all such parameters, except those already appearing.
16e same training parameters such as batch size, total steps, learning rates, see Appendix G. In particular, we adopt the learning rate from (Ye et al. 2025) (which was best tuned for the original error-free data) directly the retry data.
8

---

Published as a conference paper at ICLR 2025
Is it necessary introduce label masks to prevent the model from learning the mistakes?
Adding masking might make intuitive sense because we do not want the model to learn the mistakes only want it to learn to correct emror if needed. To answer this, we also perform controlled experiment to compare (1) standard auto-regressive pretraining and (2) pretraining with masking: adding label masks to ignore the auto-regressive loss the wrong parameter tokens (i.e., ignoring text with strikethrough in Figure 4(a)).
Result 2-3. Our results in Figure 4(b) strongly support that:
Within a reasonable range." 17 the more mistakes the better. Especially on hard problems, such as on iGSM-med the accuracy jumps from 78% to 94% by using retry-rate rate 0.5.
Masking mistakes is unnecessary. We observe that it is generally not needed to introduce label masking on the error data even for large retry _rate 0.5.
A reader seeing this for the first time may find the above results unbelievable: if the pretrain data full of mistakes (even with retry_rate = 0.5), doesn't this interfere with the model's learning and encourage it to make lots of mistakes? To address this, we compute the statistics on how many times the pretrained model performs retries (i.e., how many [BACK] tokens it uses) during evaluation. We discover that (Figure 6 deferred to full paper):
Result (Figure 6). Models pretrained on retry data hardly retry (unless retry.rate is very high).
For instance, Figure 6(a) shows if retry_ = 0.2, even when pretrained without label masking, the model retries an average of < 0.3 times even for math problems with large op. This is because at each solution step, the retry data still has a higher step. Thus, a model pretrained from such data is still incentivized to generate correct steps (e.g., using a low temperature). If retry- _rate 0.5, this average retry count becomes 2 ~ 4, but can be mitigated using label masking. Only on those wrong solutions does the model try its best to retry and fail in the end, see Figure 6(b)
0.2 0.8 chance to produce
error-free
Another concern with the retry data is whether the model can still output shortest solutions. Ye et al. (2025) discovered that language models learn to produce shortest solutions without computing any unnecessary parameters." 19 Does the introduction of math data with errors increase the solution length? We discover that:
Result (Figure 7). Models pretrained on reiry data can still output shortest math solutions.
(Defer Figure 7 to full paper, which shows that when pretrained using retry data, model's generated solutions compute the same (small) number of unnecessary parameter as using original data.)
Combining Results 2-5, see that it is safe include retry data of this format pretrain data to improve the model's 's reasoning accuracy. There is no change to the pretrain or inference process, no need increase training time, and the solutions still satisfy many desirable properties.
Finally, it is tempting to compare retry data with beam search, which lets the model generate multiple solutions at each step and pick the most likely continuation based the ligeliielihood. As long as the model is pretrained on error-free data, even with 16 or 32 beams, its accuracy does not noticeable improve in all the cases (recall Figure ((b);; while in contrast, if the model is pretrained with retry data, the accuracy easily goes up in Figure 4(b) (and this holds even for beam=1) 20)
Naturally, retry-rate rate cannot approach 1. Exploring such extreme failure settings is not particularly inter- esting. For instance, retry rate 0.5 already sufficiently extreme, indicating that half of the solution steps contain errors.
''Even when retry.rate rate = 0.5, if errors are sufficiently random (say, in step there are correct possibilities but 8 possible errors), the model is still incentivized to say correct sentences. This could be reminiscent of language model's learning on context-free free grammars with mistakes (Allen-Zhu Li 2023); even though the model is trained only using data with grammar mistakes, it can output sentences correctly respecting grammar at lower temperature.
Theey discover the model achieves so via non-trivial mental planning precompute the of necessary parameters, before starts to even generate the first solution sentence.
2TTo present the cleanest result, in Figure 4(h) present the best accuracy among beam=1 or 4; one still observe high accuracy boost for beam=1 when pretrained with retry data.

---

Published as a conference paper at ICLR 2025
Similarly, "retry upon regret" lets the model re- re-generate another solution step if error is detected, and this for instance gives only accuracy gain 78% = 80% in the iGSM-med case (recall Figure 3(b)), while pretrain with retry data can give 78% + 95%. This is strong signal that:
pg
Result 6. Error correction is skill that can be fundamentally different from beam search or retry based on the model's 's randomness.
Therefore, to truly improve the model's reasoning capability, it is crucial modify the training data to include mistakes and their corrections.
5 RESULTS 7-8
Due to space limitation, we omit the technical details for Results 7-8 in this ICLR version to en- courage readers to refer our full paper at ssrn. com/ abst 5250631 We remark that the full paper underwent the ICLR 2025 review process, but we elected to present this camera-ready version as an extended abstract, aligning with the tradition in the theory community.
CONCLUSION
In this paper, we investigate whether language models can benefit from pretraining on data contain- ing mistakes, followed by immediate error correction. Using fully controllable synthetic setting, demonstrate that models trained such data outperform those trained on the same amount of error-free data.
In addition the accuracy gain, Section 4 shows that using retry data is very safe: the model rarely makes mistakes even after pretraining with high error-rate retry data, and it is unnecessary to change the training process (simply autoregressive, no need to label-mask the errors). Retry data teaches models how to correct errors if needed, rather than encouraging mistakes.
important to note that such error correction skill does not come easily. A model pretrained with only error-free data cannot use (1) beam search (2) retry based on error detection ("retry upon regret") to achieve comparable performance, see Section 3, unless the error detection nearly perfect. This error correction skill is also very different from the original error-free reasoning and thus cannot be learned during parameter-efficien fine-tuning (PEFT) such as LoRA, see Result 7. This implies the necessity of adding retry data to the pretrain data for language models to truly learn the capability to correct errors.
While grade-school level math problems have many other difficulties (including arithmetic com- mon sense), following Ye et al. (2025), we have focused on the (logic-following) reasoning aspect, which is of the weakest aspects of GPT-4
While it unlikely that iGSM retry data will be directly used for pretraining future commercial-level LLMs, this paper aims to find guiding principles for necessary ingredients. We strongly discourage using fine-tuning to teach model to correct errors (or using beam search, or letting the model regenerate upon encountering a mistake) these are not effective. We advocate for adding mistakes and corrections at the pretrain level. While commercial LLMs use synthetic data (Team, 2024; Marah Abdin et al. 2024) and future LLMs are rumored use Q*, remains question of how to prepare such synthetic data for the model to best learn error correction. Our Result 8 suggests that it is critical to teach a model not to skip steps. This can done either through naively creating retry_ -wrak data like ours or using more advanced prompting methods to encourage an auxiliary model to rewrite math data into such format. We cannot explore such follow-up directions due to GPU resource limitations.
Finally, Part 2 of this work series focuses on how language models solve grade-school math level reasoning problems (including Part 2.1 (Ye al. 2025 We also cover how language models learn language structures in Part (Allen-Zhu & Li 2023) and learn world knowledge in Part (Allen-Zhu & Li, 2024; 2025a b), as well how these may impact architecture design (Allen-Zhu, 2025).
2l In contrast, arithmetic such as 10-digit multiplications can he (and perhaps should he) handled by calcula- save the model's capacity for other skills.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to imple- ment preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
Zeyuan Aluen-Zhu. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. SSRN Electronic Journal, May 2025. doi: 5023390
https:
// //sstnn com/ wbw.rract=5220a5555
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 1, Learning Hierarchical Lan- guage Structures. SSRN Electronic Journal, May 2023. https:/ ssrn. com/ abstract=
5250639
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. In Proceedings of the 41st International Conference on Machine Learning,
ICML 2024, 2024. Full version available at htttps: //ap::///ssrn.. com/ /tt///att=tttt0333333
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.2, Knowledge Manip- ulation. In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025a. Full version available at htths:: : com/ abstract==5256221
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025b. Full version available at https: : / ssrn. com abstract= 5250617
Zeyuan Alu and Xiaoli Xu. DOGE: Reforming AI Conferences and Towards Future Civ- ilization of Fairness and Justice. SSRN Electronic Journal, February 2025. doi: 10.2139/ssm..
5127931. https: //tp://ssnn.ee com http:/abstratttt1111a33
Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Am- brose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. Advances in Neural Information Processing Systems, 35:38546-38556, 2022.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT- NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Work- shop on Challenges & Perspectives in Creating Large Language Models, 2022. https: 06745
/ otg/viirrrgaaaaaa..
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2203. 12712, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 1..1111......... 2021.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song. and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01790 2023.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Mak- ing language models better reasoners with step-aware verifier. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5315-5333, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10.18653/11/222.aad/ 291. URL https: //s:://aaa //aaaaaaathlloge
org/ 2023.022222 acl2.aacl-onng 291
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pp. 5315-
5333, 2023b.
11

---

Published as a conference paper at ICLR 2025
Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. TinyGSM: achieving > 80% on GSM8k with small language models. arXiv preprint arXiv:2312.092411 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri
Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Marah Abdin al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprini arXiv:2404. 222404.14219, 14219, 2024.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In International conference on machine learning, pp. 6905-
6916. PMLR, 2020.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 0ett22099.1888 11995, 2022.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self- correction strategies. arXiv preprint arXiv:2308. 03188, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.
Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 12111-1111222 2021.
Yiheng Shu and Zhiwei Yu. Distribution shifts are bottlenecks: Extensive evaluation for grounding language models to knowledge bases. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 71- 88, 2024.
Marta Skreta, Naruki Yoshikawa, Sebastian DeellanoRuubcce Zhi Ji. Lasse Bjorn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting, arXiv preprint arXiv:2303. /4100, 2023.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
Llama Team. The llama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
Fei Wang, Chao Shang, Sarthak Jain, Shuai Wang, Qiang Ning, Bonan Min, Vittorio Castelli, Yas- sine Benajiba, and Dan Roth. From instructions to constraints: Language model alignment with 2024.
automatic constraint verification. arXiv preprint arXiv:2403.063:
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.09551. 2022.
Kaiyu Yang. Jia Deng, and Danqi Chen. Generating natural language proofs with verifier-guided 2022.
search. arXiv preprint arXiv:2205.1244
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process, In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025. Full version available at https: 7//arn. com/ abstraat=5250629
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jackyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self-correct rea- soning, arXiv preprint 240i 17140, 2024.
Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? study in length generalization. arXiv preprint ar1iv:2310. 16028, 2023.
12


Reviews:
[{"review_id": "owpeaEDMSn", "reviewer": "Reviewer_ZAr2", "summary": "This paper improving math reasoning in LLMs by adding error-correction data directly into pretraining, instead of using the usual multi-round prompting---that dominant self-refine approach. Using a synthetic math dataset, the authors show that training with examples of mistakes followed by corrections leads to better reasoning accuracy, even beating models trained on error-free data. The study tries to answer some interesting question: how to prepare error-correction data, is finetuning sufficient to learn self-correction or is pretraining necessary and whether how this approach compares to beam search. Although on a synthetic and a very controlled setup, the results present some fresh perspective into pretraining LLMs to do better revisions.", "questions": "See weaknesses above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written. Although the results are dense, the authors did a good job summarizing and condensing the main takeaway messages.\n\n- The studied problem is interesting: We still don’t fully understand how to pretrain LLMs for effective reasoning from the ground up. This paper explores self-correction within the pretraining phase, a fresh perspective that hasn’t been widely explored in the literature, aside from a few works like Quiet-STaR.\n\n- The experiments and ablations are well-designed. The authors clearly state their research questions early on and address them in a logical sequence.", "weaknesses": "My main concern with this paper is the uncertainty around whether these findings will generalize to practical LLM pretraining scenarios. I'll expand on some specific limitations below:\n\n- The types of problems in the i-GSM dataset don’t reflect real-world reasoning tasks. Specifically, every reasoning step required here is limited to a single type of computation—finding the next node in a directed acyclic graph (DAG) and calculating its value. But what about other reasoning types where models need to compare values, handle ambiguity, or apply commonsense knowledge? Although I appreciate the focus on math reasoning, can the authors confidently assert that these results will apply to more complex, realistic reasoning tasks?\n\n- Current LLMs struggle with error detection, as the authors note in the introduction. However, their findings in L259-260 suggest that error detection can be effortlessly embedded within the model’s internal states. This may be due to the task’s synthetic nature, where the model could have learned to encode specific errors, like “skip step,” in its parameters. But this is unlikely to generalize to other errors. For example, could the model's hidden states reliably detect other error types, like incorrect calculations?\n\n- The paper’s fine-tuning experiment is limited to simple LoRA tuning. What about full fine-tuning using a fraction of the pretraining data? The authors mention (L484-485) that the cost of full fine-tuning would match pretraining with retry data, but this wouldn't hold if we fine-tune with just a fraction of the pretraining data. Would the results remain consistent in that scenario?\n\n- I’d expect more discussion on inference-time algorithms and their impact on performance. If I’m following correctly, most experiments use greedy decoding or occasionally beam search. It would be insightful to understand how additional inference-time resources—like generating more tokens or applying consensus voting—might affect error correction. Result 4 (concerning model retries) is based on greedy decoding; how would this result change with sampling?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0LSMbkTcps", "reviewer": "Reviewer_7os6", "summary": "This paper studies the impact of training language models with \"error-then-retry\" format data on the reasoning performance. With experiments on a synthetic GSM8k-style math reasoning datasets, the authors conclude that pretraining with such data improves reasoning accuracy than on same amount of \"error-free\" normal data, and LoRA fine-tuning with such data does not help.", "questions": "I would like to hear the authors' thoughts on the inference-time scaling properties of model trained with \"retry\" data: given a fixed inference-time computation budget, is it better for model to produce reasoning chains with retry (more tokens to reach final answer, but average accuracy is higher) or without retry (less tokens to reach final answer, but average accuracy is lower)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. I like how the authors approach this problem in a data-centric, rigorous way: controlled experiments on synthetic data. \n2. Experiments are solid", "weaknesses": "The paper mainly experiments with one type of error: inserting a wrong parameter that cannot be computed next. While this is easy to implement, it would be hard to simulate all kinds of errors that language models can make in real-world reasonings scenarios. This cast doubts on how the suggested approach can be deployed to train LMs on non-synthetic data. If the authors could provide some discussion on how the method can be generalized to different errors (e.g., math calculation error, context misunderstanding, ...) in a scalable and controllable way, that would be very promising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZWSXE7n6Cd", "reviewer": "Reviewer_SYew", "summary": "The paper aimed at improving language model accuracy in reasoning tasks (grade-school math) by pretraining with \"error-correction\" where errors are followed by immediate corrections, to teach models self-correction as they generate outputs. The authors show that pretraining with error-correction data boosts reasoning accuracy, without multiple rounds of prompting.", "questions": "The paper would benefit significantly from a reorganization into a structure that aligns more closely with the familiar flow of this venue. Specifically, it would help to present the experimental settings early in the paper, followed by a dedicated focus on results and analysis. Additionally, the authors could simplify the tables to better highlight key findings, and consider using plots to present some of the results for improved clarity and reader comprehension.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper proposed an interesting idea by incorporating error-correction data directly in pretraining, rather than relying on post-generation prompting for correction with empirical evidence of improved performances. \n- The authors compared different experimental settings, such as “retry upon regret,” masking, and pretraining versus fine-tuning with retry data.", "weaknesses": "- The paper has unclear justification for synthetic data choice.\n- Writing and the structure of the paper is very unclear and hard to follow. For example, the experimental setting or conclusion section is not contained in the paper.\n- While the paper presents extensive experimental results, the paper primarily focuses on using one model, it is unknown if such techniques are generalized to different models (e.g., LLAMA, Gemma, Mixtral etc.). The paper further lacks comparisons to existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7J6iQaoVcA", "reviewer": "Reviewer_ZAFG", "summary": "This work focus on understanding the usefulness of “error-correction” data into the pretraining stage. The experiment results show that error-correction data can improve the mathematical ability of the model more effectively than error-free data.", "questions": "- Will the experiment code and data be open sourced?\n- In section 3.1, as the can_next is “a linear classifier on top of its hidden states”, is the one who “knowing can_next(A)=false” actually the linear classifier rather than the model itself? Does the author mean that because the hidden states of the model show a distribution that can be accurately predicted by the classifier, the model \"knowing can_next(A)=false\"?\n- It would be appreciated if the authors provide some insights about why “the slightly more complex retry miss data does not improve accuracy by much”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The experiments are well-designed and rigorously conducted.\n- The finding that “even when model is pretrained on retry data with high error rate, it does not tend to produce erroneous steps” is interesting.", "weaknesses": "- Some previous work[1] has already pointed out that error-correction data enables LLM to achieve higher reasoning accuracy compared to error-free data.\n- This work only evaluates on IGSM dataset, which is a synthetic data. It would be more convincing to also experiment on realistic mathematical datasets.\n\n[1] Learning from mistakes makes llm better reasoner", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)