Training Language Models to Self-Correct via Reinforcement Learning
Authors: Aviral Kumar
Vincent Zhuang
Rishabh Agarwal
Yi Su
John D Co-Reyes
Avi Singh
Kate Baumli
Shariq Iqbal
Colton Bishop
Rebecca Roelofs
Lei M Zhang
Kay McKinney
Disha Shrivastava
Cosmin Paduraru
George Tucker
Doina Precup
Feryal Behbahani
Aleksandra Faust
Venue: ICLR 2025 Oral (2025)
Keywords: language models
reinforcement learning
Primary area: foundation or frontier models, including LLMs

Abstract: Self-correction is a highly desirable capability of large language models (LLMs), yet it has consistently been found to be largely ineffective in modern LLMs.  Current methods for training self-correction typically depend on either multiple models, a more advanced model, or additional forms of supervision. To address these shortcomings, we develop a multi-turn online reinforcement learning (RL) approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely self-generated data. To build SCoRe, we first show that variants of supervised fine-tuning (SFT) on offline model-generated correction traces are insufficient for instilling self-correction behavior. In particular, we observe that training via SFT either suffers from a distribution mismatch between the training data and the model's own responses or implicitly prefers only a certain mode of correction behavior that is often not effective at test time. SCoRe addresses these challenges by training under the model's own distribution of self-generated correction traces and using appropriate regularization to steer the learning process into learning a self-correction strategy that is effective at test time as opposed to simply fitting high-reward responses for a given prompt. This regularization prescribes running a first phase of RL on a base model to generate a policy initialization that is less susceptible to collapse  and then using a reward bonus to amplify self-correction during training.  When applied to Gemini 1.0 Pro and 1.5 Flash models, we find that SCoRe achieves state-of-the-art self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively on the MATH and HumanEval benchmarks.

Published as a conference paper at ICLR 2025
TRAINING LANGUAGE MODELS TO SELF-CORRECT VIA REINFORCEMENT LEARNING
Aviral Kumar" *+ Vincent Zhuang" *+ Rishabh Agarwal", Yi
JD Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani', Aleksandra Faust' Google DeepMind
ABSTRACT
Self-correction is a highly desirable capability of large language models (LLMs), yet it has consistently been found to he largely ineffective in modern L.Mss. Current methods for training self-correction typically depend on either multiple models, a more advanced model, additional forms of supervision. To address these short- comings, we develop multi-turn online reinforcement learning (RL) approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely relffgenerater data. To build SCoRe, we first show that variants of supervised fine-tuning (SFT) on offline model-generatee correction traces are often insufficient for instilling self-correction behavior. In particular, we observe that training via SFT falls prey to either a distribution mismatch between mistakes made by the data-collection policy and the model's own responses, or to behavior collapse, where learning implicitly prefers only a certain mode of correction behavior that is often not effective at self-correction challenges by training under the model's own distribution self- seff-generated correc- tion traces and using appropriate regularization to steer the learning process into learning a self-correction behavior that is effective at test time as opposed to fitting high-reward responses for given prompt. This regularization process includes an initial phase of multi-turn RL on base model to generate policy initialization that is less susceptible to collapse, followed by using a reward bonus amplify self-correction. With Gemini 1 1.0 Pro and 1 1.5 Flash models, we find that SCoRe achieves trate-of-the-aad self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively MATH and HvaanEval.
test problems. SCoRe addresses these
INTRODUCTION
Large language models (LLMs) are a useful tool for reasoning in scientific domains such as math and coding (Shao et al., 2024; Lozhkov et al., 2024; Team, 2024). An aspirational property of LLMs in such settings is their ability to implement intaa-ttrategies or algorithms that use test-time computation to generate improved responses. However, modern LLMs do not implement such strategies reliably. For instance, consider a problem that requires models to detect and revise (or "self-correct") their own responses in order eventually arrive the best possible final response. This self-correction capability has been shown to be severely lacking in current LLMs, especially in the absence of external input (also called intrinsic self-correction) (Huang al., 2023; Kamoi et al., 2024).
To make progress towards teaching LLMs to implement meta-strategies for challenging inputs, we study a special instance of training LLMs to perform self-correction fix their mistakes "on-the-fly". This should be possible: on many queries where current LLMs fail, they still possess the underlying "knowledge" needed to arrive at the correct response but are unable correctly elicit and draw inferences about their own knowledge when needed (Yang et al., 2024). For example, strong L.MSs can often successfully complete sub-part of math proof when prompted with the remainder, but may not be able to complete it from scratch. In similar vein, leveraging their previous responses should, in principle, enable LLMs to improve their subsequent ones. Despite this, self-correction has remained elusive, highlighting the need to go beyond existing training paradigms.
"qqual contribution, 'Randomly ordered via coin flip. 'Jointly supervised. Corresponding authors: /vin- centzhuang, rishabhagarwal, yisumtv] goo, aviralku andrew.cmu.edu

---

Published as a conference paper at ICLR 2025
Gemini 1.5 Flash on MATH
Scaling Inference Compute: MATH
70
Direct Generation
70%
65
Self- Correction (SC)
+4.44%
@if-Connsstenccykk
(SC Direct)
68%
Accuracy (96)
60
66%
55
-11.2%
+0.4%
+1.8%
64%
50
62%
Parallel Samples
45
Sequential (Self-Correct)
60%
40
2
2
2'
2
2
Base Model
STaR
SFT
SCoRe
Number of samples (K)
Figure 1: Left: SCoRe achieves state-of-the-at self-correction performance MATH; Right: SCoRe
inference-tine time scaling: spending samples on sequential self-correction becomes more effective than only on parallel direct samples (Section 6.2).
How can we imbue LLMs with self-correction abilities? Prior attempts for self-correcting LLMs either rely on prompt-engineering (Madaan al., 2023; Kim et al., 2023) or on fine-tuning models specifically for self-correction. While the former approaches often fail to perform meaningful intrinsic self-correction, fine-tuning approaches require running multiple models during inference, such as separate refinement model (Havrilla et al., 2024b; Welleck et al., 2023), or rely on "teacher" supervision to guide the process self-correction (Qu et al., 2024). With the use of separate models of teacher supervision, self-correction does not necessarily outperform parallel, independent attempts. We develop an approach that is effective at self-correction without these requirements. Our approach, Self-Correcti via Reinforcement Learning (SCoRe), trains only single model that can both produce a response to a problem and also correct errors without any oracle feedback.
To develop SCoRe, start by analyzing the shortcomings of SFT-based approaches (e.g.. STaR (Ze- likman et al., 2022)) and naïve RL that optimizes final response correctness for teaching self- correction. We find that such approaches fall prey to either: (1) distribution shift, where the trained model is able to correct errors made by the base model that generated the data, but these gains do not transfer to self-correction under the learned model's own mistakes: or (2) behavior collapse, where the learning progress simply learns to produce the best first-attempt response followed by superficial or no modifications the second attempt. To address these issues, SCoRe trains for self-correction directly via on-policy, multi-turn RL. To prevent behavior collapse, SCoRe employs two-stage training: in the first stage, it produces an initialization that is less susceptible to behavior collapse by training to correct second-attempt responses while constraining the first-turn distribution to be close to the base model; followed by training on both attempts to maximize reward in the second stage. Crucially, the second stage of multi-turn RL employs reward shaping term that rewards "progress" " towards self-correction as opposed to the correctness of the final response.
Our contribution is SCoRe, multi-turn RL approach for teaching LLMs how to correct their own mistakes. To the best of our knowledge, SCoRe is the first approach to attain significantly positive intrinsic ceff-correction: relative to base Gemini models, on self-correction for reasoning problems from MATH (Hendrycks et al., 2021) and an absolute 9.1 % gain coding problems from HumanEval (Chen et al., 2021). We additionally motivate the design of SCoRe by extensively studying the failure modes of SFT and standard RL approaches, which broadly indicate that reinforcement learning plays an essential role in self-trained self-correction.
method attains absolute 15.6% gain
2 RELATED WORK
Prior works study self-correction for LLMs under a variety of assumptions and problem settings. The most prominent settings include problems where external input tokens from an environment are available, such agentic tasks (Liu et al., 2023), code repair (Jain et al., 2024), and tool use (Chen et al., 2023). While self-correction with external feedback is possible with strong models (Pan et al., 2023), even they struggle in the substantially harder setting with external input (intrinsic self-correction) (Kamoi et al., 2024; Huang et al., 2023). Prior work that attempts to amplify intrinsic correction abilities is largely based prompting and fine-tuning.
Prompting for intrinsic self-correction. Recent work demonstrates that naively prompting LLMs for self-correction can degrade performance (Huang et al., 2023; Zheng al., 2024; Tyen et al., 2024;
2

---

Published as a conference paper at ICLR 2025
MATH Example I: Arithmetic Self-Correctinn
Problem. If n = 2 (snod 7). then find the remainder when (r 2) + 4) 6) divided by 7,
SCoRe I solution (incorrect). Since
(mod 7). write 7k for integer & Substituting, get (n+2)(n +44[nn+ +6) = (7k + 4)(7k + 6)(7k + 8) 4-6-8 (mod 7) = 192 (mod 7) (mod 7). Final Answer: The
answeris hope correct.
Self-correction instruction.
There might be error the solution above hecause of lack of understanding of the question. Please correct the crror, if any, and rewrite the solution.
(mad 7), we can write 7 some integer At Subatinating we get (n + 2) 4) + 6) (7k + 4)( + 6)(7A: 4 8) 4-6.8 (mod 7) = 4- (mod 7) 24 (mod 7) =3 (mod 7). Final Answer: The final answer is hope it is correct.
SCoRe turn 2 solution (correct). Since
MATH Example 2: Reasoning Self-Corecctin
be function that is defined for all values of a in [0, oo) such that f(x)) is a real number. How many distinct values cxist in the range of (ar)?
Problem. Let ((x) - L(- 2]
increnses, the value of (-g)" decreases. Since the floor function rounds down to the ncarest integer, the range of f(z) consists of all the integers less than equal [--) Therefore, the range of f(x) is (1, 1,, ..... }. and there are distinct values in the range. Final Answer: The final answer oa. hope correct. Self-correctonn instruction. There might be error the solution above because of lack of understandiag of the question. Please correct the error, if any, and rewrite the solution.
SCoRe turn 1 solution (incorrect).
SCoRe turn 2 solution (correct). As increases from tn the value of (- [-] oscillates hetween positive and negative values, but its absolute value decreases. The floor function rounds down the nearest integer, the range of f(x) consists the integers less than or equal to - Since the absolute value of (-8)² docresses as x increases, the range f(x) finite and consists of the integers: {1, 0, 1). Final Answer: The final answer is 1 hope it is correct.
Two example traces of self-correction. In the upper example, SCoRe is able to correct an arithmetic mistake it makes in turn 1. In the lower examplc, the model is able to correct a reasoning error.
Figure 2:
Qu et al., 2024). These results contradict prior work (Madaan et al., 2023; Shinn al., 2023; Kim et al., 2023) and largely stem from mismatched assumptions about the setting (Kamoi et al., 2024). For example, Shinn et al. (2023); Kim et al. (2023) use ground-truth answers during self-correction that may not generally be available; Madaan et al. (2023) use weak prompts for initial responses, thereby overestimating the total improvement possible. Therefore, there is no major work showing successful intrinsic self-correction via prompting alone. In the context of code self-repair, Olausson al. (2023) show that even when strong models are prompted with some form of partial feedback, e.g., test-cases but not the desired outcomes, they are unable to correct their mistakes.
Fine-tuning for intrinsic self-correction. Several works that go beyond prompting rely on fine- tuning with demonstrations of revisions, e.g. obtaining revisions directly from human annota- tors (Saunders et al., 2022) or stronger models (Ye et al., 2023; Qu et al., 2024). Our work aims to train for self- coffccoreetton entirely without the use of larger models or humans, when the learner itself asked to generate its own training data. Similar to these prior works, we assume access to a reward function for evaluating model-gende generated outputs (Welleck et al., 2023; Akyürek et al., 2023; Zhang et al., 2024). Perhaps the closest to us from this set is Qu et al. (2024), which utilizes an iterative STaR-like approach self-correction. While this work largely uses oracle teacher supervision, their preliminary results from training for self-correction only show minor improvements over five turns, consistent with the results see for STaR. We show that SCoRe attains substantially better results. Other approaches train separate models for performing correction (e.g.. GLoRE (Havrilla et al., 2024b), Self-Correction (Welleck et al., 2023), Akyürek et al. (2023); Paul et al. (2023). While such approaches can be convenient, they require system design for serving multiple models at deployment.
Multi-turn RL for LLMs. Prior work the intersection of LLMs and multi-turn RL builds machinery for optimizing rewards with value (Snell et al., 2022; Zhou et al., 2024; Farebrother al., 2024; Shani et al., 2024), policy-based (Xiong et al., 2024; Shao et al., 2024), and model- based (Hong et al., 2024) approaches. We do not focus on building machinery for RL (we use the approach of Ahmadian et al. (2024)). but rather train for self-correction as an RL problem.
3 PRELIMINARIES AND PROBLEM SETUP
Our goal is to develop an approach for training LLMs to improve their own predictions entirely on self-generated data. As discussed so far, we situate ourselves in the intrinsic self-correction setting (Huang al., 2023), where models attempt to correct their initial responses without any
3

---

Published as a conference paper at ICLR 2025
Table Self-correction performance after training on DsTaR and DEFt. We find that gap between the second and first attempts (A()),22) either negative or small. Both approaches erroneously modify a correct response incorrect, i.e., reflected in a high A (t1, 12) and low A (t1,
Method
Accuracy@t1
Accuracy
12
Atl, t2)
A
At1,, (2)
(t1,
(t1, (2)
Base model
52.6%
41.4%
-11.2%
4.6%
15.8%
STaR DsuaR
55.4%
41.2%
-1.2%
5.4%
19.6%
STaR DSEaR
53.6%
54.0%
0.4%
2.6%
2.2%
Pair-SFT DSFT
52.4%
54.2%
1.8%
5,4%
3.6%
Pair-SFT DŠFT
55.0%
55.0%
0%
0%
0%
external feedback. Concretely, given a dataset D = {(,,, y, ))= of problems æ; and responses y. we will train an LLM policy To (-(,, P1=)) that, given the problem æ. previous / model attempts F1 at the problem, and auxiliary instructions P1:1 (e.g., instruction to find a mistake and improve the response), solves the problem æ as correctly as possible. This formalism is akin to the multi-turn MDP in Qu et al. (2024). We also assume access to an oracle reward F(y. y y), such as an answer checker (Uesato et al., 2022), that evaluates the correctness of response y by comparing with the oracle response y Critically, we do not assume access to this oracle at test-time; instead, the model must deduce whether there was a mistake and correct if necessary, as often the case in e.g. mathematical reasoning problems. Unlike the setup of Qu et al. (2024). we also do not run majority voting for most of our main results. An example of our problem setting given in Figure 2.
We aim to find an LLM policy π (a)o)) mapping input tokens the correctness reward obtained from the verifier at the end of I + turns (1 = 1). Formally:
output tokens that maximizes
[+1 Σ
max K0
Ex.y"" -D.i
( (
(1)
Crucially, note that unlike standard SFT prevalent RI, fine-tuning workflows, which train the policy n to directly produce y (or any other y wih Fly, )² = 1), Equation trains и over multiple attempts simultaneonsly, where intermediate turns supervised indirectly maximize the sum.
Base RL fine-tuning approach we use. We use a REINFORCE policy gradient training approach with KL-divergence penalty against fixed model (Ahmadian et al., 2024), which is widely used in RL fine-tuning of LLMs, primarily in the setting of single-turn RLHE. Formally, these methods train the policy no (-||2)) x)) optimize the following, where Trof is reference policy.
max Fe [F y²) 31 D (T (= [=x))| (+1=;))]
(2)
Metrics. To measure self-correction performance (we consider = 2 in this paper), report and analyze the following metrics: (1) Accuracy@n @t1: the model's accuracy at the first attempt; (2) Accuracy @t:: the model's accuracy at the second attempt, (3) At1,, t2): the net improvement in model accuracy between the first and second attempts, which measures the efficacy of self-correction, (4) A 1- (t1, t2): the fraction of problems that are incorrect in the first attempt but become correct at the second attempt, which measures how many new problems can self-correction solve; and (5) c- (t1, t2): the fraction of problems that are correct in the first attempt but become incorrect at the second attempt, which measures how well the model understands what makes a response correct.
SFT ON SELF-GENERATED DATA IS INSUFFICIENT FOR SELF-CORRECTIR
natural approach for training self-correction to utilize some form of supervised fine-tuning on data collected from base model. Variants of this approach have been shown to scale well on single-turn reasoning problems (Singh et al., 2023; Zelikman et al., 2022). In this section, we assess the empirical efficacy of two such approaches for self-correction: STaR (Zelikman et al., 2022), and a version of Welleck et al. (2023) that trains only one model.
We ultimately find that although these methods improve self-coorection over the base model, they fail to achieve substantially positive self-correction (A(t1,12)) By probing these models, we observe two main failure modes: (1) collapse to non-correcting behavior, where the models learn produce
4

---

Published as a conference paper at ICLR 2025
madel
Train (Piha)
Density
Demaity
20
Eval
2
12
0.0
0.2
0.4
0.6
0.8
1,0
0.0
0.2
0.4
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Edit distance ratio
Edit distance
distance ratio
(b) STaR edit distance ratios.
SCoRe
Density
Demsity
Denaity
Eval
(Dger)
0
0.0
0.2
0.4
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
Edit distance
distance ratio
Edit distance ratio
(a) Histograms of edit distance ratios on MATH 500.
(c) Pair-SFT edit distance ratios.
Figure 3: Edit distance between first-attempt and second-attempt responses from finc-tuned models, our approach (SCoRe) and the base model. While training on self-generated error correction traces learns to not make major edits primarily, SFT learns make edits but still quite conservative.
good response on the first attempt and only make minor (or no) modifications in the second attempt, and (2) an inability of offline methods to be robust to distribution shift in the first-attempt responses.
Analysis setup: methods and dataset construction. We prompt Gemini 1.5 Flash to obtain large number of two-turn self-correction traces MATH (Hendrycks et al., 2021). The STaR approach filters these trajectories to retain only those that successfully revise incorrect responses and runs SFT the resulting dataset. Another approach is to base model data from above construct "synthetic" repair traces by pairing incorrect responses with correct ones (Welleck et al., 2023). We study variant of this method that call Pair-SFT, which does not train separate corrector model and does not augment this initial dataset with multi-turn traces. Formally, we denote the datasets for STaR and Pair-SFT as DYTAR and DSFT respectively. We run 3 iterations for STaR following the protocol in Singh et al. (2024), and only one iteration for Pair-SFT, following the protocol in Welleck al. (2023) and other standard workflows on SFT.
Main empirical findings. We present the self-correction results before and af-
0.10
Fixed validation
Self generated validation
ter fine-tuning DSTAR and DEFT in Ta-
Validation (11
0.08
Fixed train
0.3
ble We find that although At1, 12) is substantially higher for PFi relative to the base model, there is only little ben-
0.06
0.2
efit to self-correction (1.8% gain). This
0.1
gain of a similar order to findings from
0.02
Qu et al. (2024). By studying A i-c and A c-i find that SFT mainly reduces the number correct problems that are mistakenly changed incorrect in the second attempt, but does not significantly increase the fraction of incorrect first at- tempts that are corrected. This result is consistent with prior works on intrinsic self-correction that have found negligible or negative (tl, t2) values.
0.0
0.2
0.4 0.6
0.8
1.0
Epoch fraction
Figure 4: Self-correction performance on different sets of first-attempt responses: (a) "fixed": first response is sampled from the initial model, (b) selff-geneaate".. first response is gen- crated by the learner itself. Throughout training, the correction rate on fixed responses increases for both train and validation problems, but degrades substantially on self-generated responses. This indicates that training a fixed offline dataset of correction traces suffers from distribution shift.
We also find that unlike Pair-SFT, training on DSTAR does not reduce A-i indicating that the STaR policy does not have clear understanding of when to make modifications and when not to. Observing this, we also trained on an extended version of DŠTAR (and DTPT), which additionally contains tuples with both correct responses. We would expect the addition such ccorrect-t--corree data to prevent the model from erroneously revising correct response. As shown in Table 1. the inclusion this data helps STaR substantially but only results in 0.4% change in At11, t2). On the other hand, for SFT, inclusion of this data overly biases the model against changing its answer.
5

---

Published as a conference paper at ICLR 2025
Train Accuracy
Att1, 02)
0.4
Stage
Multi-tum baseline
Different answer
A21..12
Stage II
0.55
0.3
Stage Multi-turn base Ine
Tum Tur
0.6 Training fraction
0.4
0.8
1.0
0.2 0.4
Training fraction (aver stage I)
(a) Training accuracy curves. When training with stan- dard multi-tura RL, the responses at both the attempts become tightly coupled together, leading to poor cover- age for subsequent iterations and worse learning progress. Stage in SCoRe is explicitly designed to alleviate this and achieves much higher (t)1,22), leading to increased exploration and better final performance.
(b) Frequency in which the learner proposes different answer in the second turn. Without explicitly modifying the policy initialization as in SCoRe, the policy quickly learns to often not change its answer, leading poor exploration. Stage I in SCoRe prevents this issue, and learns non-collapsed hchavior Stage II.
Figure 5: Behavior collapse in standard multi-turn RL for training self-correction. These results indicate that some explicit approach to avoid collapse is necessary, i.e. Stage 1 in SCoRe.
Diving deeper: analyzing self-correction behavior. To further understand how these STaR and SFT models edit their responses, we measured their edit distance ratios, defined as the edit distance between the responses normalized by the total length of both the responses. As shown in Figure 3a, while the base model sometimes makes substantially large edits to the original response, models fine-tuned on DETAR and DSFT are overly conservative and often make no edits at all. This is akin to form of behavior collapse: training to maximize likelihoods on off-policy revision traces does not teach the desired correction "behavior", even though it improves first-attempt accuracy. Similar observations of LLMs ignoring nuanced behaviors (c.g., producing mistake in a response and then correcting it in subsequent steps) have been observed in Ye et al. (2024b).
We also compared the distributions of edit distance ratios on training and test-time self-correction traces in Figures 3b/3c. While STaR produces qualitatively similar edit distance ratios on both the train and validation sets, we still observe some discrepancies between the train and validation edit distance ratios for SFT, implying that Pair-SFT is not very effective at generalizing to new problems from the same distribution. We visualized this by plotting the self-correction performance of the SFT model on a fixed set of first attempts and self gelf-generated first attempts in Figure 4. We observe vastly different behaviors between static and self-generated first-attempt distributions: while the model is able to optimize training correction accuracy and also slightly improves on first attempts appearing in the validation set (distributed i.i.d. to the training distribution). its self-correction accuracy degrades. Hence, distribution shift is a significant challenge for offline methods such as PFi
Takeaways: Insufficiency of SFT
SFT methods suffer from two distinct failures when learning self ceff-correction: (1) distribution shift, and (2) behavior collapse. Training on on-policy data can fix (1), but not (2).
5 SCoRe: SELF-CORRECTION VIA MULTI-TURN REINFORCEMENT LEARNING
The above results highlight that effective approach for training LLMs to self-correct entirely via self- self-generated data must address both distribution shift and behavior collapse. Utilizing on-policy RL a natural way to address distribution shift, and our method will do so by extending Equation 2 to multiple turns under the hierarchical framework of Zhou et al. (2024). However, is behavior collapse an issue for standard multi-turn RI. (i.e., optimizing reward at the end of the second attempt)?
To answer this question, we run standard multi-turn RL training to optimize Equation only on (xz, 12) pairs, appearing in the second attempt. Since this objective maximizes the second-attempt performance given self-generated first attempts but without training the first attempt, we expect the self-correction (t)),2 of the model to increase. However, as shown in Figure 5, while the performance of each attempt improves with training, the difference Attl, 12) does not. In other words, standard multi-turn RL collapses to be overly biased against changing its response, resulting in no self-correction ability and similar behavior collapse as what with STaR.

---

Published as a conference paper at ICLR 2025
Stage I: Train an initialization for self correcoorotton
Stage Run Multi- turn RL with reward shaping
Constrain to base model
Maximize reward anempt bg)
Maximize reward
Maximize shaped reward second atterpp bt)
atemat Dy)
Foundation on model trained with SCoRe
Foundation model trained SCoRe
promp10
pronpee 1O
Figure 6: An overview of approach (SCoRe). SCoRe trains model in two stages: Stage I: instead of running SFT (which produces pathological amplification biases) to initialize RI. training, we train a good initialization that can produce high-reward responses in the second-attempt while mimicking the base model's s initial response at the first attempt. Stage II: jointly optimizing both attempts, where the latter uses shaped reward to incentivize the discovery of the self-correction strategy instead of the simple strategy of producing the best first response followed by making any minor edits to in the second attempt.
Why does RL still suffer from collapse? Note that there are two equally good solutions when optimizing a policy with RL on the training prompts: (i) learning to correct the first attempt, (ii) learning to produce the best first-attempt response, followed by no meaningful correction. Of course only the former strategy produces self-correction behavior to new problems, but RL on pre-trained LLM may not learn (i) over (ii), since both of these strategies can appear equally good on the training ser. Abstractly, learning the "meta" strategy of self-correction during training difficult unless the "direct" strategy that optimizes reward appears less viable on the training data. Conceptually, this is similar to the memorization challenge in meta-learning (Yin et al., 2019). which suggests that when provided with mutually exclusive tasks, meta-learning is likely to recover the supervised learning solution (without using context from the few shots) that directly predicts the output. Here, this is analogous to not self-correcting past attempts, directly producing answer.
Method overview. We saw that standard RL leads to a collapse to non-correcting behavior, which optimizes accuracy of both attempts, but does not incentivize seff-correction. Hence, key insight in SCoRe is that we must more explicitly encourage ceff-correction behavior, which we accomplish via two-stage approach. The first stage (Stage I) serves the role of initialization where train the model to decouple its behavior across the two attempts by attempting to optimize second-attempt accuracy while explicitly constraining the distribution of first attempts to the base model. From here, Stage II then jointly optimizes the reward of both attempts. To ensure that Stage II does not collapse the "direct" solution, we bias the reward reinforce self-correction progress.
5.1 STAGE I: TRAINING AN INITIALIZATION TO DECOUPLES ATTEMPTS
The goal of Stage I of SCoRe is to obtain an initialization by improving the coverage of base model's second attempts given the first attempt, that subsequent training with on-policy multi-turn RI. is less prone to hehavior collapse. While this would typically be done via SFT, our results in Section show that SFT itself suffers from collapse. Therefore, attempts. To do so, we explicitly fine-tune the base model to produce high-reward responses at the second attempt, while forcing the model not change its first attempt by constraining it to be close to the base model using KL-divergence. While this may appear sub-optimal as it constrains the first attempt to the base model but we find this stage is critical in reducing the base model's bias towards collapsing the first and second-attempt distributions, thus avoiding behavior collapse when actual multi-turn RL is run. Formally, the objective is:
RI. in this stage to decouple the two
max Ee, an -mo--la) wa mal [x, p. F(u2: y²) - DRD (Te(-I1æ;;)) [ 8
(3)
where Bg is hyperparameter designed enforce a strict KL penalty only the first attempt to avoid shifting of the first-turn responses (denoted by the term in blue). As we use RLOO (Equation 2) for training the policy, there is still a default KL-divergence penalty, but with a much smaller weight and is omitted from Eq. 3 for clarity. We show that unlike standard multi-turn RL, Stage is effective at decoupling the two responses (Figure 5b) and leads to better Stage II performance.
7

---

Published as a conference paper at ICLR 2025
5.2 STAGE II: MULTI-TURN RL TO OPTIMIZE BOTH ATTEMPTS
The second stage of SCoRe is initialized from Stage and now jointly optimizes the performance of both attempts. Concretely, Stage II trains Tp (-1-)) using objective (Eq. applied to Eq. 4):
2
max Exi mee (=..p((
Σ
Fly. v") 3, Dxz (=
(4)
8
i=1
where m, i E (1,2) corresponds to the set of input tokens passed context the model.
Reward shaping to prevent behavior collapse. Optimizing Equation 4 via multi-turn RL can still learn to couple responses. This is because still attempt to maximize rewards for both attempts in Equation 4. To prevent the learning process from collapsing to a non self-correction policy in Stage II, we found it crucial to bias the RL problem towards learning ceff-correction behavior. We implement this via reward shaping: by rewarding transitions that make "progress" towards learning the desired self-correction behavior. Concretely, given an two-turn rollout sampled from the policy { D1, F(11, 7/ ") 22 D2. F(y2, y )}, we modify the reward F(12. ") in Equation 4, at the F(y1, )), where is positive constant multiplier, ideally larger than 1.0. Adding this bonus to the second attempt measures a notion of progress by only emphasizing transitions that flip the correctness of the response and assigns heavy negative penalty to transitions that change a correct response to incorrect the second attempt. Thus, the addition of this bonus should regularize the training process from collapsing on to the "direct" solution that also appears optimal on the training set but does not learn self-correction.
second attempt with bonus 6( /1/. := (f(y2,
5.3 PUTTING IT TOGETHER AND IMPLEMENTATION DETAILS
Our approach is illustrated in Figures 6 & 11. We detail all hyperparameters in Appendix B. In practice, one can also use adaptive Bg that attempts balance the magnitudes of the first-attempt KI. regularization and the second-attempt policy objective. In some of our experiments, we also choose to amplify the coverage of states used for on-policy RI. incorporating first-attempt solutions obtained by repeatedly sampling the base model as offfine prompts in RL. We find that incorporating this data, especially in Stage II where the first-turn policy may have drifted further from that of the base model can have substantial benefits especially when attempting to learn from limited data.
Takeaways and Implications
The core insight behind SCoRe is that we must make it more attractive to learn a more nuanced algorithmic strategy (i.e.. self-correction) instead of collapsing to degenerate behavior mode. To avoid distribution shift, this must be done on self- generated online data.
EXPERIMENTAL EVALUATION
The goal of our experiments is to demonstrate the efficacy and justify the design of SCoRe in training LLMs how to self-correct by only training on their own data. To this end, we perform a comparative evaluation of SCoRe against prior methods that also self-generated data to train for self-correction, and run several ablation studies on two representative reasoning tasks where error correction is crucial.
Tasks. We mainly focus on reasoning problems in math and coding: (a) math problem solving on MATH (Hendrycks et al., 2021), and (b) code generation on MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021). 1). We use the following train-test splits our experiments: (1) MATH: following Lightman et al. (2023), we augment the MATH training set with 4500 problems from the test set, and report results on the remaining 500 problems (MATH500); and (2) Code generation: we train on MBPP and report results on HumanEval, which does not expose test cases to the model. For all tasks, we use binary rewards during training, indicating whether the model's answer matches the ground truth one (for MATH) passes all test cases (for coding).
Evaluation protocol and metries. We report the self-correction accuracy on a number tasks with two sequential attempts the problem, i.e., round of self-correction. For code generation, following the evaluation protocol of Ni et al. (2024), we also report results on MBPP-R -R an offline repair task that requires correcting incorrect first-attempt programs generated from PaLM 2.
Models. For coding problems, we fine-tune Gemini 1.0 Pro and for MATH, we fine-tune Gemini 1.5 Flash, and include experiments Gemma v2 models in Appendix A.I. For all evaluations,
8

---

Published as a conference paper at ICLR 2025
Table 2: Performance of SCoRe on MATH. SCoRe not only attains higher accuracy at both attempts, but also provides the most positive self-correction performance Att,, t2).
Approach
Acc.@t1
12c.@12
A(t,, 12)
A 1T" (2)
S-T (t1, 12)
Base model
52.6%
41.4%
-11.2%
4.6%
15.8%
Self-Refine (Madaan et al., 20233
52.8%
51.8%
-1.0%
3.2%
4.2%
STaR w/ DSun (Zelikman al., 2022)
53.6%
54.0%
0.4%
2.6%
2.2%
Pair-SFT w/ DEFT (Welleck al., 2023)
52.4%
54.2%
1.8%
5.4%
3.6%
SCoRe (Ours)
60.0%
64.4%
4.4%
5.8%
1.4%
Table 3: Performance of SCoRe on HumanEval. SCoRe attains the highest self-correction performance (Accuracy@12, A(tl, t2)), and also outperforms other methods at offline correction (MBPP-R).
Method
MBPP-R
Acc.@t1
Acc.
At1, 12)
A (L1, 12)
A
(t1, 12)
Base model
47.3%
53.7%
56.7%
3.0%
7.9%
4.9%
Self-Refine Pair-SFT
30.7%
53.7%
52.5%
-1.2%
9.8%
11.0%
59.8%
56.1%
54.3%
-1.8%
4.3%
6.1%
SCoRe (Ours)
60.6%
52.4%
64.6%
12.2%
15.2%
3.0%
use greedy decoding (i.e. temperature 0), except for inference-compute scaling in Section 6.2 where we set temperature to be 0.7. For all training methods, we attempted to use a fixed budget of model samples and gradient updates, and do not vary learning rate and batch size between runs. For all RL runs, we selected checkpoints with the highest training reward.
Evaluation prompts. We use zero-shot CoT prompting for evaluation on MATH, zero-shot prompt- ing for evaluation on HumanEval, and the canonical three-shot prompt for first-attempt training samples on MBPP (Austin al., 202 l). At the second attempt, we utilize an instruction that does not reveal the correctness of the previous answer, but asks the model to attempt to deduce whether a mistake exists in its first attempt response, and if so, potentially rewrite its response. Our full prompts and self ceff-correction instructions can be found in Appendix C.
Baselines comparisons. We compare SCoRe to relevant prior approaches based on prompting or those that fine-tune only a single model for both solving the task and for revising responses, and only use self-ggnerated data. Specifically, we compare to Self-Refine (Madaan et al., 2023), representative prompting-base based approach to elicit self-correction behaviors from a model, akin to Reflexion (Shinn et al., 2023). Among the fine-tuning based approaches, we compare to Pair- SFT based on the approach from Welleck et al. (2023), and multi-turn STaR (Zelikman et al., 2022; Singh et al., 2023) that fine-tune the model by maximizing log-liieldood respectively on synthetically paired repair traces (Pair-SFT) and successful repair traces (STaR).
6.1 BENCHMARK RESULTS
MATH. Our results are shown in Table 2, as well as in Figure 1. SCoRe exhibits substantially stronger performance on both direct and self-correction accuracies relative baselines. Notably, the intrinsic self coffecorrection gain At1, t2) of 4.4% is the first significantly positive delta, despite having fewer incorrect problems correct by virtue of its higher Accuracy @ tl. Relative to the base model Gemini 1.5 Flash, SCoRe improves (t1, t2) by 15.6%, and Accuracy @ 12 by 23.0%, and over the next best prior approach, Pair-SFT, by 10.2% and 2.6% respectively. By observing the frequency of problems that change from incorrect in the first attempt to correct in the second attempt and vice versa, we see that SCoRe both improves the rate at which it fixes incorrect answers (14.5% compared to 9.5% for base) and reduces the proportion of correct answers it changes (15.8% to 1.4%)..
Code generation. Our results for the code generation task are shown in Table 3. Generally, we find that SCoRe achieves both improved self-correction and offline repair performance. For MBPP-R (Ni et al., 2024), we find that SCoRe improves the base model from 47.3% to 60.6%, which is comparable to the gap between GPT-3.5 (43%) and GPT-4 (63.2%) Despite only training on MBPP, that SCoRe is especially effective at generalizing to HumanEval, achieving a 12.2 % intrinsic self- correction delta, or 9% higher than the base model. By contrast, Pair-SFT works nearly as well on the static repair task MBPP-R, but actually degrades the base model when evaluated in the self-correction setting, thus underscoring the importances of on- on-policy sampling for self-correction.
find

---

Published as a conference paper at ICLR 2025
Table 4: Ablation studies to understand the impact of various components SCoRe. Observe that while single-turn training is effective at optimizing the first-attempt accuracy of the model, leads to degradation in the second attempt. The performance improvements without Stage I or without reward shaping in SCoRe are small when measured by the difference in accuracy over the two attempts. Utilizing STaR generally leads to worse performance even when it is run from effective Stage checkpoint.
Method
Accuracy Accuracy @12
Atl, 12)
SCoRe (Ours)
60.0%
64.4%
4.4%
w/o multi-turn training
61.8%
59.4%
-2.4%
w/o Stage
59.2%
61.4%
2.2%
w/o reward shaping
60.0%
62.6%
2.6%
w/ STaR instead of REINFORCE Stage II
56.2%
58.4%
2.2%
w/o online turn 1 samples
60.4%
60.6%
0.2%
6.2 INTERENCE-COMPUTE SCALING WITH SELF-CORRECTION
Next, we investigate if SCoRe can be used in conjunction with inference-time compute scaling strategies. To do so, we evaluate self-consistency decoding ( (Wang et al., 2022), where we sample diverse set of solutions, and then select the most consistent answer among these solutions. Typically, the default strategy is to sample all solutions in parallel to perform majority voting. However, show in Figure (right) that instead of sampling 2K solutions in parallel, is more compute-efficiet to sample K solutions in parallel, then perform round of self-correction
each solution. With 32 solution budget per problem, parallel sampling shows a 7.4% accuracy gain, while combining with sequential sampling using self-correction yields a 10.5% improvement.
6.3 ABLATION STUDIES: UNDERSTANDING THE IMPACT OF SCoRe COMPONENTS
Finally, we also present a number of ablation studies to understand the importance of various components in SCoRe. We perform these ablations the MATH dataset. Concretely, we aim to answer the following questions: (1) the importance of multi-turn training: Can RL trained to maximize single-turn performance achieve better accuracy @tl accuracy @12?; (2) the importance of multi-stage training: How essential is Stage to SCoRe? In other words, why not run Stage II directly?: (3) the impact of reward shaping. How would removing the reward shaping terms affect performance of SCoRe in Stage II, assuming Stage was done identically?? ? (4) the importance of on-policy RL: What if we replaced REINFÖRCE in Stage II with STaR?.
The results of all of these ablation experiments are shown in Table 4. As expected. single-turn training improves turn 1 performance, but has negative At11 12). As shown in Figure 5, Stage I is critical to SCoRe; without it, the model achieves 2% lower (t11, 12) and 3% lower accuracy @ 12. Similarly, we find that removing reward shaping also hurts performance, indicating that the RL objectives in both stages play significant role in teaching the self-correction behavior. We also find that replacing REINFORCE with STaR Stage II results significantly lower absolute performance with no visible improvements in seff-improvement performance, which contrasts with the findings in Havrilla et al. (2024a) that STaR and on-policy RL have similar convergence rates. This suggests that leveraging on on-policy samples is especially critical in the self-correction setting, which presents multi-turn problem that admits potentially spurious solutions. We also present additional ablations with multiple turns (for both training and evaluation) and reward function design in Appendix A.
7 DISCUSSION, LIMITATIONS, AND CONCLUSION
We proposed SCoRe and showed through extensive evaluations that it is one of the first methods to attain significantly positive intrinsic self-correction performance. We also rigorously analyzed the behavior of various SFT approaches and identified failure modes in which the model learns non-correcting strategy (c.g. learning to make no edits; behavior collapse) or falls prey to distribution shift. SCoRe trains self-correcting strategy by utilizing two-stage design and reward shaping, both of which help preventing behavior collapse into not learning effective self-corrective behavior. SCoRe has limitations that also provide avenues for future work. Unifying Stages I and II would also be interesting, since it would alleviate the limitation of running multiple runs. Incorporating a critique in between the self-correction turns would also boost performance. Finally, our results suggest that learning meta-strategies (c.g., self-correction) requires going beyond standard fine-tuning (Section 4), and incorporating regularization (e.g., progress reward).
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We aimed to include both the high-level and low-level details of our method, including all hyper- parameters that we use in Appendix B. Our training and evaluations benchmarks: MATH (Hendrycks et al., 2021), 1, MBPP (Austin et al., 2021), and HumanEval (Chen al., 2021), with all specific prompts used in Appendix C. We have also added results with the open Gemma 2 model in Appendix A.I as well to facilitate reproducibility. Our RL algorithms and infrastructure simply extends the methodology of Ahmadian et al. (2024) to multi-turn settings with relatively simple modifications. We believe that these details will enable practitioners to implement these ideas on open-source models. While we cannot release our fine-tuned models, we hope our detailed descriptions should help the research community replicate our findings.
performed on open-source
Open-source replications and new research work post the ICLR submission. After the release of the DeepSeek-RI technical report (DeepSeek-AI et al., 2025) in January 2025, there has been substantial interest in the community towards building open-source replications of RL training of long ofthhuh models. A number of these RL training frameworks (Luo et al., 2025; Liu et al., 2025; Zeng et al., 2025) can likely also be repurposed to study self-correction abilities that we focus in this work. In fact, despite the substantially different problem setting between long chain-of-thought RI. and our work, some of the abstract ideas in our work such as the idea of multi-stage training also appears in Luo al. (2025), and reward shaping (or dense rewards) has also been discussed in Setlur et al. (2025). In addition, Setlur et al. (2025) show theoretically that RL training is much more effective using more test-time compute (which in our setting translates to using more attempts at self-correction) than SFT or STaR. Their theory further corroborates the empirical claims we make in Section regarding the ineffectiveness of SFT and STaR.
ACKNOWLEDGEMENTS
The authors would like thank Satinder Baveja, Kalesha Bullard, Gheorghe Comanici, Claire Cui, Valentin Dalibard, Angelos Filos, Yang Gao, Zoubin Ghahramani, Izzeddin Gur, Raia Hadsell, Clara Huiyi Hu, Melvin Johnson, Mina Khan, Balaji Lakshminarayanan, Yiran Mao, Hussain Masoom, Junhyuk Oh, Jordi Orbay. David Silver, and Yury Sulsky for helpful discussions, feedback, and sponsorship. We thank Amrith Setlur, Yuxiao Qu, Charlie Snell, Tianhe Yu, and Xinyang (Young) Geng for helpful discussions and feedback on earlier version of the paper.
AUTHOR CONTRIBUTIONS
AK and VZ led the paper, with substantial technical contributions from RA and YS. VZ led the experimentation in the final paper with AK, with support from RA and YS. AK and RA conceived the initial idea with advice and discussions from DS, FB, AF, JDC, AS, and GT. JDC, YS, AS, RA, and AK iterated on the methodology. The development of the final method was done by AK and VZ. with inputs from RA and FB. V7. led the infrastructure development, while RA, YS, CP, SI, KB, DS, and LMZ contributed to the infrastructure. AK, RA, FB, AF, DP, GT advised on the overall direction. AK and VZ wrote the manuscript, with input from all co-authors. KM provided program management. FB, and AF co-supervised the project.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint 240. 2024.
Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. RI4f: Generating natural language feedback with reinforcement learning for repairing model outputs. arXiv preprint 2rXiv:2305.08844 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 07732, 2021.
Mark Chen. Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 1rXiv:2107.03374 2021.
11

---

Published as a conference paper at ICLR 2025
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:23004055228 2023.
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue. Bingxuan Wang. Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen. Guowei Li. H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang. Lei Xu, Leyi Xia, Mingchuan Zhang. Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou. Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. 7. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepscek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
URL wwthhhtts//// /wtp/v/aavvvv/v///////e../a.aa org/ abs/2501. 12948.
Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. Mathodyssey: Benchmarking mathe- matical problem-solving skills in large language models using odyssey math data. arXiv preprint arXiv:2406. 18321, 2024.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint MXXi::2403.03955 2024.
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability, Advances in neural information processing systems, 34:25502-25515 2021.
Alex Havrilla, Yuqing Du Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprini arXiv:2403.0444 2024a.
Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Railneau. Glore: When, where, and how improve lim reasoning via global and local refinements. arXiv preprint arXiv:2402. 10963, 2024b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Jiwoo Hong, Noah Lee, and James Thome. Reference- free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07660 2024.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023.
12

---

Published as a conference paper at ICLR 2025
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint vrXiv:2403.07974 2024.
Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can Ilms actually correct their own mistakes? a critical survey of self-correction of llms. arXiv preprint arXi::2406.01297, 2024.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint arXhv:2303.1799/ 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.200550 2023.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms agents. arXiv preprint S3Xi::2308.03688, 2023.
Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. There may not be aha moment r1 zereooel training - pilot study. https:/ /oat 1mm. nww.mm..oiiooiioooooooooooo site/ oat-zero, 2025. Notion Blog.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint 241V:2240..19133, 2024.
Michael Luo, Sijun Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol-preview with a 1.5b model by scaling rl.
hota::/ppettttttattooooooooooooo not.ooi.o...ntttt...... site/ radio-radio--o-- notion
2025. Notion Blog.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinmment with self-feedback. arXiv preprint arXiv:2303.17653 2023.
Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. Next: Teaching large language models to reason about code execution. arXiv preprint Aix::22404. 0..4404.46600 14662, 2024.
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang. Jianfeng Gao, and Armando Solar-Lezama. Is self-repair a silver bullet for code generation? In The Twelfth International Conference on Learning Representations, 2023.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint trXiv:3308.03188 2023.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.0/1 1.000001944, 2023.
Yuxiao Qu Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how self-improve. arXiv preprint arXiv:2407.1822/9 2024.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint 2008002206.05802, 2022.
Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or rl is suboptimal. arXiv preprint arXiv:2502. 12/18, 2025.
13

---

Published as a conference paper at ICLR 2025
Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, et al. Multi-turn reinforcement learning from preference human feedback. arXiv preprint arXivv2405.
2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 240.:32402003300,, 2024.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303. 1366, 2023.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil. Peter Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.0658 2023.
Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar. Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem-solving with language models, 2024.
Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning, arXiv preprint arXiv:2206.1 11877, 2022.
Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402. 19450, 2024.
CodeGemma Team. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406. 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at a practical size. arXiv e-prints, pp. arXiv-2408 2024.
Gladys Tyen, Hassan Mansoor, Victor Cärbune, Yuanzhu Peter Chen, and Tony Mak. Llms cannot find reasoning errors. but can correct them given the error location. Findings of the Association for Computational Linguistics ACL 2024, pp. 10994-13908.. 2024.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14225. 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXivv2203.111777 2022.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference Learning Representations, 2023. URL htttts://oppnreeeee ew ww.net//.eeeeeee..e.e forum?
iddhhH3JJe2DDa.
Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bital Piot, Mohammad Saleh, et al. Building math agents with multi-turn iterative preference learning, arXiv preprint arXiv:2409.0233 2024.
Sohce Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402. 16837, 2024.
14

---

Published as a conference paper at ICLR 2025
Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. Selfee: Iterative self-revising Ilm empowered by self-feedback generation. Blog post, 2023.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint arXiv:2407.20311 2024a.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics language models: Part 2.2, how to learn from mistakes on grade-school math problems, 2024b.
org/
abs/2408 16293.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-lcarning without memorization. arXiv preprint arXiv:1912.038200 2019.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning, Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https: /ttw/hhuutttnnn not onn. site/ / mmleerrresson, 2025. Notion Blog.
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self-correct rea- soning. arXiv preprint 1111v:2404.17140, 2024.
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V Le, Ed H Chi, et al. Natural plan: Benchmarking Ilms on natural language planning. arXiv preprint arXiv:2406.0 04520, 2024.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar, Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402. 19446, 2024.
15

---

Published as a conference paper at ICLR 2025
Appendices
A ADDITIONAL. EXPERIMENTS
A.1 GEMMA EXPERIMENTS
Table 5: Evaluation on MATH.
Approach
Acc. @tl
Acc. @12
Acc. @t
Base model
16.8%
16.8%
17.0%
SCoRe Stage (a))
17.6%
20.0%
19.8%
SCoRe Stage 1(b)
16.6%
18.4%
23.2%
SCoRe Stage 2
23.0%
24.0%
24.0%
Table 6: Evaluation on Functional MATH.
Approach
Ace. @t1
Acc. @ 12
Acc. @1
Base model
21.4%
20.7%
20.9%
SCoRe Stage (a)
17.5%
20.3%
20.9%
SCoRe Stage 1(b)
17.7%
20.4%
24.8%
SCoRe Stage 2
23.4%
25.7%
25.6%
Table 7: Evaluation on MathOdyssey.
Approach
Acc.@t1
Acc. @ t2
Acc. @t
Base model
4.1% 1%
3.9%
3.6%
SCoRe Stage 1(a)
3.1%
3.1%
3.4%
SCoRe Stage 1(b)
2.3%
2.8%
4.1%
SCoRe Stage 2
3.9%
5.2%
5.7%
We conduct additional evaluations SCoRe on the open-source 2B Gemma 2 model (Team et al., 2024). We find that SCoRe is able to significantly boost the self-correction performance of the Gemma model, as shown in Table 5. All of the above tables evaluate performance SCoRe on Gemma 2 models including results disti--oist evaluation benchmarks.
Multi-turn training. We consider an extension of SCoRe to multiple turns as follows. In the three-turn section, break Stage into two sub-stages, say Stage 1(a) and Stage 1(b), with Stage remaining unchanged. In Stage I(a), the model is trained to maximize reward at the second attempt while keeping the first attempt close the base model. Stage 1(b) repeats this process but for maximizing reward at the third attempt, while keeping the first two attempts close to the model obtained from Stage ((a). Abstractly, with more than two attempts possible, Stage iteratively optimizes each attempt to maximize reward while keeping previous attempts constrained to the base model. This way we are able avoid collapse of each stage and address distribution shifts over multiple attempts. Stage 2 then proceeds as usual, optimizing the reward across all attempts and applying reward bonuses to incentivize the difference between rewards at given attempt and the immediately previous attempt.
In Tables 5,6, 7, we show direct comparison SCoRe Stage 1(a) and Stage 1(b). On benchmarks, Stage 1(a) boosts the self-correction performance of turns and 2, but does not lead to large improvements at turn 3. By contrast, Stage ((b) is able to further improve the performance of turn 3, suggesting that our method can be generalized to beyond two turns.
Heldout datasets. We additionally benchmark the performance of SCoRe on Functional Math and MathOdyssey, two variants of mathematical problem-sol solving datasets (Srivastava et al., 2024; Fang et al., 2024). We find that the self correctrot improvements generalize to these datasets, including the highly-difficult benchmark of MathOdyssey (Tables 6,7).
16

---

Published as a conference paper at ICLR 2025
A.2
PERFORMANCE OF SCoRe BY PROBLEM DIFFICULTY
0.07
0.06
Incorrect=CCoreett Concect>iIncorect
Proportion
0.05
0.04
0.03
0.02
0.01
0.00
Level 1
Level
Level
Level 4
Level 5
Difficulty
Figure 7: Performance of SCoRe problem difficulty.
We additionally analyze the performance of SCoRe the MATH 500 benchmark broken down by problem difficulty, with results shown in Figure 7. We find that SCoRe is able to achieve mistake- free self-correction the casiest levels (Levels 1 and 2), and achieves relatively similar positive self-correction rate on both medium and difficult questions (Levels 3-5).
A.3
EVALUATION ON MULTIPLE ATTEMPTS
65
Base model
60
Pair-SFT
Accuracy
SCoRe
50
45
2
6
8 10
Attempt
Figure 8: Performance of the base model, Pair-SFT, and SCoRe attempts MATH.
We investigate the performance of various models when asked to iteratively self-correct multiple attempts, despite only being trained over two attempts (or not at all, in the case of the base model). As shown in Figure 8, find that the performance of the base Gemini 1.5 Flash model is quite noisy, but never surpasses that of the first attempt. Similarly, Pair-SFT does not improve past the second attempt. By contrast, the performance of SCoRe increases slightly past two turns, although it does plateau likely because the distribution over responses shifts quickly as more revision attempts are performed We leave improving the scaling properties of self self-correction, form of meta-learning, to future work.
A.4 REWARD FUNCTION DESIGN
We clarify the reward function used in SCoRe in Figure 9.
In all of our experiments, used only the instantaneous reward in policy gradient objective, which equivalent to returns with discount factor y = 0. We additionally investigated whether leveraging 0, in conjunction with reward shaping, can elicit self-correction paper. As presented in Figure 10. we find that with 7 0.8 and a 1.0, multi-turn RL still suffers from the same non-correcting behavior collapse the standard multi-tur RI approach.
17

---

Published as a conference paper at ICLR 2025
Stage an initialization self-correction
Stage
reward shaping
Constrain to base model
Reward for attempt applied
Maximize reward for only first attempt tor
Shaped reward only applied here weoelaeeee
Bhit atenpe
Foundation model trared SCoRe
propp
Figure 9: A simple Illustration showing reward values used to multiply different sub-sequences of tokens SCoRe. Note that the first attempt always multiplied by own correctness reward in Stage 2 and reward shaping is only applied to the second attempt. In Stage the first attempt response is constrained to that the base model, while second attempt is trained optimize own correctness reward.
Train Accuracy
y= 0.0
Tum
0.60
0.50
0.0
0.2
0.4
0.6
0.8
1.0
Training fraction
Figure 10: Impact discount factor ry on standard multi-tum RL training.
A.5 WIIY CAN SELF-CORRECTION HELP CONCEPTUALLY?
One conceptual intuition for why self-correction should improve performance over simply maximizing first-attempt performance is that model performance should increase as is able leverage more tokens (analogously to how LLM reasoning performance increases with higher depths (Ye et al., 2024a), i.e., self-correction is able benefit from test-time token budgets.
Alternatively, from RL literature, one could theoretically characterize self- self-correction policies under the notion of adaptive policies. These adaptive policies condition action predictions not only on the current state but also on past attempts previous episodes. It is known in the RL literature that such adaptive policies especially excel in generalization settings. For example, the benefits of adaptive policies are studied by Ghosh et al. (2021). The sequential classification setting in this paper is conceptually similar to self-correction (though not the same).
B ADDITIONAL EXPERIMENT DETAILS
Table 8: Hyperparameters for SCoRe on MATH (left) and MBPP (right)
Hyperparameter
Value
Hyperparameter
Value
Base model
Gemini 1.5 Flash
Base model
Gemini 1.0 Pro
Optimizer
Adam
Optimizer
Adam
Learning rate
5e-6
Learning rate
lc-5
Training steps
3000
Training steps
1500
Batch size
512
Batch size
128
Sampling temperature
1.0
Sampling temperature
1.0
10
10
B1
0.01
B1
0.01
82
0.1
B2
0.25
18

---

Published as a conference paper at ICLR 2025
Base model
Online Multi-Turn RL
SFT
Behavior collapse and distribution shift
Behavior collapse optimized naively
Stage Train the model to decouple attempts
Avoid behavior collapse by keeping first attempt close to base model and improving second attempt
Stage 2: Train the model to optimize reward
Optimize reward of each tum. Avoid behavior collapse via reward-shaping.
Figure 11: flowchart explains our chain logic guiding the inclusion of component SCoRe.
include the hyperparameters used for training SCoRe in Table 8.
C
PROMPTS
MATH Zero-shot Prompt
You are math expert. When you respond, respond only with the Solution of the final Problem, thinking step by step. At the end of the Solution, when you give your final answer, write it in the form "Final Answer: The final answer is SanswerS. I hope it is correct."
MATH Self-Correction Instruction
There might be enror in the solution above because of lack of understanding of the question. Please correct the error, any, and rewrite the solution. Only output the final solution! At the end the Solution, when you give your final answer, write it in the form "Final Answer: The final answer Sanswer$. hope is correct."
MBPP 3-shot Prompt
You are an expert Python programmer, and here is your task: Write a function to find the similar elements from the given two tuple lists. Your code should pass these tests:
assert similar clements((3, 4, 6), 7, 4, 10)) == 5)
assert similar elements((1, 3, 4),(5, 4, 7)) (3.4)
assert similar_clementt((11, 12, 14, 13), 15, 14, 13)) = (13, 14)
[BEGIN]
def similar elements(test _tppl Lest____ug [est]] res = tuple(set(estt_upp & (et(ttstttpp))) return (res)
[DONE]
You expert Python programmer, and here is your task: Write python function to identify non prime numbers. Your code should pass these tests:
19

---

Published as a conference paper at ICLR 2025
assert is (s_not_prime(()) False
assert ps_nnt_primec( (0) = Truc
assert is 1s_nno_prrmm((55 == True
[BEGIN]
import math def deriisnot pot_prime(n):
result False
for in rangee2,in((math.qrron))) I): if == 0:
result = True
return result
[DONE]
You are expert Python programmer, and here is your task: Write function to find the largest integers from given list of numbers using heap queue algorithm. Your code should pass these tests:
assert hhapheaaaeeeeeeeee Sarp__qeee_laar largesst [25, 35, 22, 85, 14, 65, 75, 22, (8],3)==(85, 75, 65)
assert heap_auapppe_ppeee areap_queue__eeeee qquue_largeste [25, 35, 22, 85, 14, 65, 75, 22, (8),2)==185, 755
assert heap qeeee queue_largest(( [25, 35, 22, 85, 14, 65, 75, 22, (8),5)-(85, 75, 65, 58, 35]
[BEGIN]
import heapq as hq def lecap_quce__lrgee laggesstnstttms largest alarggsss,,, nums) return largest nums
[DONE]
MBPP/HumanEvll Self- Ceff-Coreection Instruction
There might be error in code above because of lack understanding of question. Please correct the error, if any, and rewrite the solution. Only output the final correct Python program!
D QUALITATIVE ANALYSIS OF SCoRe
We also perform qualitative investigation into how SCoRe addresses the self-repair shortcomings of base LLMs, and provide several examples in Appendix E. We find that SCoRe is able to refine its own responses in variety of manners rewriting the entire solution when necessary, or reproducing the correct parts of the solution, while revising the incorrect ones. For the latter, we interestingly find that SCoRe is especially adept revising its computational mistakes, and even demonstrates a bias towards showing more steps in certain computations and manipulations in order to increase its probability producing a correct answer. We additionally observe that the model learns to occasionally self-correct within turn, e.g. MATH example 4.
E EXAMPLES OF SCoRe SELF-CORRECTION
HumanEval Example
Problem:
def cycpattem__hhee b):
"""You are given words. You need to return True if the second word or any of its rotations is substring in the first word
cyepatter__check" check("abed") "abd") => False
cyepattern_ Serpatter__chcc((1111 "iaassup"."psuss cyepattern_ hat_echhecehlee check("whassupp
=> True
> False
cyepattern_ "ecaatten_ehec((" """"aaa""""aal "haaa") => True
cyepattern_dee
"eppatter__chcc(""" check("efef" "ceff") False
20

---

Published as a conference paper at ICLR 2025
eyepattern Serpattem_checc("hiiieee check("himens"", """iimee""""""""""""" => True
Turn 1 solution (incorrect):
cyepattem_ (aaarmaehhhhaaaee b):
"""You are given words. You need to return True if the second word or any of its rotations a substring
in the first word
cyepattern check("abed") "aadd"""""""") => False
cyepattern srppattem_chec(("" checcc"helloo" "ell") True
grepateem_chedleee "aeeck("wassspp" "paasssup"""sss""""""""""""""""" => False
eyepattem_cheee cettrrm_hheckhhle (aacck"""bb",, "baa") => True
eyepattern eycpattern theck("himens"" "simen") => True
sycpatter__checc("eeee check("efef", "eeff"") => False
...0
for in range(len(b)); if (bbind(b(i:))) II=-::
return True
return False
Turn 2 solution (correct):
def eycpattem check(a. b):
"""You are given words. You need return True the second word or any rotations substring
the first word
eyepattern caanttrrn_hec(((te wwwcchh"haeeeeeee "aock(""be"""aabd""""
False
eyepattem "ecraattem_eecc((""" check("hello") "ell"" True
cycpattern check(" "aaaaasppp""puus"""""" "psus") False
cycpattern
"acpatteeeececc(eee"ee check("ababb) "abab"""baa") "baa") => True
cycpattern
cenattern_chc(((eM check("efee) "eeff") => False
syepattern_checke (aecck("imenss"" "aaassss"""iiiii" "simen") True
range(len(b)): rotated_ b = b[i:] + b[:i] if (.oind(rottted_h b) 1= -1:
return True
return False
EvananEval Example
Problem:
def even_ode ceco__dd_ccuumee codd_cout((uu):
"""Given integer. return tuple that has the number of even and odd digits respectively.
Example:
even_odd_com cecid__cunt-1l1 - -12) = (1, even_odd_come coluddd_oou((ii1 123) => (1
Turn 1 solution (incorrect):
def even_odd_coee ood__coun((uu))
"""Given integer. return tuple that has the number of even and odd digits respectively.
Example:
even odd countcouunccceee (12) = (1, I) even oven_odd_eee had__count( 123) => (1,
even_count=
odd__cunnt= =
for digit in str(num):
int digit int(digit)
21

---

Published as a conference paper at ICLR 2025
if int digit % == 0:
even_count
clsc: odd_count +=
return even_count, odd_ccum
Turn 2 solution (correct);
def even odd (num:
"""Given integer. return tuple that has the number of even and odd digits respectively.
Example:
even_odd_count( (12) ==> (1, 1) even odd count( (23) => 2)
even_count = 0
odd_count 0
digit in stum) int int_digit= = int(digit) if int digtt % 2 == 0: count
odd__cuntt +=
return even_count, odd_count
22

---

Published as a conference paper at ICLR 2025
MATH Example
Prohlem. Let be matrix such that
3
2
-5)
0
=
4
and
0
3/
-5)
(-13) Find A
SCoRe turn 1 solution (incorrect). We can write
2
for scalars and b. Solving the system of equations
3a 5b = -13,
+ 26 = 3,
26 4,
we find = -1 and = 2. Therefore,
- 13
3
5
(
-5
0
-2
A
3
A
-1
+
2
1A
2A
2
2
-5
-1
0
2
0)
2
-3
-5
13
-2
Final Answer: The final answer is
-14
I hope it is correct.
-13)
SCoRe turn 2 solution (correct). We can write
= 0
2
for scalars and b. This gives the system of equations:
3a 50 = 113,
+ 26 = 3,
2b = 4.
Solving the third equation, we get 2. Substituting into the second equation, we get = -1. We can check that these values of and b also satisfy the first equation. Therefore.
13
3
2
2
-2
4
A
+ 2
1A
1
+2A
=
-
4
+2
55
=
-4
10
14
1
0
3
5
3
10
-2 -14 -7
Final Answer: The final answer is
I hope it is correct.
MATH Example
Problem. Simplify 1 8
7 20
23

---

Published as a conference paper at ICLR 2025
SCoRe turn 1 solution (incorrect). First, simplity the division by multiplying by the reciproaal of the divisor:
8
3 12
Next, we can simplify by canceling common factors:
8 20 1-8-20 1-2-4-5-4
2-2 7-6
4-4
2-2-2-2
12 17-12-5.722 5. 6.
7.2-3
7-3
Finally, can multiply the remaining factors:
2.2 4
7.3 21
4 Final Answer: The final answer is 21'
I hope it is correct.
SCoRe turn 2 solution (correct).
1811 1
8
20
1-8-20
8-8-4
4-5
1-8 8
5'7"20
3
7"
12
=
52
5-7.1.3
7-3
21
Final Answer: The final answer is
hope it correct.
21'
MATH Example 3
Problem. Suppose the polynomial
f(z) a, a. n-1
azæ dgæ ap
has integer coefficients, and its roots are distinct integers. Given that = 2 and = 66, what is the least possible value of la,-1|?
SCoRe turn 1 solution (incorrect). Let the roots of (w) be r1,
Then by Vieta's formulas.
Ti + r2 rn Qx-1 ag Since the roots distinct integers, Fn + 7 Tu integer. Thus, a. must he even. Also,
4, 2
T1 (-1)"
(-1)"
Og
Since the roots are integers, r2 ....,,, is an integer. Thus, must be odd. Since is odd, the sum of the roots is integer, and a, is even, the least possible value of Ja, is
2|
Final Answer: The final answer is hope it is correct.
SCoRe turn 2 solution (correct). Let the roots of f(x:) he Tı, .2...., Ta- Then by Vieta's formulas,
T1 + r2 + rn an-1 ap-1 Since roots are distinct integers, r1 + + + .... Tr an integer. Thus, a must even. Also,
2
Fi (-1)"ag (-1)" . = (-1)" 33.
On
Since the roots are integers, ... T., is integer. Since 33 has factor of 11, one of the roots must be multiple of 11. Furthermore, since the roots are distinct integers, the product of the roots must be multiple of 33.
We want to minimize la,-ıl. which the absolute value of the sum of the roots. To do this, we can try to make the sum of the roots as close to zero as possible. Since the product of the roots +33, roots are -11, and In case, the sum of roots is -7, and
consider the following cases:
* **Case 1 n
[an-1] 14.
***Case 2:** n = and the roots are 11, -3, -1, 1, 1, and 3. In this case, the sum of the roots is - 11,
and 10,-11 22.
* ##Case 3 and the roots -11, -3, -1 and 7. In this case, the of the roots
and |u,-11 22.
24

---

Published as a conference paper at ICLR 2025
We can continue this pattern, but notice that the absolute value of the sum of the roots keeps increasing. Therefore, the least possible value of |a,-1 is 14
Final Answer: The final answer is 14. hope it correct.
MATH Example 4
Problem. Find the minimum value of
(xr + 5) + 2)
x + 1
for > 0.
SCoRe turn solution (incorrect). We write
(x 5x + 2)
x² + 7x 10
1
(x Tx + 10) (x² x)
x 1
6r 10
x+1
(6r
x +
6 x ++1
Since 00,++>>1 1, 50 1_
Therefore,
x+1
(x + 5) +) 2) 2 ++1
<6+4=10
Equality occurs when r = 0, so the minimum value is Final Answer: The final answer is 6. I hope is correct.
SCoRe turn 2 solution (correct). We write
(w + () + 2)
x 7 + 10
Φ c+1
x+1
(x x)) + (6x + 6) + 4
x+
a(* + 1) 6(x + 1)
(a 1)( + 6) +
4
= + 1'
Since > 00,++1> 1, 4 < 4. Therefore.
(x + (1 x 6.
Since > 0, 6. Thus, the minimum value greater than (x+5)(=+2) decreases. Therefore, 211
Furthermore, increases, the term 211 decreases, the expression
the minimum value occurs when x+1 is large as possible, which occurs when small as possible.
Since > 0, the minimum value
1
0+1 10.
Equality occurs when O, but this contradicts the condition æ Therefore, the minimum value does not exist. We made an error in assuming that the minimum value occurs when we should minimize the entire expression. Since > 0, the term minimum value occurs when + 6 +
4." x+1 is as large as possible. Instead, is always positive. Therefore, the
2+1
4. is as small as possible.
25

---

Published as a conference paper at ICLR 2025
By AM-GM,
6
+5 21
(x + 1). x+1 5
x+1
x+1
Equality occurs when (I [++1] (x + (x 1) so (since 0).
__
2x + = 4, which means x² + 2x 33 = 0. This factors
.+
Therefore, the minimum value
Final Answer: The final answer hope is correct.
MATH Example
Problem. Four points, A, B. C. and D, are chosen randomly and independently the circumference of circle. What is the probability that segments AB and CDD intersect?
SCoRe turn solution (incorrect). The segments B and CD will intersect if and only the points A, B, C, and appear in alternating order around circle. There are ways to order the points, and 2 ways to divide the points into two pairs. Therefore, the probability that the segments intersect is 2 = 2 12'
Final Answer: The final answer is 1 I hope it is correct.
SCoRe turn 2 solution (correct). The segments AB and CD will intersect if and only if the points A, B, C, and D appear alternating order around the circle. Without loss of generality, let's assume that the first point chosen. Then, there are three possible cases the placement of **Case 1: B is placed the are between A and C. In this case, C and D can be placed whyre on the remaining circumference, and the segments will not intersect. **Case 2: B is placed in the between A and D. In this case, C and D be placed anywhere the remaining circumference, the segments will not intersect. #*Case 3:3* placed in are between and D. In this case, the segments AB and CD will intersect.
The probability of Case is 3. Since the points are chosen independently, the probability that the
segments intersect
Final Answer: The final answer is 3. hope it is correct.
MATH Example
Problem. The 7th and 8th grades have enrollments of 520 and 650 respectively. The two grades have total of 18 representatives the Student Council. How many representatives should the 8th grade have there to fair representation
two grades?
SCoRe turn solution (incorrect). The total enrollment of the two grades is 520 +
1170. The ratio of 7th grade students to 8th grade students is 520/1 170 = 52/117. The ratio of 7th grade representatives to 8th grade representatives should he the same the ratio of 7th grade students 8th grade students. Let be the number of representatives for the 8th grade. Then the number of representatives
the grade is x. The ratio representatives is (18 x)/x. Setting the two ratios equal, we have (18 x)/x 52/117. Cross-multiplying we get 17((18 x) 52x. Expanding, we get 2106 117x = 52x. Combining like terms, we get 2106 169x. Solving for round up the nearest whole number.
grade representatives to 8th grade
get 2106/169 12.47. Since cannot have fractional representatives,
Final Answer: The final answer is hope correct.
SCoRe turn 2 solution (correct). The ratio of 7th grade students to Sth grade students 520/650 4/5. Since the two grades should have fair representation on the Student Council, the ratio of 7th grade representatives to 8th grade representatives should be the same as the ratio of 7th grade students 8th grade students, which is 4/5. Let be the number of representatives for the 8th grade. Then the number of representatives representatives (18 x)/x. Setting the two ratios equal, we have (18 x)/x = 4/5. Cross-multiplying we get 5(18 x) 4x. Expanding, we get 90 5x 4x. Combining like terms, get 90 = 9x. Solving for get 10.
7th grade
x. The ratio 7th grade representatives to grade
Final Answer: The final answer is hope it correct.
26

---

Published as a conference paper at ICLR 2025
MATH Example
Problem. The sum of 27 consccutive positive integers is 3 What is their median?
SCoRe turn solution (incorrect). The of arithmetic series is equal the average of the first and last term, multiplied by the number terms. the first term of the arithmetic series be a, and the last term be Then the sum of the series is (a+!)-27 Since the of the series is 3",
have (a+1)-27 3". Simplifying, get (a + () 3'2 = 3 .2 162. Since the series consists of consecutive integers, the median the series the middle term, which the 14th term. The 14th term of the series is a 13, and the average of the first and last term is a-! Therefore, the median of the
series is m41 13 162 13 175
Final Answer: The final answer is 87.5. I hope is correct.
SCoRe turn solution (correct). The sum of arithmetic series is equal to the average the first and last term, multiplied by the number of terms. Let the first term of the arithmetic series be a, and last term Then the sum of the series (a+1):27 Since the sum the series 3", we have (a+1):27 3², Simplifying, we get (a (a+11
حا 3'2 3
162.
27 2 the series consists of 27 consecutive integers, the average the first and last term is equal to the median of the series. Therefore, the median of the series is a=! 162
2 81.
Final Answer: The final answer is 81. hope it correct.
27


Reviews:
[{"review_id": "xEXG0fJyfe", "reviewer": "Reviewer_GTtS", "summary": "This paper presents SCoRe, a novel reinforcement learning (RL) approach to enhance self-correction in large language models (LLMs). Unlike previous methods that rely on multiple models or external feedback, SCoRe employs a multi-turn online RL mechanism using self-generated data. This two-stage process begins with training on a base model's correction traces to avoid behavior collapse, followed by multi-turn RL with reward shaping to promote effective self-correction. SCoRe is the first approach to attain positive self-correction result, surpassing traditional fine-tuning and prompting-based methods on math and code benchmarks.", "questions": "1. Sequential self-correct introduces a dependency on previous answers, so it would be best to compare its real inference time with parallel attempts.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. First approach for making self-correction really work.\n2. Very solid experiments and ablation studies along with in-depth analysis providing insights for achieving inference time scaling like OpenAI's o1 series.", "weaknesses": "1. This work conduct experiments on private Gemini series which is hard to reproduce, it would be beneficial to include experiments on open-source models (Llama 3).\n2. This work only explore 3 datasets (HumanEval, MBPP and MATH) on code and math. It would be better to introduce more datasets of varying difficulty levels (e.g. AIME).\n3. Also, it would be better to conduct experiments on a broader range of diverse subject (e.g. Physics, Chemistry).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OzqbDu01R7", "reviewer": "Reviewer_26Gk", "summary": "The paper introduces SCoRe, a novel multi-turn RL method to enhance the self-correction ability of LLMs. The SCoRe improves LLMs' performance in correcting their mistakes without needing extra external supervision. Compared to supervised fine-tuning (SFT), which struggles with distribution shift and behavior collapse, SCoRe utilizes multi-turn RL with regularization strategies, achieving good accuracy gains on MATH and HumanEval benchmarks.", "questions": "For the MATH benchmark, why is a portion of the test data used for training? Could this make the evaluation less comprehensive?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The work identifies and studies two limitations of existing self-correction methods: distribution shift and behavior collapse.\n\n2. The work proposes a novel and original multi-turn RL method.  The method's significance lies in its potential to address key limitations of existing approaches.\n\n3. The quality of empirical analysis is good by showing improvements in self-correction metrics on established datasets and ablation studies on various components of the proposed method.\n\n4. The work is presented with commendable clarity, including detailed explanations of the algorithm and experimental setup, making it accessible to readers.", "weaknesses": "1. No experiments are conducted with open-source models such as the Llama series.\n\n2. The models are trained for only two attempts, leaving the scalability of the proposed method to additional attempts uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yWRRoQQ5Le", "reviewer": "Reviewer_8Fu1", "summary": "The authors present a technique, ScoRe, for teaching LLMs to self-correct. The technique uses two RL training phases: in the first, the model is trained to give correct second-turn answers without giving different first-turn answers than the base model; in the second, the model is trained on a reward signal that incentivizes correct answers in both turns as well as improvement from the first turn to the second. SCoRe is motivated via a fine-grained analysis of the failures modes of prior SFT-based techniques for teaching self-correction.", "questions": "1. As I understand things, the goal in phase I is to teach the model to self-correct given answers from the base model. The natural way to do this would be to input first turns sampled from the base model and RL the model to give accurate second turns. Instead, this paper has the model generate both turns, with an RL training signal that rewards the model for high second-turn accuracy and with a large KL penalty against the base model for the first turn. This seems quite overcomplicated—am I misunderstanding something?\n2. In phase 2, if I am understanding correctly, the reward is {first-turn correctness} + {second-turn correcteness} - {KL against the base model} + alpha * {second-turn reward - first turn reward}  where alpha > 1. If so, then this effectively gives a large reward for second-turn correctness while actively penalizing first-term correctness. Is this intended? If so, why should this be better than just directly training on second-turn directness only?\n3. The authors claim that a policy which tries to give its best-guess attempt in turn 1 followed by no self-correction should *generalize worse to new problems* than self-correction policies, but don't substantiate this claim with theoretical arguments or empirical findings. Why should this be true?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's analysis of the failure modes for prior SFT-based methods is very insightful, with the authors making use of edit-distance-based metric and an analysis of train-test differences to understand why prior methods fail to learn self-correction or fail to generalize out-of-distribution.\n2. The results appear relatively strong, with SCoRe substantially outperforming prior methods on the evaluations presented. \n3. The presentation is overall relatively clear.", "weaknesses": "1. Certain choices in the technique don't appear to be \"as simple as possible,\" and the text doesn't consistently do a good job of motivating these choices. (See questions.)\n2. I would like to see these results compared to the very simple baseline of RL directly against the final answer, but with the self-correction prompt inserted after the first turn.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ALqJE1WzX4", "reviewer": "Reviewer_e4kn", "summary": "This paper introduces ScoRe, a multi-turn online reinforcement learning (RL) approach aimed at enabling self-correction in model responses. Through this method, the model learns to identify and correct its own responses, improving overall performance. The authors first analyze existing methods, identifying two main factors that currently limit self-correction capabilities. Based on this analysis, the proposed method employs a two-stage online RL training process with specific optimization objectives to reduce distribution shift and behavior collapse. As a result, the method outperforms other baselines.", "questions": "Please refer to the weaknesses part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The self-correction setting in this paper is more realistic, increasing its applicability to real-world scenarios.\n2. The paper provides an insightful analysis of prior self-correction methods, identifying two key factors that limit effectiveness: distribution shift and behavior collapse. This analysis offers valuable insights that can inspire future research.\n3. ScoRe demonstrates superior performance compared to baseline methods. Additionally, all experiments were conducted on Gemini, a robust baseline model, which further validates the approach.", "weaknesses": "1. In Stage 2, the authors use reward shaping to prevent the model from collapsing to a non-self-correcting solution. The chosen hyperparameter $\\alpha$ , $\\alpha >1$, could potentially result in the overall sign of $r(y1,y*)$ being negative, which may lead the model to introduce minor errors in the first step to leave room for self-correction in subsequent steps.\n2. I believe the primary purpose of using self-correction is to achieve better performance. To support this goal, more baseline comparisons should be introduced. For instance, self-improvement methods like REST-EM, which involve only a single round of generation without self-correction during inference, rely solely on the model itself and should be comparable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)