InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning
Authors: Matthew Y. R. Yang
Hao Bai
Ian Wu
Gene Yang
Amrith Setlur
Aviral Kumar
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
reasoning
intervention
SFT
RL
Primary area: foundation or frontier models, including LLMs

Abstract: Outcome-reward reinforcement learning (RL) has proven effective at improving the reasoning capabilities of large language models (LLMs). However, standard RL assigns credit only at the level of the final answer, penalizing entire reasoning traces when the outcome is incorrect and uniformly reinforcing all steps when it is correct. As a result, correct intermediate steps may be discouraged in failed traces, while spurious steps may be reinforced in successful ones. We refer to this failure mode as the problem of credit assignment. While a natural remedy is to train a process reward model, accurately optimizing such models to identify corrective reasoning steps remains challenging. We introduce Intervention Training (InT), a training paradigm in which the model performs fine-grained credit assignment on its own reasoning traces by proposing short, targeted corrections that steer trajectories toward higher reward. Using reference solutions commonly available in mathematical reasoning datasets and exploiting the fact that verifying a model-generated solution is easier than generating a correct one from scratch, the model identifies the first error in its reasoning and proposes a single-step intervention to redirect the trajectory toward the correct solution. We then apply supervised fine-tuning (SFT) to the on-policy rollout up to the point of error concatenated with the intervention, localizing error to the specific step that caused failure. We show that the resulting model serves as a far better initialization for RL training. After running InT and subsequent fine-tuning with RL, we improve accuracy by nearly 14% over a 4B-parameter base model on IMO-AnswerBench, outperforming larger open-source models such as gpt-oss-20b.

Published as a conference paper at ICLR 2025
INT: SELF-PROPOSED INTERVENTIONS ENABLE CREDIT ASSIGNMENT IN LLM REASONING
Matthew Y. R. Yang', Hao Bai", Ian Wu', Gene Yang', Amrith Setlur', Aviral Kumar'
Carnegii Mellon University, University of Illinois Jibana-Champarr
ABSTRACT
Outcome-reward reinforcement learning (RL) has proven effective at improving the reasoning capabilities of large language models (LLMs). However, standard RL assigns credit only at the level of the final answer, penalizing entire reasoning traces when the outcome incorrect and uniformly reinforcing all steps when it is correct. As a result, correct intermediate steps may be discouraged in failed traces, while spurious steps may be reinforced in successful ones. We refer to this failure mode as the problem of credit assignment. While natural remedy is to train a process reward model, accurately optimizing such models identify corrective reasoning steps remains challenging. We introduce Intervention Training (InT), training paradigm in which the model performs fine- aneegrained credit assignment its own reasoning traces by proposing short, targeted corrections that steer trajectories toward higher reward. Using reference solutions commonly available in mathematical reasoning datasets and exploiting the fact that verifying model- generated solution casier than generating a correct one from scratch, the model identifies the first eπrоr in its reasoning and proposes single-step intervention to redirect the trajectory toward the correct solution. We then apply supervised fine-tuning (SFT) to the on-policy rollout up the point of error concatenated with the intervention, localizing error to the specific step that caused failure. We show that the resulting model serves as far better initialization for RL training. After running InT and subsequent tie-ttnnigg with RL, we improve accuracy by nearly 14% over 4B- 1B-parameter base model MO-AnsserBench. outperforming larger open-source models such as gpt -osetoss-2000
SFT Upweight off policy reference solutions
RL Uniformly updote on-policy rollouts
Intervention Training
Upweight interventions & steps before mistake
Seif-verify to find mistoke
Propose
intervention
Base model
Base model
Base model
Base model
problem
problem
problem
Incorest
referurro
Correct
Incorrect
(
Update
No update
rollout
solution
Figure I: Intervention training (InT) for improving credit assignment. InT proposes single-step interventions to replace incorrect intermediate steps in reasoning traces (1). Conditioned on these localized corrections, the model can generate counterfactual continuations that succeed where the original failed (2). We then perform supervised fine-tuming these interventions, enabling effective credit assignment by upweighting the likelihood of the interventions in place the mistakes.
INTRODUCTION
Post-training eeattrriinig large language models (LLMs) with reinforcement learning (RL) has proven to be a highly effective strategy for improving their reasoning capabilities. As RL scaled to more challenging tasks, training becomes increasingly dominated by incorrect rollouts, which are often

---

Published as a conference paper at ICLR 2025
long-horizon trajectories composed of many smaller, structured reasoning steps. Many such rollouts contain substantial portions correct reasoning, yet outcome- based RL treats these correct steps the same the critical mistakes that cause failure. This indiscriminate penalty can lead to undesirable behaviors including increased verbosity or frequent, premature shifts in the reasoning process, which have been widely observed when applying outeome-reward RI. (Chen et al. 2025 Wang al.) 2025b). Moreover, on sufficiently difficult problems where no correct rollout is produced at all, outcome-reward RL provides no learning signals, as advantages collapse to zero.
In principle, addressing this challenge requires solving the problem of credit assignment: if we could identify the step at which reasoning trace goes astray and penalize that particular step while reinforcing other possible promising steps, we could drive the model toward more favorable outcomes. However, existing approaches to credit assignment are difficult to apply in the L.M setting. For reasoning traces, estimating the "'redit" (i.e., value) of each step typically requires running multiple branched rollouts conditioned on a given prefix, which is prohibitively expensive (Kazemnejad et al.) 2024 Luo et al.) 2024 Kim et al. 2025). Prior work attempts to amortize this process by training explicit value functions (or process reward models) (Setlur et al., 2024b Luo et al.] 2024). but training such functions over long reasoning traces remains open question. Moreover,
with reasonably accurate value function, optimizing it to identify an alternate step is challenging (Wang et al.
2025a Zhang et al. 2025c) due to the vast space of possible future steps.
How can we perform credit assignment without training a value function? Rather than explicitly learning value function, we ask the model itself to identify how failed trajectories be locally corrected. This approach relies on an LLM's ability to implicitly perform value estimation and optimization jointly in order to propose an alternative, improved step, which we refer intervention. Our idea is to reduce the task of generating end-to-end solutions to the substantially easier task of verifying individual steps, which can be accomplished via textual comparison against reference solution (a textual "diff"). In our method, we instruct base model to analyze the differences between a reference solution and an incorrect reasoning trace that previously generated, with the goal of identifying the first point at which the trace goes wrong and proposing a single-step corrective intervention. When conditioned on this intervention, the model can then generate counterfactual reasoning traces that succeed from right before where the original rollout had failed.
corrective
We then train the model to internalize these interventions by applying supervised fine-tuning (SFT) on them, followed by RL. By fine-tuning the model on targeted single-step interventions, selec- tively reduce the likelihood of the incorrect reasoning steps that the base model would otherwise produce, shifting it toward more favorable steps instead. Our approach remains notably simple and computationally efficient, as it avoids branched rollouts, explicit functifn training, or modifying the RI. objective include step-level rewards. Throughout this procedure, never rely on a larger model; instead, we leverage the asymmetry (Setlur et al.) (025bb) in task difficulty between cetructtio--ollooinng verification, and generation within the same model to perform credit assignment. We call this approach Intervention Training (InT).
We compare InT to several alternative approaches that leverage reference solutions for learning, including methods that train policies directly on reference solutions. We find that InT produces more "on-policy" trajectories (those with high likelihood under the base model), whereas other approaches tend to produce more "off-policy" ones. After online RL, In'T improves performance by nearly 10% on average across four challenging mathematical reasoning benchmarks, including a ~~14% gain MMO-AnswerBench (Luong et al., (2025), which consists of IMO-level problems curated by former medalists. These results demonstrate InT as a simple and effective paradigm for improving the reasoning capabilities of LLMs, through improved credit assignment.
PRELIMINARIES, NOTATION, AND PROBLEM STATEMENT
Learning objective. Training LLMs for reasoning typically involves an LLM n, model-generated reasoning traces ~ (- produced on input prompts x ~ p. and binary reward r(x,y) {0. 1} quantifying the correctness of the final answer. The goal is to maximize average reward, given by:
max (Tx := Ex-py (x) [)(
We can maximize the objective by updating the policy with the policy gradient (Willimms 1992)
n' +- + Ex D +()b)))) (r(x V, log (y))))),
(I)
2

---

Published as a conference paper at ICLR 2025
where Duain is the set of training problems and A is the policy used to generate samples for training. In policy gradient methods such as GRPO (Shao et al. 2024), के = Told is a periodically updated copy of rt, with additional correction terms to account for the distribution shift between Vold and æ.. We can also express SFT with the same rule by. where स represents a pre-collected offine dataset consisting of only correct trajectories, that Fquation[] Teffetivell maximizes likelihood on these traces.
Credit assignment. Reasoning trajectories y can naturally be decomposed into a sequence of individually complete segments y (yo. ,T),, where each reasoning step yt contributes unequally to the final answer. Ideally, learning signals should reflect this uneven contribution. However, in Equation II the policy is updated with binary rewards that entirely depend on the correctness of the final answer ("outcome rewards"). Outcome-reware RL normalizes these re- wards across responses sampled for the same prompt x, yielding advantages. When the advantage A(x, y.) = y.) - 1/m Σ}= r,), yT) is positive, every step in the response y: is uniformly rein- forced: when it is negative, all steps are uniformly discouraged. With this form of reward allocation, steps that play no role in reaching the solution may still be upweighted, while steps that are locally correct but followed by later mistakes may be suppressed. In long-horizon settings where trajectories Y; consist of hundreds of steps, the resulting noise can overwhelm the learning signal.
This limitation of reward RL. manifests in two ways. First, when training with large token budgets, rewards often plateau early and do not improve substantially over the course of training (Setlur et al. 2025b Luo et al. 2025 Yeo et al. 2025). Second, responses become either excessively verbose or prematurely truncated, often failing to maximize the underlying outcome reward (Qu et al., 2025b Chen et al.
2025 Wang al. 2025b). Avoiding these issues requires accurately identifying which steps are responsible for success or failure and disentangling correct steps from incorrect responses and incorrect steps from otherwise correct responses.
This problem of determining how individual intermediate steps contribute to the final outcome is known as credit assignment. If we can estimate the "credit" of each intermediate step with step-level value function (also called Process Reward Models, or PRMs), we can update π by encouraging the correct parts of solution while discouraging the incorrect ones. However, this approach faces two major obstacles in practice. First, reliably estimating such value functions is difficult due to its cost (Luo et al., 2024| (Setlur et al.) 2024b; Kazemnejad et al. 2024) and training instabilities (Wang et al. 2025a ZZhang et 20.5cc. Second, even when suffeiently accurate PRM is available, simply penalizing incorrect parts of solution does not ensure that they will be replaced with higher-value, correct alternatives. As result, policy optimization remains a challenging problem, since efficiently searching for suitable replacement steps is a difficult task in itself.
Problem statement. Our goal is to develop effective approach to credit assignment. In particular, we focus on assigning credit within incorrect rollouts by pinpointing the incorrect steps that derail the solution. In our setting, we assume access to reference solution (e.g.. a human-written one) for each training problem, as they are commonly available in existing open-source math datasets.
Why focus credit assignment for incorrect traces? As we move toward more challenging RL tasks, incorrect rollouts dominate trajectories observed during training. For example, Figure TT illustrates that, curated set of Olympiad-level Math problems, more than 80% of rollout groups contain no successful trajectories at the start of training. Being able to extract useful learning signals from these failed rollouts would therefore tap into an abundant source of supervision that would otherwise be completely discarded by outcome-reward RL, as advantages within these groups collapse to zero
Where within incorrect trajectories is credit as- signment needed? In Figure 2 we visualize the locations of the first error that derails incorrect trajec-
% of proble ems
10 5
% of problem
10
tories. We find that these errors can occur at a diverse set of points throughout the trajectory, often far from the initial steps, with more than 60% of the first errors arising after the 50th step. Consequently, effective credit assignment for incorrect rollouts requires a method that can precisely identify errors occurring at many different points along trajectory.
5
250 500 Step # of first mistake
0
5000 10000
Token # of first mistake
Figure 2: Location first occurring mistakes in incorrect rollouts. Incorrect rollouts may contain many correct preceding steps before the mistake. Detected by Gemini 2.5 Pro with Prompt For the M. we study, this decomposition is implemented by splitting the generated text using the delimiter 1 "inninn (see Figure 3) following prior work (Qu et al. 2025b).
3

---

Published as a conference paper at ICLR 2025
atii all elAt... be aniiffaee sequanee intages that not eventully ressadic which more there do not oust stagers frand di-i such that of emagers
had the namber pohroondds
fit1-00 ag Ad 45
toples conta Jain 1I.
sach
stogers
comprix planchove
2025]]
Sokdion hrle
sequence eventul perfodic, then the cannet eventuslly become teddcc S asd have tritaly mane disties alomants we just cnn't values comntallly repout an partods fart
Therstor,, rost hav [c] unken recprocal la abc then la] 1 contradction.
But hand' the crcial
whe - lothor mrue?
Whr in thet mar?
Hotakey
thee
fichely
only
Wher sboun
Ovan the sequarecs one toal pertodi, it might so heve only linitaly many diaind 2025-
The coeflidunt
iragaas jAS
Kay Thecrum In cumpho analvil)
A2 Al AQ di and alll la rosta and ce aboot valuc Howowos, cirte
Intervention
Fiail Arower
nondiume 1033 couituilly
unit
Iscomett
appesm eacchy
pertodi, bur every bicdi of angit
certane
X
Micoo lengh think of vay to co contmmut suct secuunce? to dedstarddda the
must howe Liniinee
periiiiiits
Final Ansoer 2005
t
Cormet
correct
Figure 3: Example mistakes encountered by Qwen3-4B-Instrue and proposed interventions. By correcting the mistakes with interventions, the rollouts lcad to correct final answers as opposed to originally incorrect ones.
CREDIT ASSIGNMENT VIA SELF-PROPOSED INTERVENTIONS
We develop a scalable approach for credit assignment in LLMs. We reduce this problem to two challenges: (i) identifying which reasoning steps are problematic and should receive low credit, and (ii) searching for alternative steps that attain higher credit. Most existing solutions to problem (i) rely on estimating value functions, which remains computationally expensive for LLMs. Approaches to problem (ii) typically require step-level on-policy RL; however, given the inherent difficulties of RL training for LLMs, these methods introduce an additional layer of complexity for practitioners. In this section, we provide an alternative approach to credit assignment that bypasses these challenges by leveraging the difficulty gap between cnstruction-followeeee
and problem-solving tasks.
Main idea. Our approach collapses the two steps above into a single procedure that directly proposes an improved reasoning step. While such a procedure might appear to require a stronger LLM, we show that meaningful corrective steps can be generated using the very same model. The key is to exploit the asymmetry task difficulty between generating correct solutions from scratch and verifying individual steps by comparing against the reference solution. In practice, base models are often far more reliable at the latter: they can compare a generated reasoning trace against a known correct solution even when they cannot produce the solution themselves. Accordingly, base model to use test-time compute to effectively perform a "diff" between a reasoning trace and human- written reference solution, identify the step at which the trace goes astray, and propose single-step corrective intervention. Next, we explain our approach for implementing this idea.
instruct the
3.1 PROPOSING INTERVENTIONS THROUGH SELF- VERIFICATION
We leverage instruction-follovin LLMs to verify their own reasoning traces and propose counterfac- tual steps that lead improved outcomes, circumventing the need for expensive rollouts explicit training and optimization of PRMs. We implement this idea via a two-stage prompting procedure.
First, given model-generated reasoning trace, instruct the base model to verify each step of the trajectory for logical correctness, conditioned on access to a reference solution. This produces a list of reasoning steps that are potentially incorrect, which we require the model to index explicitly by their position in the trajectory. We then focus on proposing a corrective alternative for the first incorrect step. This design choice motivated by the fact that the initial eтror must corrected for an incorrect rollout to become correct. Moreover, errors that occur later in the trajectory are often consequences of earlier mistakes; once an early incorrect step fixed, the remainder of the trajectory is unlikely to follow the same erroneous path, rendering corrections to later errors potentially unnecessary.
Upon identifying the first incorrect step y:- with step number t", we proceed to the second stage, in which the model generates an alternative next step y.- to replace yr.. We refer to this alternative as an intervention, whose purpose is to steer the remainder of the trajectory toward the correct answer. We provide concrete examples of generated single-step interventions in Figure Bl with additional examples in Appendix D and
E Concretely, we obtain the error location t* and corrective step y.- using a single query (- y,y, ; PIaT); the instruction Per is provided in Box 2 Our
4

---

Published as a conference paper at ICLR 2025
prompt borrows components from Huang & Yang (2025) to produce step-by-step verification logs of the trajectory and to identify the first mistake by classifying each step. Because t" and Yt- are generated by the same model that produced the original y. approach enables credit assignment via self-generated feedhack. Additionally, in the few cases where the intervention contained the final answer itself; discard these cases to prevent any leakage from the reference solution, that the model does not simply learn to memorize the final answer, but rather learns to complete the solution. Next, evaluate whether interventions generated in this manner results in successful rollouts on problems that the base model could not solve at first.
3.2 DO INTERVENTIONS IMPROVE SUCCESS RATES?
We empirically evaluate self-generated interventions using the Qwen3-4B- 11022-4--Instuutt-5000 base model by measuring whether a proposed intervention improves the probability of success on rollouts that would otherwise fail. Specifically, identified error y<t-, followed by the intervention Yt-. We then sample continuations from the model conditioned this prefix, i.e., T(- x, yrr yx-) (row 4 "4B-Instruct" in Table 4). and evaluate resulting trajectories to obtain an unbiased estimate of the expected reward. In Figure f (left), we isolate the effect of interventions by comparing this value against two baselines: (1) continuing the
concatenate problem with reasoning steps preceding the
rollout from the erroneous step without applying the intervention TT- X, y Yr-) (row 3 above I4B-rustruct"), and (2) resampling from the step immediately preceding the error πT x, yx(e)) (row 2). Our results show that interventions increase the average reward from 0.0713% to 1 1.56%. This 22x increase demonstrates that while the base model may be unable to solve these problems initially, it can leverage self- celf-verification capabilities to generate single-step interventions yr- to replace the original erroneous step yr- and successfully steer the model toward a correct solution.
Beyond average reward, we also examine if uti- lizing our proposed interventions enables the model to solve new problems. To assess cov- erage, we compute the number of problems on which we can generate at least one correct roll- out (out of 32) under each of the three settings in Figure 4 (right), which correspond to rows 2, 3, in Table Continuing from interventions achieves the highest coverage, solving 80 out of 334 unique problems compared to 29 out of 334 when continuing from the error. This jus- tifies the efficacy of interventions, and further suggests that fine-tuning models to internalize interventions could potentially increase the frac- tion of problems receiving non-zero rewards during downstream RL, which would enable learning across a broader range of tasks. We include additional ablations and results in Appendix
1.5
1.58%
80/-334
20
accuracy
Goverau
1.0
10
31. 33.334 29
0.0726%
0.0
Y:"
ye
Y1" ye
Figure 4: Performance comparison of interventions. Continuing the rollout by concatenating the proposed intervention step y-- the prefix Y<r- yields sub- stantially higher accuracy (left) and a larger number of unique problems solved (right) than either continu- ing with the original erroneous step yr- continuing directly from the prefix without any intervention (-).
Takeaways: Effectiveness of our proposed interventions
Conditioning rollouts on proposed interventions increases success rates by 2 x0 Hint-guided rollouts are complementary to interventions. Both improve overall reward. cntructiio-followiii capability is critical: the instruction-tuned 4B-Instruct model solves 7 more problems (out of 334) than the rrasoning-focuseee 4B model. Intervention quality scales with model capacity: interventions proposed by 30B-A3B- Instruct achieve 2x higher accuracy and solves more problems compared to 4B-
4 INT: INTERVENTION TRAINING FOR CREDIT ASSIGNMENT
Next, we use the generated interventions for training. Our training procedure is simple: we perform SFT on the collected intervention data to patch the base model, preparing it for subsequent RL training. Then RL training goes as usual, but now it can learn from these very challenging problems that would have otherwise largely produced incorrect rollouts during RL. As we show in our experiments, this patching step is sufficient to enable the model to solve previously unsolvable problems and leads to improvements in performance. We now describe our training procedure detail.
4.1 TRAINING VIA SUPERVISED FINE-TUNING ON INTERVENTION DATA
5

---

Published as a conference paper at ICLR 2025
1. Generate interventions
2. SFT
Step 1 Step 2 Step 3
Step
x (- (x.)
Fr n (-Ixyy,Pir)
RL
y-u (-1x)
0.50
V
0.40
Self-verity against ref. solution
16
32
64
128
Figure 5: Intervention training (InT). Int verifies incorrect rollouts against reference solutions to identify the step t" at which the first error occurs, proposes alteraative step yee, and performs SFT on these steps before RL.
A natural starting point for leveraging generated interven- tions during training is by rumning supervised one-tuning (SFT) on the collected intervention data. The goal of this step is to teach the model to internalize the correc- tive patterns expressed by interventions, so that at test time, when reference solutions are no longer available, it can spontaneously generate similar corrective steps. This immediately raises the central design question surround- ing this simple training procedure: which parts of an intervention-guiae -uuiiedd rollout should be used in training?
Configuration
Coverage
Accuracy
Prefix, no suffix, filter
202/235
7.71%
no filter
196/233
5.06%%
prefix suffix
162/235
2.87%
111/235
2.31%
Table I: A study of design choices for SPT on interventions. We compare diffcrent SFT configurations: whether the prefix y is cloned, the suffix y>i is included, and in- terventions are filtered to retain only those that yield at least one correct rollout. Cloning the prefix and excluding the suffix while ap- plying the correctness filter yields the highest coverage and accuracy. The 235 problems were suhsampled from DeepScaleR.
Consider generating full solution using an intervention. For problem x and an incorrect response y x(- |x), we generate an intervention ýe- ~ x(- x,y, PirT) to replace the erroncous step yr. We then continue the response by sampling V>e" ~ x( yerr Se.), Given the resulting intervention-guidee trajectory, we consider several design choices for SFT: whether to train on the prefix y<r. the suffix ý > and whether to include only the interventions whose continuations Y> lead to a correct final answer.
Main idea. We hypothesize that cloning only the proposed intervention without its preceding prefix insufficient. If the prefix y<r- is not reinforced during SFT, the fine-tuned model may generate alternative prefixes y'e-- for the same problem x, introducing different errors than those encountered in the original rollouts used for training, rendering the intervention y.- irrelevant. While one could mitigate this issue by collecting multiple prefixes, doing is impractical: covering the space of possible prefixes is already expensive, and anticipating which prefixes the fine-tuned model will produce even more challenging as the output distribution T(. x) shifts after fine- tueittnnng. Motivated by these considerations, we follow Qu et al. (2024) and choose to clone both the prefix yer- and intervention. We observe in our results in Table ! that this choice leads to substantially improved coverage and accuracy, solving 40 more problems out of 235 compared to not cloning the prefix.
Next, we examine whether cloning the suffix ý>-- beneficial. We hypothesize that cloning a correct suffix could reduce the space of future sequences explored during subsequent RL training, thereby limiting exploration. Empirically, we find that this is indeed the case and cloning the suffix is detrimental. As shown in Table I. training on full trajectories (y car Fr., the number of unique problems solved compared to training only on the prefix and intervention (y<r.,rr Vr-). Finally, observe that filtering interventions to retain only those that lead to at least one correct continuation over 32 rollouts, further improves post-SFT coverage by 6 out of 235 problems. In summary, the optimal SFT config consists of cloning the prefix and intervention, excluding the suffix, and applying a correctness filter. Our final SET objective is given by:
nearly halves
V RJ Dww.Dnnnnnnn--))e (tx.)) V log T \yer Σ v log T(y: Siggt(yyleee lyeo). (2)
4.2 FINE-TUNING VIA REINFORCEMENT LEARNING
After fine-tuning the base LLM on intervention data for small number of steps, we proceed with RL post-training using standard ouward methods such as GRPO (Shao et al. 2024). If the model has successfully "internalized" the intervention patterns, its rollouts should avoid

---

Published as a conference paper at ICLR 2025
many of the emrors observed in the base model while reinforcing previously correct behaviors. Starting RL from this SFT-initialized model then further amplifies these corrective behaviors, sup- presses unproductive reasoning segments, and en- ables the extraction of learning signal from roll-
Algorithm InT: Intervention Training
Require: Base LLM π, Intervention generation
prompt Pur. Problems Dinin
1: Data Collection: Dunt + +
2:
for each x E Peain do
3: Generate y ~ (-|xx)
outs that previously yielded none. We refer our approach of training LLMs on intervention
4: fr ~ (+1122, y, Por)) (error location, inter-
vention)
data as InI (Intervention Training). Our practi-
5: DiNTT <- Diar U (x, co
cal approach shown in Algorithm| | first patches
6: end for
7: Patching: r' SFT Dua))
the LLM on intervention data (Line 7) and then subsequently runs RL. We provide a schematic illustration in Figure 5
8: RL training: xT" <- RL((', Duain)
9: return π²"
Summary: Intervention Training (InT)
Performing SFT on the proposed interventions with the prefix, without the suffix on filtered traces leads to the best performance and provides good initialization for online RL.
EXPERIMENTAL EVALUATION OF INT
In our experiments, we compare models trained with InT against alternative approaches on challeng- ing problems where standard RL training predominantly produces incorrect rollouts. Overall, we find that InT achieves the strongest performance across all settings it reduces the fraction of problems that yield only incorrect rollouts during RL, consistently increases pass @ k across all evaluated values of kt and delivers strong gains on standardized benchmarks.
5.1 EXPERIMENTAL SETUP, COMPARISONS, AND EVALUATION METRICS
To construct a dataset of challenging problems, we draw from the following sources: (1) Polaris (53k problems), filtering for instances that achieve zero accuracy under 64 rollouts; (2) AceReason- Math (50k problems), similarly filtered for zero accuracy under 64 rollouts; (3) Omni-MATH (4.4k problems), filtered for zero accuracy under 128 rollouts; and (4) IMO-AnswerBench (360 problems), after removing those reserved for the test set. Reference solutions are available for most problems from Polaris, AceReason-Man remaining problems, we generate reference solutions using Gemini. After applying these difficulty filters, we obtain a pool of approximately 4500 hard problems. We then generate interventions for this set and retain only those that yield non-zero reward in at least one of 32 rollouts, resulting in 1076 problems with corresponding interventions. As shown in Table (top), we further select a subset of 334 problems to compare naïve rollouts against rollouts with interventions.
(Luo et al. (025). and Omni-MATH (Gao al.) 2024); for the
Baselines and comparisons. To evaluate the efficacy of InI, we compare it against several alternative approaches for patching model behavior on Duain. followed by rumning RL. Our primary comparisons are: (1) Reference Solution SFT + RL, which distills reference solutions into the base model prior to RI; (2) Standard RL, which applies RI. directly to the base model; (3) Self-Reflection SFT + RL, where the model is prompted to rewrite entire solutions rather than propose single-step interventions (see Appendix HH. followed by SFT and RL; and (4) Hint-guided RL, which runs RL on int-augmented problems (we use the dataset released by (Qu et al. 2025a), a hint-guided RL method targeting problems of comparable difficulty). For all RL runs, we run GRPO for 400 steps.
Cost analysis of PRM training. A natural compari- son to our approach is to train a process reward model (PRM) and use step-level rewards derived from it. However, high-quality PRMs are currently not avail- able for the Qwen3 models we use in this work. While prior studies have demonstrated that PRMs can be trained and utilized (Setlur et al.) 2024b), do- ing so requires substantial training and data collection compute. For example, the Qwen2.5-Mait Math-PRM se- ries relied on roughly three million rollouts together with complex filtering pipelines. Additionally, recent
0.80
0.60
--- In!
0.40
0.40
0.20
16
64
256
1024
16
64
256
1024
k
Figure 6: Performance of the post-SFT model. SFT on interventions increases pass@k k on both train and test splits for l = 16, 32, ... 1024, on 32 randomly sampled problems from each split.
7

---

Published as a conference paper at ICLR 2025
work suggests that existing PRMs tend to underperform difficult problems (Feng al. 2025 Given these challenges, we do not include a PRM-based comparison, and focus on alternative methods that leverage reference solutions to improve performance. Compared to TRM-based approaches, InT provides a much simpler and more scalable mechanism for credit assignment.
Evaluation benchmarks and metrics. Prior work on RL-trained reasoning models primarily evaluates performance on intermediate competition math benchmarks such as AIME 2025 and HMMT 2025. However, these benchmarks are already quite old compared release dates of current base models. To address this limitation, we evaluate InI on substantially harder standardized benchmarks. Specifically, we report results on MMO-AnswerBende (Luong et al. 2025), A.MO-Bench (An et al.) 2025), and Apex Shortlist (Balunović et al., 2025), all of which consist of Olympiad-level problems released after the public release of our base model, -reen--B-Insttuut-e
addition, we scrape the November 2025 HMMT competition from the official HMMT website to evaluate on problems constructed after the release of the base model, further reducing the risk of training-set set contamination. manually select 40 problems: 10 each from Algebra, Combinatorics, Geometry, and Number Theory, restricting to problems with easily verifiable answers (e.g., integers simple fractions) to ensure that evaluation reflects mathematical reasoning rather than sensitivity to answer parsing. We apply an analogous filter to AMO- removing problems whose answers cannot be verified automatically (e.g., proof-style responses).
For IMO-AnswerBencch
5.2 RESULTS: SFT WITH INTERVENTIONS
We examine the performance of InT immediately after SFT on inter- vention data (Section 4 "patching"). Prior work suggests that RL LLMs primarily reallocates probability mass toward already-existing successful trajectories, effectively converting high pass@ k in a base model into high pass@ @ after RL training (Yul 2025 He et al. 2025 Yue et al., 2025). From this perspective, pass @ k serves both as diagnostic of whether the model is capable of sampling correct solutions and as an upper bound on downstream accuracy achievable through subsequent RL training. Motivated by this view, we com- pare the pass @ k of the SFT model to that of the base model on both the training and test sets in Figure 6 The SFT model consistently 1024 on both splits,
0.4
Babe
In
Ply
0.2
0.0
Test Figure 7: Average probability of intervention tokens. Intervention tokens receive higher probability under the trained model vs. the base model on both the training and test sets, indicating that SFT causes the model to learn to pro- duce intervention like steps even on unseen test problems.
Train
achieves higher pass@k k across ho = 16, 32,
indicating that fine-tuning on intervention data enriches the distribu- tion of successful trajectories and provides a strong initialization for RL. In the next section, we show that further RL. training from this checkpoint indeed translates these gains into improved final performance. We also find that likelihood of intervention tokens increases PFst-SFT (Figure 7. indicating that InT enables the model to internalize interventions successfully.
Next, we compare InT against alternative approaches to generate corrections to incorrect rollouts. Specifically, we evaluate performance after running SFT on Self-Reflection traces, Reference solutions, and thinking traces from DeepSeek-RI R1 (R1 Think and R1 Summary, which correspond to content inside and after the <t <think tags, respectively), to compare In. with fine-tuning on traces generated from more capable LLM. Across both training and test sets, we find that InT T consistently improves pass @1 and pass@k & relative to all comparisons in Figure train middle, test right).
Finding: InT traces more likely under the base model. Be-
1.66
yond improvements in pass@ k, In.T generates traces for SFT that yield the highest likelihood under the base model. Figure 9 reports the average negative log-likelihood (NLL) of tokens drawn from trajectories generated by different sources, including human- written reference solutions, Gemini-generated reference solutions, DeepSeek-RI outputs, as well as self-reflection and InT trajecto-
Entropy
left)
0.26
Ref. Salf.
InI
Base
Solution
Reflect
SFT Model
Figure 8: Entropy of the next- token distribution for various models. SFT on off-policy traces leads to elevated entropy, while InT preserves a low-entropy dis- tribution closer to the base model, yielding more stable initialization for subsequent RL.
ries. We find that InT trajectories are the most likely under the base model distribution, followed by self-reflection, RI outputs, and reference solutions. This observation is important because fine- tuning on highly off-policy traces is known to be problematic. Prior work (Kang al. (024)) shows that such traces often memo- rized by the model, and fitting them can significantly distort the base model's next-token distribution. Consistent with this perspective,
8

---

Published as a conference paper at ICLR 2025
1.0
0.60
SFT
0.5
pass@k
0.40
--- InT
0.50
Reff-Reflect
0.0
R1 Think
Human Ref. Sal. Gemini Ref Sol.
R1 Think
R1 Summary
Self Reff-tent
InT
0.25
0.20
--- A1 Summary
Solution
0.00
0.00
64128
32 64128
k
Figure 9: InT trains on tokens that are more likely under the base model and aftains highest performance on both train and test problems. (left) Average negative log-likelihood (NLL) computed over 64 sampled traces. Ref. Sol. denotes reference solutions written by humans the Gemini-2.5-Pro model. R1 Think and R1 Summary correspond content inside and after the DeepSeek-RI <thinkk nk> tags, respectively. Self-reflect refers to the self-reflection baseline which the model prompted rewrite entire incorrect solutions given a reference solution. InI trains on the most on-policy traces as exhibits the lowest NLL. (middle) Train pass @ l on 64 sampled training problems. In. achieves the highest pass @ & across all values of k. (right) Test pass@k k on IMO-Bench, AMO-Bench, and Apex Shortlist. n'I again attains the hest performance across all &.
Figures D miiddle, right) show that the relative performance of different methods closely tracks the ordering of their NLL values. While this correlation is not necessarily causal, it motivates a closer examination of why highly off-policy traces pose challenges for fine-tuning.
Why does cloning off-policy traces perform poorly? Fine-tuning on highly off- traces requires shifting probability mass away from tokens with high likelihood under the base model toward tokens considers unlikely. This redistribution naturally increases the entropy of the next-token distribution, an effect we observe empirically in Figure g Such high- entropy initializations are problematic for subsequent RL training, as they lead to overly random rollouts that hinder effective exploration and learning. One might attempt to mitigate this issue by training for longer, allowing the model to eventually assign higher likelihood to the
Mean: 135.8
Percentage
0.0
1000
off-policy tokens. However, this introduces a different failure mode: prolonged fine-tuning on off-policy data (e.g., human- reference solutions) distorts the base model's existing reasoning patterns, since these traces dif- fer substantially from the behaviors and skills the model has already learned. As a result, cloning off-policy tra- jectories performs poorly regardless of whether training stopped early or run for longer, and might not produce good RL initializations.
Mean: 6835.6
3.2
0.0
6
2500
5000
7500 10000 12500 15000
Number of Tokens
Figure 10: Interventions
short. Top:
Intervention are typically under tokens. Bottom: full rollouts are ~ 7k tokens on avg.
contrast, InT generates traces with most tokens coming from the base model because interventions are short compared to full rollouts TO leading to trajectories that with higher likelihood under the base model. This avoids large shifts in entropy, as shown in Figure g the resulting entropy of the next-token distribution after applying InT is roughly comparable to that of the base model and can thus support effective exploration during subsequent RL training.
5.3 RESULTS: INT RL SOLVES MOST HARD TRAINING PROBLEMS
Next, we show that on our curated training set of difficult problems, initializing RL from the patched checkpoint produced by InT leads to the largest increase in reward and the greatest re- duction in the fraction of problems that produce no correct rollouts (the "zero-advantage ratio"), as shown in Figure II This result confirms that training interventions can effectively help avoid errors made by the base model during RL. Moreover, we observe a similar effect test problems. On two unseen problems selected from the IMO shortlist, InT is able to correct errors made by the base model, as illustrated qualitatively in Appendix!!
0.3
InT
+ RL
Zero Adv. Ratio
Self Reflect SFT + RL
RL
Qel Sol. SFT + RL
0.8
0.1
0.6
0.0
200
Gradient steps
Gradient steps
Figure 11: RL training stats. Left: RL training reward. Right: Zero-advantage ratio fraction problems for which the model never produces a correct rollout. InT yields both higher reward and substantially lower zero- advantage ratio, enabling learning problems that pre- viously yielded no signal. The best approach Reff-Refecction SFT RI., hut leads to much weaker test performance (sce Table

---

Published as a conference paper at ICLR 2025
InT + RL
SFT on Ref. Sol. RL
RL
Comparison
0.70
0.60
0,70
0.60
pasek
0.60
Passekk
0.60
0.40
0.50
0.50
0.50
200
0.20
200
200
0.40
400
400
400
0.40
32
16
16
16
Figure 12: Pass@ le across RL training iterations test problems. We plot pass @ l for three model initializations: (i) base model patched with LnT,, (ii) base model fine-tuned on reference solutions, and (iii) base model itself. Across all values of k, InI achieves the highest pass @ l: through training (right).
5.4 RESULTS: INT EXPANDS THE PASS @ k FRONTIER ON TEST PROBLEMS
We next present the main results InT on the full evaluation set described in Section Specifically, track pass @ & performance across RL training iterations from 0 to 400 for three models: (i) the base Qwen3-4B--nss
model, (ii) -wenn3-BBIInsten Swen3-4BBInnttuu patched with TnT. We find that initializing RL from the model produced by InT leads to the largest improvements in pass @ throughout training. As summarized in Figure [2] InT outperforms both the base model and reference solution SFT across RL iterations.
fine-tuned on reference solutions, and (iii)
5.5 RESULTS: PERFORMANCE ON STANDARDIZED BENCHMARKS
In the previous section, we showed that InT pushes the pass k frontier across all values of k on both training and test problems. We now examine whether these gains translate into improved pass @
performance on held-out benchmarks. To this end, we evaluate InT on four challenging reasoning benchmarks: MMO-AnsweBeench. HMMT 2025 (November), AMO-Bench, and Apex Shortlist.
As shown in Table InT achieves an average score of 33.77 72 across these benchmarks, corre- sponding to a ~599 relative improvement over the base model (21.17) and ~99 improve- ment over the SFT+RL bascline (28.26). On IMO-AnswerBenchh a high-quality test set cu- rated by IMO medalists, In' attains a score of
Model
IMO-Answereench
11 11.88 (16K)
DespSeck-RI-OSS Ri-SSek--R--S2--eeaaae
18.44% (32K)
gpe-oss-20b
23.36% (32K)
Qwen3-18- -an-------nsttt-2550777 + InT + RL (Ours)
25 62%6% (16K)
Table 2: Evaluation IMO-AnswerBencha Per- formed different "official" token budgets. 25.62, more than double the base model performance (11.68) and surpassing standard RL (23.46).
Moreover, we find that performing SFT on the reference solution does not synergize well with subsequent RL, attaining an average score of 20.76 that is lower than that of the untuned base model (21.17). This suggests that naïvely tine-tuning on off-policy reference solutions may restrict effective exploration in downstream RL, as discussed in Section [..2 In contrast, by remaining on-policy, InT enables RL to further improve performance, as reflected by consistent gains across all metrics. While Self Reff-Reflection SFT shows promise on certain benchmarks (e.g.. 36.72 on AMO-Bench), it generalizes less robustly than InT. On IMO-AnswerBench, Self- Reff-Reflection attains only 15.53, compared to 25.62 for InT. InT achieves the highest reward value of 28.83, boosting the training reward higher than any other method while not falling into memorization traps of off-policy SFT.
Model
IMO-AAnweeBench
HMMT 2025 Nov
AMO-Bench pass
Apex Shortlist passee
Average
Base
11.68
41.61
26,25 24
20.79
5.53
21.17
RI.
23.46
46.46
35.21
22.72
13.47
28.26
Hint-guidel
47.27
SFT ref. solutions RI.
11.56
27.49 4745
25.19
20.51
19:07
201.66
+ SFT on self-reilcttions + RL
15.53
38.65
36.72
23,93
23.19
27.60
IoT (Ours)
25.62
49.77
36.16
28.22
28.83
33.72
Table 3: Performance challenging math benchmarks. Pass@) I (IMO-AnwwerBencch, HMMT 2025 Nov) and @ (AMO-Bench, Apex Shortlist) estimated using 128 rollouts. Dunin reports the average train reward step 400, smoothed using exponential moving average. InT achieves the strongest overall performance followed by hint- lunt-g RL, standard RL, and other SFT approaches.
Summary: Intervention Training (InT) Results
More "off-policy" SFT data leads to higher entropy and worse pass@k @k performance
InT leads the greatest increase in training reward and decrease in zero-adv ratio.
InT achieves the highest test accuracy on a variety of difficult benchmarks, including a nearly 14% increase in IMO MOO-AnsweBBench, surpassing gpt-oss-206 20b with a 4B model.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Mohammad Hossein Amani, Aryo Lotfi, Nicolas Mario Baldwin, Samy Bengio, Mehrdad Farajtabar, Emmanuel Abbe, and Robert West. RI for reasoning by adaptively revealing rationales, 2025. URL
org/ abs 2506 112551611111... 18110
Shengnan An, Xunliang Cai, Xuezhi Cao, Xiaoyu Li, Yehao Lin, Junlin Liu, Xinxuan Lv, Dan Ma. Xuanlin Wang, Ziwen Wang, and Shuang Zhou, Amo-bench: Large language models still struggle in high school math competitions, 2025. URL htttp::/arxxv.orr//Baaat
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating Ilms on uncontaminated math competitions, February 2025. URL https://maah
arena ai/
Angelica Chen, Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Samuel R. Bowman, Kyunghyun Cho, and Ethan Perez. Learning from natural language feedback. Transac- tions on Machine Learning Research, 2024. ISSN 2835-8856. URL oppeneeviwwwww
.net/ / forum?
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang. Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like Ilms, 2025. URL https
7/arxiv. wwt. abs///bb///// 2412.42222 21187
Jie Cheng, Gang Xiong, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Yisheng Lv, and Fei- Yue Wang. Stop summation: Min-form credit assignment is all process reward model needs for reasoning, 2025. URL htttt::////aaee
wtwp//g/bbb........ /200.25544... 15275
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao. Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang. Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen. Shanghao Lu, Shangyan Zhou. Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie. Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu. Yao Li. Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li. Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu WU, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL
abs////////////////////////// /501. 12948
Zhangying Feng, Qianglong Chen, Ning Lu, Yongqian Li, Siqi Cheng, Shuangmu Peng, Duyu Tang, Shengcai Liu, and Zhirui Zhang. Is prm necessary? problem-solving rl implicitly induces prm capability in Ilms, 2025. URL htt.hhttss///aaitii
/2505 210.5.11..1227.7.

---

Published as a conference paper at ICLR 2025
Zeyu Gan, Yun Liao, and Yong Liu. Rethinking external slow-thinking: From snowball errors to probability of correct reasoning, 2025. URL httpps//a/xxx...////////////////
15602
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang. Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni- math: A universal olympiad level mathematic benchmark for large language models, 2024. URL
bttps::/arxiia org/ abs 2410.4000000 07985
Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting grpo beyond distribution sharpening, 2025. URL T https::/aaxxi...oooa/p
abs/ 2003//550 206.00600000 02355
Zheyuan Hu, Robyn Wu, Naveen Enock, Jasmine Li, Riya Kadakia, Zackory Erickson, and Aviral Kumar. Rac: Robot learning for horizoorizon tasks by scaling recovery and correction, 2025. URL
https://arxiv org/ abs 07953
Yichen Huang and Lin F. Yang. Winning gold imo 2025 with a model-agnostic verification-and- refinement pipeline, 2025. /RRLattpss///axxx..rrr/aaaaa//b/
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https
//github.com/ h/tp://aabb..oo///gggggaaaaaaaaa
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, et al. Openai ol system card. arXiv preprint wrXiv:2412.16720, 2024.
Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. Unfamiliar finetuning examples control how language models hallucinate, 2024.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
Michael Kelly,
Chelsea Sidrane, Katherine Driggs-Campler and Mykel J. Kochenderfer. Hg dagger: Interactive imitation learning with human experts, 2019. URL httpss://axxiv.rrgg aba// 1810. 02890)
Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng. Moontae Lee, Honglak Lee, and Lu Wang. Process reward models that think, 2025. URL https
//arxiv.org/
2504 16828
Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Kiril Gashteovski, Carolin Lawrence, Julia Hockenmaier, Graham Neubig, and Sean Welleck. Scaling evaluation time compute with reasoning models as process evaluators, 2025. URL https:://aaxi..tttt///
077503.19877
Yoonho Lee, Joseph Boen, and Chelsea Finn. Feedback descent: Open-ended text optimization via pairwise comparison, 2025. URL
07919
Jiazheng Li, Hongzhou Lin, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Yi Wu, and Jingzhao Zhang. Questa: Expanding reasoning capacity Ilms via question augmentation, 2025. URL
httpss///rxiv.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, and Yuke Zhu. Robot learning on the job: -luman-in-the-logg autonomy and learning during deployment. The International Journal of Robotics Research, 44(10-11:1127-17444 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and
Min Lin. Understanding r1-zero-like training: A critical perspective, 2025b. URL ht https:
7/ warxiv. wwwpv..rr///bb//55333333 20783
12

---

Published as a conference paper at ICLR 2025
Zijun Liu, Peiyi Wang, Rumxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling, 2025c. URL
abs/2504.0244
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoming in language models by automated process supervision, 2024. URL BtLLtttss///axt...ooo
2406.06592
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing op-peevew with a 1.5b model by scaling rl.
radio b75 htt.aaooooo.....oo.o.eeeeeeeee
ca 819 01900c146000eeeec333331011111110
2025. Notion Blog.
Thang Luong, Dawsen Hwang, Hoang H. Nguyen, Goinaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu H. Trinh, Quoc V. Le, and Junehyuk Jung. Towards robust mathematical reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025. URL https: senthology.ors 2025. emn 1-mmlp-maii maammnnn nn. 1794/
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refinc: Iterative refinement with self-feedback, 2023. URL/attps://arxe .rg. hbtpg//ab///33... 17651
Ajay Mandlekar, Danfei Xu, Roberto Martin-Martín, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Human-in-thh--oopp imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733, 2020.
Alexandre Piché, Ehsan Kamalloo, Rafael Pardinas, Xiaoyin Chen, and Dzmitry Bahdanau. Pipelinerl: Faster on-policy reinforcement learning for long sequence generation, 2025. URL https
//arxiv.org//bs//// 225..../rrg//bb//05.// 19128
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how self-improve. arXiv preprint arXiv:2407.18219, 2024.
Yuxiao Qu, Amrith Setlur, Virginia Smith, Ruslan Salakhutdinov, and Aviral Kumar. How to explore to scale RL training of L.MSs hard problems? https://bllg ml cmu wdw.le25/1222111
166/how-te exp Sooepporrettee to- -all-ri-ttaiiing---eee 1isfe-1mmm--e hard 2025a. CMU Machine Learning Blog.
Yuxiao Qu, Matthew YR Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine- tuning. arXiv preprint arXiv:2503.0757. 2025b.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 627-635, 2011.
Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RI on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold. arXiv preprint arXiv:2406.14:: 14532, 2024a.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. arXiv preprint arXiv:2410.08116, 2024b.
Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test time compute without verification is suboptimal, 2025a. WtRl/httss///arxiii org abs /2002 12118
13

---

Published as a conference paper at ICLR 2025
Amrith Setlur, Matthew Y. R. Yang, Charlic Snell, Jeremy Greer, lan Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for Ilms, 2025b. /ttpst/s::/// wwtp/aaiivvv...../ee/eaaa org/ aa//a/////////// 2506. 09026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL
http///wwvvvr.../aa.aao abss///////// /402
03300
Zhihong Shao, Yuxiang Luo, Chengda Lu, Z. Z. Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, and Xiaokang Zhang. Deepseekmath--22 Towards self verifiblee mathematical reasoning, 2025.
URL htapL//tpss///rxii.orr//////////
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with lims. arXiv preprint arXiv:2501. 12599, 2025.
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce lims step-by-step without human annotations, 2024.
Teng Wang, Zhangyi Jiang. Zhenqi Hc, Shenyang Tong, Wenhan Yang, Yanan Zheng, Zeyu Li, Zifan He, Hailei Gong, Zewen Ye, Shengjie Ma, and Jianping Zhang. Towards hierarchical multi-step reward models for enhanced reasoning in large language models, 2025a.
//arxiv.org aww.abb//535 13551
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts are all over the place: On the underthinking of ol-like Ilms, 2025b. URL httpt:ttp///mm //arxii//arxvv.... orgg ab
s/2501 18585.
R. J. Williams.
Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-256, May 1992.
Philipp Wu, Yide Shentu, Qiayuan Liao, Ding Jin, Menglong Guo, Koushil Sreenath, Xingyu Lin, and Pieter Abbeel. Robocopilot: Human-in-the-loop interactive imitation learning for robot manipulation. arXiv preprini arXiv:2503.07771 2025a.
Yuyang Wu, Yifei Wang. Ziyu Ye, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in 1lms, 2025b. URL
02. 07266
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance, 2025.
wwwpas/wwwrrr//////.//./.... www.gggg//b////oo/a////e////// /5004
14945
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in Ilms, 2025. URL http::tttts/riiiiii wwwpriii..r./ooooooooooooo /abs///////////// abaaabss550 220222522.00 03373
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source I1m reinforcement learning system scale. arXiv preprint arXiv:2503. 14476, 2025.
Yang Yu. Pass metric for rlvr: A diagnostic tool of exploration, but not objective, 2025. URL 6231
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model?,
2025. URL htt.pstts:: http:///wwwwwwwoooaaoao//aso 13837
14

---

Published as a conference paper at ICLR 2025
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic "differentiation" via text, 2024. URL https://arxiv.org/
abs/2406.0 07496
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 200: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URL htttps:///axxi Ttttpp///ax....ogg//b/////////// 18892
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint 1xXi:22408.152400 2024.
Xiaoying Zhang, Hao Sun, Yipeng Zhang, Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng. Critique-grpo: Advancing Ilm reasoning with natural language and numerical feedback, 2025a. URL
http:///wg.abb/////////e//// 03106)
Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak. Bread: Branched rollouts from expert anchors bridge sft for reasoning, 2025b. URL https
(axxiv.org//bs//5555.1
2506.17211
Zhenru Zhang, Chujic Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning, 2025c. URL
15

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Bhavya Agrawalla, Taeyoun Kim, Yuxiao Qu, Zheyuan Hu, Baolong Bi, Jiahe Jin, and the rest of the AIRe lab at CMU for informative discussions, feedback and input on our results and a previous version of this paper. This research is supported by gift from Apple, Schmidt Sciences A12050 Early Career Fellowship, and an Open Philanthropy grant on technical Al safety.
This research used the DeltaAl advanced computing and data resource, which is supported by the National Science Foundation (award OAC 2320345) and the State of Illinois. DeltaAl is a joint effort of the University of Illinois Urbana- Chaaaaaan and its National Center for Supercomputing Applications. We thank the staff at DeltaAI for their continual and timely assistance with compute allocation. Experiments in this paper also utilized H100 GPU resources from the Orchard cluster in the FLAME center at CMU for which we are grateful for. AK also thanks the TRC program at Google Cloud for their continued support.
A RELATED WORK
Credit assignment LLM reasoning. The effectiveness of long-horizon RL with outcome re- wards (DeepSeek-AT et al. 2025 Jaech et al. 2024 Team et al. 2025 Hugging Face, 2025 is
often crippled by credit assignment: it is unclear which intermediate steps in a long response should be "credited"" for the outcome reward. Credit assignment is usually not problem when responses are short and the model can produce enough successful rollouts to contrast them against incorrect ones, but pose as significant bottlenecks when the horizon increases (Setlur et al., 2025a Wu iet al., 2025b; Gan et al. 2025). While most methods reward each token with the outcome- level advantage (Yu et al. 2025 Zeng et al. 2025 Liu al.) (0225b), others use process reward models
(PRMs) to assign dense token or step-level rewards (Lightman et al. 2023 Wang et al. 2024 Qu al. 2025b) that can reinforce correct steps and promote unlearning of incorrect ones. Although PRMs may improve RL compute efficiency (Setlur et al., 2024b] a). the notion of what a process reward should measure and how to calculate precisely
not yet settled. Some work derives process rewards by running "branched" rollouts which is costly (Luo et al. 2024 [Setlur et al. 2024b) Kazemnejad et al. (024)), while other work trains PRMs with human annotations and suffers from reward hacking (Cheng et al. [2025] Wang et al.) (025a). Even after training PRM, one must still devise mechanisms to optimize it at a per-step level in order to identify alternative steps that lie on a correct trajectory, task that is itself nontrivial (Wang et al. 2025a Zhang et al. (025c). In contrast, our work leverages the asymmetry between a model's instruction- cotrructtoo-folooinnnn and solution generation capabilities to enable direct verification of reasoning trace and construction of a corrective step that redirects the trajectory. This approach effectively amortizes value estimation and policy optimization into a single procedure, resulting in simpler and cheaper approach for credit assignment.
While our approach, in principle, can be viewed as using generative models for some form of verification (Zhang et al., 2024, Liu et al., 2025c; Kim al. 2025 Khalifa et al.) 2025). is distinct from these prior works in that we task the model with explicitly pointing out the location of a single, key mistake, rather than verifying every step and judging the solution to produce binary judgement of outcome-level or process-level correctness. We amortize value estimation with policy optimization to directly produce an improved step (intervention). Moreover, unlike the majority of work on generative reward models or verifiers, our focus is improve training directly rather than using inference-ting time methods (e.g.. best-of-N search).
Learning from natural language feedback. A related line of work explores using natural language feedback to improve RL training. These approaches typically leverage feedback to refine rollouts, which are then used to update the policy. For example, Chen et al. (2024) combine human feedback with a separate refinement model to improve Caaaple,Chhnnee
seeer-ggeeetted outputs that are distilled back into (2025) use teacher model generate correction trajectories for off- (025a) employ critique-guider self-refinement to generate improved trajectories for off-policy RL. In contrast, our approach generates short, targeted natural language feedback to correct individual steps within otherwise on-policy trajectories. Moreover, interventions are proposed by the same model that produced the incorrect reasoning trace, without relying on human feedback or stronger external models.
the policy via SFT. Yan et al. policy RL, while Zhang et al.
Additionally, there have been a class of "text gradient" methods (Lee et al. 2025 Madaan al.. 2023 Yuksekgonul et al. 2024) that leverage LLMs to generate textual feedback to improve test-time
16

---

Published as a conference paper at ICLR 2025
performance. While method is similar in that it also employs textual learning signal, it differs in two key ways: the feedback is short and targeted, and it is used to improve training rather than solely to enhance inference-time behavior. Finally, as we show in Section [ the use of reference solution substantially improves performance compared to approaches that only rely on the priors in the base model to generate corrective feedhack.
Hint-guided RL.. Several concurrent approaches also leverage reference solutions in RL, but differ in how this information is used. Rather than using reference solutions for verification and credit assignment, these methods guide the policy directly. For example, (Li et al. 2025 Amani et al. 2025; Zhang et al.] 2025b; Qu et al., 2025a) incorporate
guidance the form of nartial-solution prefxes, le.. static rationale snippets that the model conditions on during training. While such approaches substantially improve exploration by enabling the model to sample successful trajectories on otherwise unsolvable problems, they do not directly address credit assignment on incorrect traces. As a result, our method is complementary to this line of work and can be naturally combined with it, synergy we demonstrate empirically in Section 3
Moreover, we find in Figure 2 that for a substantial fraction of problems, the first incorrect step occurs deep into the reasoning trace. In particular, over 70% of errors appear after the first 50 steps. Existing hint-guided methods typically repurpose a prefix of a reference solution as a hint, and cannot directly tackle emrors that arise later in trace. One could attempt to mitigate this by providing longer hints, but this approach risks enabling the model to simply rely on the hint itself, effectively learning to "cheat".
Training with interventions outside of LLMs. Applying interventions at points of failure has been explored in domains beyond LLM training. For example, DAgger (Ross al. (011) mitigates compounding errors in imitation learning by querying an expert for labels on states visited by the learner's 's own policy. HG-DAgger (Kelly et al. (019) adapts this framework to self-driving, and subsequent work extends it to robotic manipulation and long- mng-horizon control tasks (Hu et al. 2025 Wu et al., [025a] Liu et al. 2025a Mandlekar et al. 2020). Across these domains, intervention- bassed methods typically achieve improved data efficiency and faster convergence compared to behavior cloning, due to more effective credit assignment. In contrast to this line of sprior work, where interventions are provided by humans, we show that model can self-propose interventions to correct its own reasoning given a reference solution, by leveraging asymmetries in its capabilities.
B DISCUSSION AND PERSPECTIVES ON FUTURE WORK
In this work, we introduce InT, a simple yet effective approach for assigning credit to incorrect rollouts. InT addresses this challenge by instructing the model to self-verify and propose single-step corrective interventions. We then fine-tune the model on these constructed interventions and perform RL post-training on the resulting patched model. By enabling more precise credit assignment, InT achieves strong performance in both training and test settings, allowing off-policy reference solutions to improve already capable models while avoiding the typical drawbacks of off-policy supervised fine-tuning. Looking ahead, several promising directions emerge from this work:
Self-improvement by combining different LLM capabilities. In this work, we leverage the asymmetry between generation and verification capabilities of L.MSs to assign credit incorrect rollouts. A natural next step is to further strengthen verification by explicitly training models on verification tasks, potentially eliminating the reliance on reference solutions. This would enable more autonomous sell-improvement by removing the dependency on human-provided solutions. Moreover, stronger verification would allow LLMs to tackle tasks that require step-by-step correctness checking, such as IMO- Pro-ProoBBench Luong et al.) (2025). Ideally, verifiers would be trained in continual fashion, improving alongside the solution generation model. Concurrent work by /Shao et (2025) explores this direction by training a meta-verifier that validates verifications themselves, but training this veta-verifier itself needs human or bigger model supervision. Beyond verification, another key capability for celf-improvementi is problem generation. For example, could LLMs learn to construct their own training problems by modeling trajectories encountered by expert problem proposers, such as those in IMO-style competitions? If these components can be combined, they suggest fully autonomous system in which one model proposes problems, another solves them, and a verifier provides reward signals that drive continual parameter updates.
Credit assignment in continual learning. In this work, we study credit assignment in a single- turn setting for solving math problems. An important direction for future work is to extend these
17

---

Published as a conference paper at ICLR 2025
Intervention Proposed By
Rollouts Are Conditioned On
Coverage
Accuracy
Omni-MATH subset with pass@ 128 = O, valuess estimated with 32 rollouts
problem x problem x correct prefix Yer"
40 40/334 334
0.0984%
2.-
31 / 334
0.0726%
problem x correct prefix Y<r* original step y
/
0.0713%
4. 4..-Isstruct (Ours)
problem correct prefix y-c- intervention Y+-
80/334
1.56%
5. 4B (Ours)
problem x correct prefix y<r* intervention y:-
334
2.65%
6. 308-A33--Inseee Instruct (Ours)
problem correct prefix y-c* intervention y:*
334
2.87%
7. 1B-Inntruct w/o. ref. sol.
problem x correct prefix y+ + intervention y:*
334
0.171%
Subset with mint-guuded pass@ 16 0, values estimated with 16 rollouts
8.
problem x + hint h + correct prefix y<r- problem x correct prefix y-r* intervention Y1- problem hint h correct prefix y yeer intervention yr-
11 176
2.38%
9. 4B-Instruct (Ours)
176
3.05%
4B-Instruct
176
4.62%
Final Training Set
11. 4B-Instruct (Ours)
problem x correct prefix Ycr* intervention y:-
1076 4048
4.27%
Rollouts conditioned on proposed interventions and alternative strategies. Accuracy measures the average reward, and coverage measures the number of problems that see at least one correct rollout. Conditioning rollouts our proposed interventions yields the highest coverage and average reward on the hard problem set, outperforming approaches that do not interventions (e-g-, naive rollouts conditioning on hints) and those that generate interventions under weaker supervision (e.g., without reference solutions). We also find our interventions can be combined with existing approaches such as hint-guided rollouts (see rows 8-10) to obtain stronger results. 4B-Instruct denotes Qwen3-4B Bee-----nnsttt Instruct-2507, 4B denotes Qwen3-4B, and 30B-A3B denotes Qwen3-30BB --B--3-BB-BB-B---- Instruct-2507.
Table 4:
ideas to continual improvement regimes, where rollouts are progressively compressed into LLM- generated summaries and neural memory modules, and the effective context evolves over time. For example, how can credit be traced back to earlier decisions that may persist only through memory representations? How can we disentangle errors arising from poor generation from those caused by imperfect memory, and jointly improve memory architectures alongside the generation model? These questions are interesting to study in future work.
C ABLATIONS ON DIFFERENT WAYS OF PROPOSING INTERVENTIONS
In this section, we study ablations that isolate which ingredients are most important for proposing effective interventions. Overall, interventions provide a more fine-grained mechanism for steering rollouts than hint-only conditioning, and their effectiveness depends on access to reference solutions, strong cnstruction-followiii behavior, and model capacity.
Ablation: Interventions outperform hint-guided rollout generation. Having established that interventions improve reward and enable solving problems that were previously unsolvable, now compare this approach to alternative ways of leveraging reference solutions. As discussed in Section [ several concurrent works incorporate reference solutions into RL by conditioning the policy on partial solution prefixes as hints. These hints exploit the instruction-follooinn capabilities of base models to "steer" generation toward the correct region of the solution space, enabling successful completions on otherwise difficult problems. Since hints cannot directly reveal the solution, we construct them by randomly sampling a prefix from the initial one-third of the reference solution. We study whether hint-guided rollouts are more effective than intervention based rollouts, and whether the two approaches can be combined.
Table 4 compares these settings. Conditioning on interventions alone (row 9; πT( x,y<r . )) achieves higher problem coverage (18/176 problems solved) than hint-guided rollouts (row 8) that solves only 1 11/176 problems. However, the two methods are complementary: conditioning on both hints and interventions (row 10) yields the best overall coverage (25/176 problems solved). This result is perhaps natural: hints guide the model toward promising initial direction, while interventions enable fine-grained credit assignment when errors arise in the middle rollout.
Ablation: Reference solutions improve effectiveness of interventions. We next isolate the role of reference solutions in generating effective interventions. In particular, we test the alternative hypothesis that the verification generation asymmetry of the base model alone (Setlur et al.) 2025a) is sufficient to produce high-quality interventions. To do so, we evaluate an ablation shown in Table 4 row 7, where we prompt the model propose intervention given only incorrect reasoning
18

---

Published as a conference paper at ICLR 2025
trace, without the reference solution. On the one hand, conditioning interventions outperforms the naive baselines (rows 1-3), doubling accuracy and increasing coverage by 7, supporting the without privileged information from reference
claim that effective interventions solutions. On the other hand, interventions generated in this manner are significantly weaker than those generated with reference solutions (row 4), which clearly highlights the critical role of reference solutions in producing high-quality interventions.
be yielded
blation: cnstructioo-folloiing capabilities of the base model are critical. We find that the instraction-following ability of the base model is a key determinant of intervention effectiveness. In row 5 of Table 4 we evaluate Qwen3-4B, a model optimized primarily for reasoning rather than instruction-folloinng Despite its strong performance on math reasoning benchmarks, this model frequently fails to adhere the instructions required for intervention generation. For ex- ample, it may switch to directly solving the problem midway through comparing the reasoning trace with the reference solution, or neglect enclose proposed interventions within the required tags. Such instruction-followwe
failures prevent the model from producing usable interventions, resulting in lower coverage (73 out of 334, row 5) compared self- ceff-generated interventions from the instruction-tuneds model (80 out of 334, row 4). These results demonstrate that effective intervention generation depends not only on raw reasoning ability, but also robust iesrucctionfollowiiig abilities.
More capable base models propose better interventions. Finally, we study how the size of the model used propose interventions affects performance. In addition to using the base model, Bwee3-4B-Instruct-222
Ablation:
which also generates the original reasoning traces, we experiment with (row 6), generate interventions.
larger model from the same family, Dwwem3-30B-A3B-nnstuut Using the larger model yields an approximately 2x improvement in accuracy and solves 21 additional problems out of 334, indicating that the quality of generated interventions scales with model capacity.
19

---

Published as a conference paper at ICLR 2025
D PROMPTS
Box I: Prompt for intervention self-generation
You are an expert mathematician teaching a Math Olympiad class. You will be given problem and high-level reference solution to the problem. Your task is to solve the problem step-by-step guided by the high-level reference solution.
Problem {insert problem}
High-Leevel Reference Solution
(insert reference solution)
Great job! Now, student in your class has solved the problem incorrectly. You must leverage your understanding the reference solution to help him correct his attempt at the problem.
Detailed Instructions
Detailed Verification Log
You must perform step-by-step check the student's attempt against the reference solution. The steps here refer to each numbered substep in the Incorrect Student Attempt (i.e., Substep 1: Substep 2:, ..... and not the high-level steps (e.g., Step 1:) which may contain multiple numbered substeps.
This analysis will be presented a Detailed Verification Log, where you justify your assessment of each substep:
For correct substeps, brief justification suffices.
For substeps with errors gaps, you provide detailed explanation.
Be careful and check every intermediate result, they are very easy miss.
2. Identify the First Critical Error
For each issue in the detailed verification log, you MUST determine whether it is critical error. A critical error must pass the following two checks:
1. A critical етrоr is either a factual error (e.g.. calculation error like 2+3-6) or logical fallacy (c.g., claiming that A>B, C>D implies A-C>B-D) that disrupts the current line of reasoning. Procedure:
Explain the specific eтror.
State explicitly that it invalidates the current line of reasoning.
20

---

Published as a conference paper at ICLR 2025
2. A critical error must not be recovered from.
Procedure:
Double-check that the error is not corrected disputed in later steps (e.g., statements such as "Wait, but let me double-check this claim...") ").
As long as the issue passes both checks above, it is considered a critical error. We are interested in the first critical emor that the student makes.
3. Propose an Intervention Step
After finding the critical error, you should propose intervention step that will be inserted before the occurrence the critical error to steer the student towards the correct solution.
The intervention must not give away the exact answer.
The student should still be able to learn from the error.
You should provide guidance by sketching the next steps that lead toward the correct solution.
4. Output Format
Your response MUST structured into three main sections:
1. Detailed Verification Log
2. Critical Error Report
3. Intervention Step
4.1 Detailed Verification Log
Provide the full, step-by-step verification log as defined above, structured in bullet points. When referring the student's attempt or the reference solution, quote the relevant text before providing analysis.
4.2 Critical Error Report
In this report, include a bulleted list summarizing every issue discovered. For each issue, include:
Location: direct quote of the key phrase or equation where the issue occurs.
Issue: A brief description of the problem and whether it is Critical Error.
Stop once the first critical error has been found.
21

---

Published as a conference paper at ICLR 2025
4.3 Intervention Step
Provide a single intervention step to be inserted before the critical error. The content must be written from the student's ss perspective. If there are critical errors, leave this section empty.
Draft of the Intervention Step Format
1. Content: The text of the intervention step.
2. Location:
The substep number where should be inserted, including a quote from the student's attempt.
Self-check
Answer each question below with:
A brief explanation, and
A final verdict: Yes or No.
Does the content of the intervention step include the exact answer from the reference solution (i.e.. "
2. Is the content of the intervention step missing any key insights necessary to solve the problem?
3. Is the location of the intervention step later than the substep containing the first critical error?
If the answer to any question is "Yes", the draft has failed the self-check. Revise and repeat until all answers are "No".
If all answers are "No", write the final intervention step using the format below.
Final Intervention Step Format
INSERT STEP CONTENT should be the content of the intervention step, and IN- SERT,STER NUMBER SERT. should be integer indicating the insertion substep.
<intervention>
{I "content": INSERT STEP CONTENT, "location": INSERT STEP NUMBER }}
</interventio>>
Incorrect Student Attempt
(student solution]
22

---

Published as a conference paper at ICLR 2025
Box 2: Prompt for Gemini intervention
{insert problem}
(insert reference solution)
You have solved the problem correctly. Now, a student in your class has attempted the same problem. Your task now to go over his solution step-by-step and write down detailed verification log, identify the first critical error, and suggest locations in his solution to insert a replacement step such that he follows the replacement step, will guide him away from the error. Detailed instructions are listed below.
Detailed Instructions
1. Detailed Verification Log
You must perform step-by-step check of the entire solution. This analysis will be presented in Detailed Verification Log, where you justify your assessment of cach step:
For correct steps, brief justification suffices.
For steps with errors or gaps, you must provide a detailed explanation.
Please be careful and check every intermediate result, they are very easy to miss.
Identify the First Critical Error
For each issue in the detailed verification log, you MUST determine whether a critical error. A critical emoor must pass the following two checks:
1. A critical error is either a factual error (e.g., calculation error like 2+3=6) or logical fallacy (e.g., claiming that A>B, C>D implies A-C>B-D) that disrupts the current line of reasoning. Procedure:
Explain the specific error.
State explicitly that it invalidates the current line of reasoning.
2. A critical error must not be recovered from.
Procedure:
Double-check that the error not corrected or disputed in later steps (e.g., statements such as "Wait, but let me double-check this claim...") ").
As long the issue passes both checks above, it is considered critical error. We interested in the first critical error that the student makes.
3. Propose Replacement Steps
After finding the critical error, you must now identify existing steps in the student's solution that you can rephrase such that if the student were to begin from your rewritten step, he will be guided away from the critical error.
23

---

Published as a conference paper at ICLR 2025
Note that replacement steps can occur either:
Before the error, cireumvent it completely,
After the error, to recognize the mistake, dispute it, and recover with correct argument.
There may be multiple valid replacement locations. Do not omit locations simply because they close each other. An entire region (e.g., Step X-Y) may be valid, and you should include each step in the region.
4. Output Format
Your response MUST structured into three main sections:
1. Detailed Verification Log
2. Critical Error Report
3. Replacement Steps List
4.1 Detailed Verification Log
Provide the full, step-by-step verification log as defined above, structured in bullet points. When referring specific part of the solution, quote the relevant text before providing your analysis.
4.2 Critical Error Report
In this report, include bulleted list summarizing every issue discovered. For each issue, include:
Location: direct quote of the key phrase or equation where the issue occurs.
Issue: A brief description of the problem and whether is Critical Error.
Stop once the first critical error has been found.
4.3 Replacement Steps List
Summarize all potential recovery locations and replacement steps. Write the steps from the student's perspective. The student should be able to continue naturally from your rewritten step.
Finally, include a curated list of replacement steps to be processed by parser. This list must strictly follow the format below. Do not include any additional explanations or justifications, or the parser will fail.
Format:
<intervention>
"location": INSERT.STEP STEP NUMBER, "content": INSERT.STEP }.
"location": INSERT STEP NUMBER, "content": INSERT STEP CONTENT)
</inervention>>
24

---

Published as a conference paper at ICLR 2025
If the student's 's solution is correct and contains no critical errors, leave the list empty.
Student Solution
{insert student solution]
25

---

Published as a conference paper at ICLR 2025
Box 3: Sample output for intervention generation
Detailed Verification Log
Substep 1: "We are given an infinite sequence of integers do, d1, a2,... that is not eventually periodic..."
/ Correct. This sets up the problem accurately.
Substep 2:
Defines S the set of 2025-tuples (Ax+1,-... an+2025) for 0.
/ Correct. Matches the problem statement.
Substep 3:
"We are to find the minimum number of distinct elements in S"
/ Correct. The goal is clear.
Substep 5-7: Understanding the problem and the non-eventually periiodi condition. / Correct. The student correctly identifies the constraint.
Key insight: if S is finite, then the sequence is eventually periodic. X Critical error in reasoning- this claim is false in general. The student incor- rectly assumes that finite number of 2025-tuples implies eventual periodicity. This is not for sequences over infinite alphabet like even if the block length is fixed.
Substep 10-14:
However, the student's reasoning in Substep 25-34 is flawed. They state:
"If the set of 2025-tuples is finite, then the sequence is eventually periodic."
This not valid conclusion general. For example, consider a sequence where the values grow without bound but the blocks of length 2025 repeat in non-periodic fashion but such thing is not possible due to the structure of sequences. However, the critical flaw is that the student assumes that finite number of 2025-tuples implies eventual periodicity without justification, and this is only true in the case of finite alphabets under stronger conditions.
But more importantly, the student fails to recognize that the sequence can have finite alphabet (c.g.. binary) and still be non-eventually periodic (like the Fibonacci word), while having exactly 1: + distinct blocks of length k.
The student's s assumption that finite S implies eventual periodicity logical fallacy that invalidates the entire line of reasoning.
Substep 15-142: The student argues that because cannot be finite (due to eventual periodicity), must be infinite, and thus the minimum number of elements is infinite.
This is incorrect because:
The claim that finite S + eventually periodic not universally true over infinite alphabets.
In fact, over finite alphabet, sequence can be non-eventually periodic and have exactly & + distinct blocks of length (e.g., Sturmian sequences).
26

---

Published as a conference paper at ICLR 2025
The student ignores the possibility of finite-alphabet, non-eventually-periie sequences with finitely many 2025-tuples - which is exactly what the refer- ence solution shows is possible.
Therefore, the student's conclusion that |S| is invalid.
Critical Error Report
Location:
"If the set S is finite, then the sequence is eventually periodic." (Substep 14)
This is factual and logical error. The student incorrectly assumes that finite number of 2025-tuples implies eventual periodicity. This is faase in general for sequences over infinite alphabet (like Z), and even more so when the values can grow without bound.
Issue:
However, the key point is that this claim does not hold in general. For example, sequence over a finite alphabet (e.g., {0, 13) can be non-eventually periodic and still have only finitely many blocks of length A: - specifically, exactly k + 1 as Fibonacci word.
This a critical error because:
1. is a logical fallacy it invalidates the reasoning that S must infinite.
2. is not recoverable - the student never revisits or corrects this assumption. Later steps (e.g., Substep 145) accept oo the answer based on this flawed premise.
Therefore, this is critical error.
Intervention Step
Draft of the intervention step:
1. Content:
Consider sequence that takes only finitely many values (e.g., only and 1). Even if the sequence is not eventually periodic, it might still have only finitely many distinct 2025-tuples. For example, the Fibonacci word over {0.1) is not eventually periodic, but every block of length & appears exactly once in a certain pattern in fact, it has exactly i: distinct blocks of length Can you think of way to construct such sequence? Try to understand how the number of distinct blocks relates to the periodicity of the sequence.
2. Location: Substep 14
27

---

Published as a conference paper at ICLR 2025
Self-check:
1. Does the content include the exact answer from the reference solution (i.e., "2026")?
No. It avoids giving the answer and instead prompts the student to explore finite-alphabet, con-eventually-periodde sequences.
2. Is the content missing any key insights that are necessary to solve the problem?
- No. It introduces the idea of finite alphabets and the possibility of finite block weveentalll-periieee sequences, which essential.
complexity
3. Is the location of the intervention step later than the substep number that contains the first critical error?
- No. The critical error occurs in Substep 14, and we are inserting the intervention
Substep 14. This is correct it comes before the erroneous conclusion.
/ All checks pass.
Final intervention step:
<intervention>
"content": "Consider sequence that takes only finitely many values (e.g., only 0 and 1). Even if the sequence is not eventually periodic, it might still have only finitely many distinct 2025-tuples. For example, the Fibonacci word over [0 is not eventually periodic, but every block of length $ appears exactly once in a certain pattern fact, it has exactly $ k+1 $ distinct blocks of length $ $. Can you think of way construct such sequence? Try understand how the number of distinct blocks relates to the periodicity of the sequence.", "location"; 14 </intervention>
in
28

---

Published as a conference paper at ICLR 2025
E INTERVENTION EXAMPLES
In this section, we provide a few concrete examples of interventions.
Prublem Statemment: LCM and Sum Constraint
If 8,6, and d are pairwise distinet posstive integers that satiddy cmm(a,b.,, 1000 and & d, compute the largest
poesible value 4
Stepe 241-243: Shared Context
Step 16-9-
over. Step 243: Walt keep gesting LCM 1000.
Guidaace
LCMs 1000, with zumbere exponente primo. combining asuraption
Steps 244-247: Student's Wrong Path
princs lurgr mar shored largo com-
might woing What mon foctor, say ge The mumbers would "g"a", x"""". "gtc", 'gta", The LCM woull be ' lenfal, bi c. d'y- If Himmss bi c: d'' is very small. "a" could be quite large. and the sum 19' could large too. Let me exploce this structural appreech.
Step 244: What if we use only numbers that are powers
Step 245: Try numbers:
1,2.3.4,6,,,911,,,, 26,,,,,,,,,7,,888 27,32,36,,,,, 68, 11,,,88,,,,,,,111111 16,1088,24, 128, 21,,,19,,,,,2222222 25,,222,54622,,2222444,,,6
Stope 206-247) Post Correction
Stop Step
Final Answer: Studees Gets Wrong
Final Anawer: Modal Geta Correct
270
Figure 13: Example error of Qwen3-4B-Instruct and generated intervention.
29

---

Published as a conference paper at ICLR 2025
Probiem Statensent: Regriar Polyigon with Equilatoral Triangle
Poist to extecior the regilar mekiee polyge: As Ar Any and As equilsttral trikngle. What the larges valle which As, Auy and are oxneecttive vetions regalar pelygon?
Steps 38-40: Shared Context
Step We
Step 39: A
arg
w- 221//
erg - 23/nn- B = (w 1ie
Step 40: We want At, An, 13 to be *consccuttvve ver- ticee** of regular polygon.
Sleps 41-44: Student's Wrong Path
Step di: IIT
Guidance
Step 41: So the angular differences between them must
So the distazess between cansccttvve points must be ecual. Let's consider the prosible urdzings the order Au, A, R. thin must have AA, J4: Bi. Lat's colenaat these lengtii and concition holds check
equal.
Step 42:
That is:
arg(An) arg(As) arg((D) arg(A.)
Let's is demote = arg(B) - arg(An)
Step 43: her:
Staps 43-18:
Ports Connection
ar,(A.,) arg(-A,)
Step Seep gntmatry
arg( - A..)
Strppoe
Soc
ang (B) = 2arg - arg(As))
Step 4d: Sinne arg(AA) arg( /T,
arg( 41//12
Final Answer: Studmt Gets Wrong
Final Answer Model Gets Correct
Figure 14: Example enror of Qwent--4Bllnsrruct and generated intervention.
30

---

Published as a conference paper at ICLR 2025
Problem Statement:
bubble through distiet mol mambers be sccending order by
"bubelo pouare"
the second with
comparing comparing:
third
example eequemor 7, trbble paee. The mumbers compared step underinnt. 1 S.S. 2 sequzence rn, Pa... Fro dlettnce one andt her and
seguenoe
7 Suppron that (t) and that ace in rondom order Lct p/o. in lowest tecuss, be the probobility that the sumber that begias as fax will end up, after oue bubile s0
Steps 241-243:
Shared Contaxt
Step 241: Sa, stay or move to position 26, it must not. sma lor than any the elements its left. Step 242: In particular, to **end
position 30**, it
muss:
Step 243: 1. *Not be smaller than any the elements positions 1 through 19 otherwise it will be pushed left.
Step Gridoace
zeper postiion through
Steps 244-247: Student's Wrong Path
Step Whas if only numbers powers
and 3?
Step 245: Try sumberw:
Stope Si5-5sh Pest-Currccion
1,2,3,4.6,8, 12.10,,,,,,,7,22222. 34,6,,66,,,55,4,,77,8,,, 96,,7,2,,1,8,1,11111. 144, 192, 216,2216,6666 29,44,,545,,9444444555,,,
Step 24tr War. IT lage tso na. will rove
Stop 247
Fizal Amswer: Studenn Gata Wrong
Final Amswer: Model Gets Correct
11000049
Figure 15: Example error of Qwen3 -Bn 4B-Instruc and generated intervention.
31

---

Published as a conference paper at ICLR 2025
Probium Statement: Suppose Haroord Yard is It 1i..
Suppose Harrond Yard is 17 1 equare. Thhre ace 14 doems located sc the pertmetee of the Yard, If the minimum
distance betwees docus, the moodmica poesilte volue of can be oxpressed as v3 wioce 6.0 are positive insees..
Compute 100s
Steps 210-212: Shared Context
Step 210: So need d
Step 211: Distance between point on cne side and
point adjacent side.
Step 212: For example, point at (z,0) on bottom and point at (1 right.
Step
Now, la vomasdrr the dastsnce beissen durta acusarn sdla, wlach cross unocupped coracr Lile (IT, 117.0) The lur doem the bottom aide at 65. By systimetry. flss. domn on the rigtt dide mas be jusced rtlative to some
an
Steps 01 Student's is Wrong Path
(0.0).
Step 213:
Distance:
v(7 2/2 0)
adjiomt domee (1x,0) (1T, 17 Sa).
(17, therrtee
<(17 4
Step 214: Similarly, between point (r. 0) and point at
(17, d) right: distance v(7 x)a
Step 2.55: want to maximioe the minimol distance be- sween any two polats.
Step By symmetay, the minimal distanee be minimun of:
Seep 210 siellaityy the ditaiee banver adjoret poizas
Final Anewer Studkot Gets Wrong
Final Azeweri Model Gete Correct
Figure 16: Example error Owen3-4BBIIntreee and generated intervention.
32

---

Published as a conference paper at ICLR 2025
F EVALUATION CONFIGURATION
For experiments with Swen3-44-Instrueee we follow the official recommended configuration, using temperature 0.7, top-p 0.8, and top-k 20 unless specified otherwise.
G TRAINING HYPERPARAMETERS FOR INT
Hyperparameter
Dwen3-4B-Instruut-25000
effective_train wwt.rriianaccsiiii..........
32
ppo mini mio. batch size
16
learning rate
1.0e-6
kl_loss _coe_ entropy_comm .coeff temperature top-p rollout.n
0
1.0
1.0
8
Table 5: PipelineRL
Piché et al. (2025
hyperparameters used for RL runs.
Hyperparameter
BWen3-4B-Instrueee
dataset_size effective.batch.sie
1076
64
num commtrain.epochs
4
learning rate Ir scheduler type min Ir rate
1.0e-6
cosine with min
0.1
warmup_ratio
0.1
Table 6: LLaMa Factory Sheng et al. 2024
hyperparameters used for SFT runs.
33

---

Published as a conference paper at ICLR 2025
H SELF-REFLECTION BASELINE
Self-reflection. We add details on how we establish the self-reflection baseline. The idea similar to generating interventions, but rather than outputting single-step interventions, ask the base model to re-write the entire solution.
We find that method InT consistently outperforms the self-reflection baseline, as shown in Figure 12 and Table 3 The prompt for generating self-reflection traces is shown below. The reference solution is a high-level summary of the solution written by humans, and occasionally, a stronger model. Similar to InT, we ask the base model to generate its own self-reflections.
Box 4: Prompt for self-reflection
You are an expert mathematician teaching a Math Olympiad class. You will be given a problem and a high-level reference solution the problem. Your task solve the problem step-by-step guided by the high-level reference solution.
Problem
[Insert Problem}
High- Reference Solution #
[Insert Reference Solution)
[Insert Model Response]
Great job! Now, student in your class has solved the problem incorrectly. You must leverage your understanding of the reference solution to rewrite a refined version of his attempt the problem. ** Youu rewritten solution should be complete solution the problem. ** Incorrect Student Attempt # {Insert Student Solution]
I ARE INTERVENTIONS MEMORIZED?
We would like to understand whether InT leads to memorized interventions or it actually learns to generalize to unseen problems. As such, select two example problems from the IMO Shortlist 2024 outside of our training set, and compare the traces InT and base model -rren3-4B-Instruee
IMO Shortlist 2024, Problem C1. As shown in Figure 17 both models started with incorrect assumption of the formula, but when the InT trained model got 3 when = 3, it questioned how this possible and thus correctly updated the hypothesis (?)
IMO Shortlist 2024, Problem C2. Although both models are able to successfully conclude that even cool numbers must be multiples of four Figure [8| only the InT model able to try considering that n 12 may not be cool and therefore the pattern may be more selective than simply being multiple of four. This leads the second model to the right hypothesis.
34

---

Published as a conference paper at ICLR 2025
Studest Seare Stodmizattion (E30 Sacrilar 2014, Prosien C1.)
Iet a a peslive decuf in siaken mel of sled thy colled with en dows Acteiin
fight rezza postine tnieere they oue top place loan maoss Thst the posslld Fire contree (eeall
wore stakees
Hyperkens
Step sax an- Check
Obeerntka:
Maidar Conjetuon Hudsnas sner n-
SiFCCreecos via Counar-Scanpp
Putioes Muieking Ferce
Step (Tharigg dy Tiet's in w Me crgjo rgy I Catealation: Centraciig
Stop Lunkin steple indowesid patson, Hy-
slmetber "hee nn ten in > be rejetted
(IP (Maacia))
Dedisons Aerase de peller ladle willeii Alpmas pecel dhnilnn largr meredly
Post
(Now Hapethank)) Variitatione
CJ
(nseeet)
5-m 12 (plander maead chick)
12
Incoreet termat
Cancllaton: tem []) ""g"".
Figure 17: Diverging Solution Paths between In I' and base models on IMO Shortlist 2024, C1.
Eives Cool Numbber (150 Sharclat 2004. Probiem C2)
Let i be postie langee The lategro 1, sae be wittiin the calle as a bout outh that each kanger bead diikks boadd ouid? acconcllptag adl-ooods
mach that mata cesaazed is
noust ope aatdeanddd
of
i d to the dedi the bead, the siza of lntepeee wittiss eaca ie exflooard not cauitipa of sene
Base Model Solution Path
In'l Model Solution Path
Eatital
Austyaia
Stop cad. e 200 and (pette contudaation ford
and Step
Stop in and the ai party senowen liids odd auulees, Madall (louak) mus odd aveal.
nod
Cittical SetsCoenetion
Step (Mteembaation) Check trediteee of is Thanght
2.5.4. Dualin Sate- trng castitinn to mntrpe diitem Ble in
Macee
Fosh alee corm rivialie he wome cesill for parto filk hes
Ooo.. Poller
taun stoll thete steple Plest "Lac only wak? (Phnoss of onl
avending palse
diralii by Bul party cack. Tamntoos God Nandecs
Revbec Hopotheds Calcutation
Step 22.00
cod canders any poinc
Pioel Chicsistions (lansnvve))
(Flat 201.
Rowal
2000
Figure 18: Diverging Solution Paths between InI and base models on IMO Shortlist 2024, C2.
INT WITH INTERVENTIONS FROM A STRONGER MODEL (GEMINI 2.5 PRO)
While prototyping and developing our method, we also evaluated the efficacy of IntT with interven- tions generated by Gemini 2.5 Pro (referred to as Gemini from now on). In particular, we answer the following questions: (1) does SFT on Gemini-generated interventions improve the ability of the fine-tuned model to sample correct traces on hard problems? and (2) how does InT compare with distillation of full expert reasoning traces sampled from the oracle (i.e., Gemini 2.5 Pro)? To this end, we run several experiments comparing InT against running standard RL training and distillation on Gemini-generated solutions, in an attempt patch the capabilities of 3-1.7B (Setlur al.) 2025b): a strong, open-source reasoning LLM fine-tuned on top of Qwen3-1.775 - on a set of difficult math reasoning problems.
J.1 INFERENCE- ONLY RESULTS
35

---

Published as a conference paper at ICLR 2025
InT + RL
Distill + RL
Base + RL
Comparison
0.25
0.25
0.25
Pass@k
0.12
0.12
0.12
pass
0.12
0.06
50
0.06
50
0.06
50
0.06
Base RL
100
100
100
Distill RL
0.03
0.03
0.03
150
0.03
4
8
16
32
64
4
8
16
32
64
4
8
16
32
64
4
8
16
32 64
Figure 19: Pass@ across RL training iterations: We plot pass & performance from 0 to 150 RL itcrations for three initializations: (i) base model patched with InT; (ii) base model distilled on oracle traces; and (iii) directly the base model. Int patched model improves pass @k consistently while others mainly sharpen.
We provide inference-tinge time results achieved by rolling out from generated interven- tions concatenated the prefixes. experiments on different subset of 235 DeepScaleR problems that have pass 32 O. Our ear- lier prompt uses Gemini solutions in place of the reference solutions and does not double-check the generated intervention (shown Box 3). In this earlier setting, we obtained a 3.8x improvement in accuracy and solve 47 more problems compared to naive rollouts. Moreover, we also conducted an additional ablation "from intervention" which perform rollouts from the location of the mistake but is not conditioned on the intervention (i.e., T x, y<t- )) to isolate for the benefits of localizing the mistake. We find that pinpointing the location of the mistake alone leads to superior gains in accuracy (2.4x) and coverage (+ 22 problems compared to the baseline).
Interveation Proposed
Rollonts Conditioned
Unique
ScudeR subret N/A
probeas
235
0.97%
pribiem
prefit your
120 235
2.97%
profis your intervattion yor
145/235
Table 7: Rollouts conditioned on differently generated interventions. As shown, rollouts conditioned on interventions double the number of problems with at least one correct rollout, and improve the rollout accuracy by more than order of magnitude.
J.2 EXPERIMENTAL SETUP AND EVALUATION METRICS
Constructing dataset of hard training problems. We begin experiments with state-of-the-an <2B parameter model, 331.BB (Setlur et al. 2025b), already trained with curricula and several best practices for RL to attain strong performance in its scale. Despite its strong performance, this model still fails on a large fraction of problems from its hard training set (a 2.5K subset of DeepScaleR each problem and collect the subset of
problems from Luo al. (2025)) ). We run 32 rollouts problems the model cannot solve at all. We utilize Gemini 2.5 Pro (as of 2025-08-01) as our oracle. Among these 472 unsolved problems, the oracle solves 16% of them in single attempt, suggesting can provide meaningful interventions on these problems. We retain this subset as our hard problem set Phard study the efficacy patching with different methods. Our main findings are that (i) RL with just a small dataset of 64 problems on top of InT outperforms RL on a much larger set of 1.2K problems on top of distillation or the base model. On the other hand, (ii) RL with the small dataset on top of the distillation and the base model are infeasible due to collapse of behaviors on OOD sets or zero learning signals.
Baselines approaches and comparisons. To evaluate the efficacy of InT, we compare against alternate approaches for patching model behavior on Dhard- Our primary comparisons are: 1) "Distillation + RL," which first distills entire reference solutions into the base model before running RL, and 2) "Standard RL," which directly continues RL on the hard problem set from the same base checkpoint. Both simulate continued RL run where new hard problems are introduced during training. We also consider SFT-only baselines, where the model is patched via supervised learning on reference solutions intervention traces for the hard problems, without any further RL. To our knowledge, no existing method is designed to explicitly handle this setting of patching model behavior on previously unsolved hard problems in a way that leverages oracle interventions while preserving the benefits of RL. Therefore, we limit our exposition of training trends to Dpard but also compare with alternate approaches for using intervention data on holdout standardized test sets.
Evaluation metrics. Prior work primarily evaluates RL-trrined reasoning models on competition math benchmarks such as AIME2025 and HMMT2025. However, progress on these alone does
36

---

Published as a conference paper at ICLR 2025
RL
Zero adv. ratio
Distillation
R
0.8
Reward
0.2
InT RL
Pass@1
0.2
0.6
0.1
Distill RL
Distillation
0.1
InT RL
0.4
InT RL
0.0
100
100 Gradient steps
50
150
100 Gradient steps
50
150
Gradient steps
(a)
(b)
(c)
Figure 20: Comparison of InT with distillation of reference solutions. (a) Since these are hard problems, rumning RL initialized from the base model does not improve training reward, while running RL on top of the distilled model or the model produced by InT does improve training reward. We observe that running RL on top of the distilled model degrades model capability (decreasing pass@ score held-out (b) with more training), even though it continues to make progress on the training set, as shown by a decreasing ratio of the percentage of unsolved problems ("zero advantage ratio") in (c).
not capture whether models actually learning from hard training problems, nor whether such training transfers to equally challenging evaluation problems. To address this, we evaluate our trained models on several standardized benchmarks from 2025, including AIME2025, HMMT2025, BRUMO2025, CMIMC2025, and others, as well as an in in-distribution (i.i.d)) test set of hard problems Dhard similar P The i.i.d. test set consists of 64 problems held out from the training pool using the same methodology as used to select D In addition, we also report performance directly on the training problems to track how RL modifies behavior on seen examples. Across all three settings and standardized benchmarks, we report results at output length of 32,768 tokens.
J.3 INT UNIFORMLY PUSHES THE PASS @ FRONTIER UPWARDS ON TEST PROBLEMS
We present our main results for InT on an holdout set of hard problems Diest (Fig 19 Here, we plot the pass@ k performance across different RL training iterations from to 150, for three models: (i) base e3-1.7B, (ii) e3-1.70 distilled full reference solutions; and (iii) e3-1.7B patched on interventions from the oracle (InT). We find that running RL on the base or distilled model does not make any improvements in pass @ k throughout all training steps. On the other hand, running RL on Daard after we patch 1.BB on oracle interventions (InT) leads to consistent improvements in pass @ k during RI.
J.4 INT OUTPERFORMS DISTILLATION STANDARDIZED EVALUATIONS
OlymMATH
OlymMATH Hard
Model
RL Data Size
Easy
HMMT
BRUMO
e3-1.7B + RI.
1216
38.75
6.75
22.50
46.25
c3-1.7B Distill + RL
1216
37.38
5,75
22.50
47.08
c3-1.7B InI+ RL
64
41.62
7,50
24.58
$3.75
Model
AIME
Beyond AIME
CMIMC
Average
Disi pass @
c3-1.7B +RL
36.25
20.88
23.75
27.73
15.85
Distill
36.67
21.75
c3-1.7B
InT+ RL
36.25
22.00
21.88
29.65
23.56
Table 8: Pass@1 I performance (8 rollouts avg.) of models across standard mathematics benchmarks and @ performance the i.idd test set, Dim Observe that InT followed by RL attains the highest pass @88 performance on this in-distribution test set patching 13-1.BB model.
Previously, we saw that InT improves pass @ over baselines on training and hold-out sets. This mainly tells us that InT makes progress on the hard training problems that were previously unsolved. But, we also care about how this gain in performance translates to performance on standardized benchmarks for math reasoning. Here, compare the performance of our approach InT top of the c3-1.BB model.
To stress test InT in the setting where we simply continue to run RL from the intervention checkpoint, we run RL training on this checkpoint with only 64 problems D on which we collected the interventions. We compare the performance of this RL trained model with an RL on the distilled and base models. To boost the baselines, we run RL for both using an expanded set of about 1.3k problems sourced from DAPO Yu et al. 2025). including the 64 we used for InT. The main reason
37

---

Published as a conference paper at ICLR 2025
perform this injection is that in preliminary experiments which trained the distilled model only on the small set of 64, we noticed that post RL the model capabilities on standardized evaluations fell drastically (Figure (0gure(0(b)) b)), perhaps due off-policy nature of the reference solution. When RL on the base model, we also expand the training set for RI, since we find that the reward curve does not rise otherwise (Figure 20 a)), thus train the hase model from DAPO and the 64 problems with InT interventions. Unlike the RL runs on distilled and base checkpoints, InT improves the test performance averaged across multiple hard test datasets, despite being trained on just 64 problems (Table 8). Compared to distillation, we see gains on both in-distribution (Dadg) and standardized benchmarks for hard problems mainly because intervention does not alter the base model distribution much as distillation on the entire trace.
mixture of easy problems
38


Reviews:
[{"review_id": "oMlZpjcJ7j", "reviewer": "Reviewer_jykS", "summary": "This paper introduces Interventional Training, a novel framework designed to address credit assignment issues in LLM reasoning during reinforcement learning. The key idea is to leverage single-step oracle interventions (e.g., from a human or another LLM) to correct intermediate errors in model-generated reasoning traces, followed by supervised fine-tuning (SFT) and continued RL. The authors demonstrate that InT improves performance on hard problems where standard RL fails, achieving gains in pass@k metrics and solving previously unsolvable tasks. The contributions include a cost-effective alternative to full trace cloning and a method to patch LLMs without distorting their base distributions.", "questions": "How can the dependence on an external oracle be minimized? For instance, could the base model be trained to self-correct interventions over time?\nWhat are the trade-offs in using different oracle types (e.g., humans vs. LLMs), and how do they impact reproducibility?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "InT creatively uses localized oracle interventions to patch errors, avoiding the pitfalls of full trace distillation.\nExperimental results show consistent improvements in pass@k and problem-solving rates, with ablations validating design choices.", "weaknesses": "The primary weakness is the reliance on an external oracle model (e.g., Gemini 2.5 Pro) for interventions. This introduces practical constraints, such as the cost and availability of high-performance oracles, and potential biases if the oracle's capabilities do not generalize.\nWhile InT reduces data-writing burden compared to full traces, it still requires oracle access, which may not be feasible for all practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MQDulBuzvf", "reviewer": "Reviewer_ufkp", "summary": "This work tackles a highly valuable problem, the failure of reinforcement learning (RL) to improve large language models (LLMs) on “hard” reasoning tasks where no correct rollouts are sampled, resulting in zero reward and stalled learning. The authors propose Interventional Training (InT), which introduces targeted external interventions at the first failure point in a model’s reasoning chain. By forcing a corrected trajectory and continuing RL from this modified trace, the method effectively injects learning signals into regions where RL previously could not make progress.", "questions": "Please refer to the concerns in Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method directly tackles the zero-reward problem in RL for reasoning models, enabling continued learning even beyond the model’s existing competence boundary.\n2. By identifying and correcting the first erroneous step in the reasoning chain, InT provides much finer-grained credit assignment than standard RL, improving learning efficiency and self-correction capability.", "weaknesses": "1. The core idea is not new — applying single-step interventions at failure points has already been explored in other RL and imitation-learning domains. The paper mainly transfers this known concept to LLM reasoning without introducing new techniques.\n2. The method depends on access to ground-truth answers and a strong evaluation model to identify and correct errors, raising concerns about scalability and whether such oracle-dependent training has an inherent upper bound on achievable improvement.\n3. The evaluation primarily compares InT against ablated variants such as Distillation + RL and Standard RL, rather than against more conceptually related methods like process-reward modeling or critique-based self-refinement, which limits the persuasiveness of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9IVX2gbxuD", "reviewer": "Reviewer_ikPh", "summary": "This work is based on the observation that RL is difficult to train and converge when the initial success rate is very low. To address this issue, the paper proposes an Interventional Training (InT) framework that replaces challenging reasoning steps with oracle interventions. The base model is then tuned using these oracle interventions. Experimental results demonstrate the effectiveness of this approach.", "questions": "1. How do the authors identify where oracle interventions should be pinpointed? How to control the length of each generated intervention?\n\n2. Why does the distilled model perform worse than the base model? Does this suggest that the teacher model lacks sufficient reasoning capability?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well organized and clearly written.\n2. The motivation is well-defined. The authors also discuss other approaches they have tried and provide reasonable explanations for their limitations, which makes the proposed framework convincing.\n3.  Experimental results support their findings.", "weaknesses": "1. Literature review is not sufficient. There is extensive prior work on improving RL performance using oracle guidance or hints. However, this paper does not adequately discuss or compare with such related works. For example, recent works have explored combining RL and SFT to enhance RL performance.\n2. Although the authors repeatedly mention credit assignment, the explicit connection between InT framework and credit assignment is unclear.\n4. Experimental validations are not sufficient. The experiments are conducted with only one base model and one oracle model. It would be helpful to include results across multiple model pairs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4iSbJmAq0G", "reviewer": "Reviewer_MBkX", "summary": "This paper introduces Interventional Training (InT), a method to improve reinforcement learning (RL) training of LLMs on hard reasoning problems where the model consistently fails to generate correct rollouts (i.e., receives zero rewards). The key idea is to perform targeted credit assignment by identifying and correcting a single intermediate reasoning step where the model goes wrong, using a lightweight oracle (e.g., a stronger LLM). The corrected step is then used to patch the model via supervised fine-tuning (SFT), enabling it to generate counterfactual correct traces and resume RL training.\n\nKey contributions:\n1. A method to collect single-step interventions from an oracle to correct reasoning errors;\n2. Fine-tune the model on its own prefix + oracle correction, avoiding full trace cloning;\n3. Resume RL from the patched model, now capable of receiving non-zero rewards on previously unsolvable problems;\n4. InT solves 14–16% more hard problems and improves pass@k on both in-distribution and standardized math benchmarks.", "questions": "Q1: How does InT perform with weaker or human oracles? Can the method still work if the oracle is not significantly stronger than the base model?\n\nQ2: What happens if the oracle misidentifies the error or provides a suboptimal fix? Is there a mechanism to filter or validate interventions?\n\nQ3: Does the model learn to self-correct, or does it just memorize interventions? How does it behave on similar but unseen problems?\n\nQ4: Can InT be scaled to larger problem sets or more diverse domains? What are the computational or stability limits of continued RL after patching?\n\nQ5: How does InT compare to process reward models (PRMs) or iterative self-refinement? Would training a PRM be more efficient or effective in the long run?\n\nQ6: Why not train the model to predict its own interventions? Could InT be extended to automate the oracle role via meta-learning or self-critique?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1: This paper addresses a real and underexplored problem—RL stalling on zero-reward problems due to execution errors in long reasoning traces.\n\nS2: InT is lightweight, requiring only ~100 short interventions, and avoids the cost of full oracle traces or dense reward models.\n\nS3: This paper demonstrates consistent improvements over baselines (e.g., distillation, standard RL) on both training and test sets across multiple benchmarks.\n\nS4: Unlike full trace distillation, InT retains model diversity and avoids catastrophic forgetting by only patching on-policy prefixes.\n\nS5: This paper connects to credit assignment and on-policy vs. off-policy learning, with clear ablations and training dynamics analysis.", "weaknesses": "W1: This work requires access to a stronger oracle model (e.g., Gemini 2.5 Pro), which may not be available or practical in many settings. The paper does not explore weaker or human oracles or automated error detection.\n\nW2: How to define a step and how to determine whether the split step is reasonable? What is the maximum number of steps after segmentation in this work?\n\nW3: This work was evaluated only on math reasoning tasks with verifiable answers. Generalization to open-ended reasoning, symbolic tasks, or subjective domains is unclear.\n\nW4: RL is only run on 64 hard problems, which raises questions about scalability and stability of the method when applied to larger or more diverse datasets.\n\nW5: While the paper shows where errors occur, it does not analyze what types of errors are fixed, how often the model recovers, or whether it learns to generalize from interventions.\n\nW6: The main baselines are standard RL and full trace distillation, but the paper does not compare with process reward models (PRMs), self-correction, or iterative refinement methods, which are more closely related.\n\nW7: In addition, there are many typos, such as blank pages (page 13, page 17), and incorrect references (Table 3, Table 4, Table5)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)