Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
Authors: Yihe Deng
I-Hung Hsu
Jun Yan
Zifeng Wang
Rujun Han
Gufeng Zhang
Yanfei Chen
Wei Wang
Tomas Pfister
Chen-Yu Lee
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning
Reasoning
Large Language Model
Agent
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) often struggle with problems that require multi-step reasoning. For small-scale open-source models, Reinforcement Learning with Verifiable Rewards (RLVR) fails when correct solutions are rarely sampled even after many attempts, while Supervised Fine-Tuning (SFT) tends to overfit long demonstrations through rigid token-by-token imitation. To address this gap, we propose Supervised Reinforcement Learning (SRL), a framework that reformulates problem solving as generating a sequence of logical ``actions''. SRL trains the model to generate an internal reasoning monologue before committing to each action. It provides smoother rewards based on the similarity between the model's actions and expert actions extracted from the SFT dataset in a step-wise manner. This supervision offers richer learning signals even when all rollouts are incorrect, while encouraging flexible reasoning guided by expert demonstrations. As a result, SRL enables small models to learn challenging problems previously unlearnable by SFT or RLVR. Moreover, initializing training with SRL before refining with RLVR yields the strongest overall performance. Beyond reasoning benchmarks, SRL generalizes effectively to agentic software engineering tasks, establishing it as a robust and versatile training framework for reasoning-oriented LLMs.

Published as a conference paper at ICLR 2026
SUPERVISED REINFORCEMENT LEARNING: FROM Ex- PERT TRAJECTORIES TO STEP-WISE REASONING
Yihe Deng2+t
I-Hung Hsu Jun Yan' Zifeng Wang' Rujun Han' Gufeng Zhang?
Yanfei Chen
Wei Wang²
Tomas Pfister Chen- Yu Leel
Google Cloud Al Research UCLA Googge Cloud
ABSTRACT
Large Language Models (LLMs) often struggle with problems that require multi- step reasoning. For scall-scale open-source models, Reinforcement Learning with Verifiable Rewards (RLVR) fails when correct solutions are rarely sampled even after many attempts, while Supervised Fine- Tuninn (SFT) tends to overfit long demonstrations through rigid token-by- eaken-by-tokee by-token imitation. To address this gap, propose Supervised Reinforcement Learning (SRL), a framework that reformulates problem solving as generating a sequence of logical "actions". SRL trains the model to generate an internal reasoning monologue before committing to each action. provides smoother rewards based on the similarity between the model's 's actions and expert actions extracted from the SFT dataset in a step-wise manner. This supervision offers richer learning signals even when all rollouts are incorrect, while encouraging flexible reasoning guided by expert demonstrations. As result, SRL enables small models to learn challenging problems previously unlearnable by SFT or RLVR. Moreover, initializing training with SRL before refining with RLVR yields the strongest overall performance. Beyond reasoning benchmarks, SRL generalizes effectively to agentic software engineering tasks, establishing it as a robust and versatile training framework for reasoning- oriented LLMs.
AMC23
AIME24
AIME25
Minerva
Average
60
Model
Owcn2.5
50
SFT ressoning)
RLIVR)
SPT (outline) >x HUVI))
30
SAL RLVR
SRL
20
AMC23
AMC23
AIME24 (Avg8832
AIME24 (Greedy)
AIME25
AIM225
Minerra
Average
(2x
(Greedy)
(Awg6932)
(Greedy)
Figure 1: Performance of our method (SRL) against baselines on math reasoning benchmarks, with all models trained on the challenging slk dataset (Muennighoff al. (025).. Our key observations are: (1) Directly applying SFT on this dataset leads to performance degradation compared to the base model. (2) While RLVR can improve generalization over SFT, the gains are marginal. (3) Our proposed SRL method substantially outperforms these baselines, and the SRL - RLVR pipeline achieves the highest performance, overcoming the challenges of training on difficult data.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable versatility across diverse domains, ranging from mathematical groblem-solving (Wang et al. 2025) and search-resule integration Jin
Equal contribution. This work was done while the author interned at Google Cloud Al Research Correspondence; yihedeng @g. gul.. ihung. gmail.com. Picnyulee@googlle...

---

Published as a conference paper at ICLR 2026
Hsu et al. 2024 to tool-use Yin al. 2025 Qian et al. 2025 and long- oong-horioon agentic interaction (iee et al. 2024; Li et al. 2025c Tan et al. 2025). A pivotal recent advancement involves
al. 2025
the use of reinforcement learning (RL) to bolster complex reasoning capabilities (Shao al. 2024] Ahmadian et al. 2024 Lambert et al. 2024). By optimizing models through reward signals tied to verifiable outcomes such final answer accuracy RL provides scalable framework for eliciting
sophisticated problem-solving behaviors, including self-reflection (Guo et al. 2025 Xie et al. 2025).
The effectiveness of these outcome-based RL methods fundamentally depends the policy model's ability to discover correct solutions within a limited rollout budget (Brown et al. 2024). However, given practical computational constraints, this learning paradigm struggles on challenging problems from the training data, where the model's s success rate is effectively zero (when the pass @ rate remains zero even after sampling rollouts). Such cases are increasingly common tasks requiring complex, multi-step reasoning (Yue et al. 2025 Wang et al. 2024). For these problems, an incorrect intermediate step can derail the entire reasoning chain for a 7B-scale L.LM, resulting negative learning signals regardless of any partially correct solutions. Furthermore, naively penalizing all incorrect final outputs can further introduce training instability and hinder progress, making these difficult reasoning tasks largely intractable for standard outcome-basee RL methods (Xiong al. 2025).
An alternative approach is imitation learning, commonly implemented via Supervised Fine- Tuning (SFT) on expert demonstrations (Ross al., 201T). While SFT can instill valuable reasoning behaviors, its next-token prediction objective enforces rigid, token-level imitation, limiting the model's ability to generalize beyond the training data. This problem becomes particularly pronounced when training data are modest in scale and when the model itself is relatively less capable. Under such conditions, long and complex demonstrations often lead to overfitting and shallow reasoning behaviors (Chu et al., 2025a; Li et al. (025b), as illustrated by the performance decline in our Figure [ Consequently, both SFT and outcome-baat based RL struggle on challenging reasoning tasks, leaving critical gap for training small source models to effectively learn difficult problems.
To address this gap, we introduce Supervised Reinforcement Learning (SRL), framework that reformulates problem-solving as sequential decision-making process. Rather than optimizing for a final answer or imitating an entire expert trajectory, SRL trains the model to reproduce the sequence of key actions underlying expert reasoning, following an RL-style objective. Specifically, expert demonstrations are decomposed into a series of intermediate actions, each representing a meaningful decision step. During training, the model first generates an internal monologue to articulate its reasoning and then commits to an "action". At every step, SRL provides a reward based on the similarity between the model's predicted action and the corresponding expert action, thereby providing fine-grained, efficiently computable supervision that scales to large datasets.
Our work makes the following contributions:
We propose SRL, a novel framework designed to enable effective learning on difficult reasoning tasks where SFT and RLVR struggle, by providing dense and smooth rewards based on similarity with expert actions.
We demonstrate the effectiveness and robustness of SRL through extensive experiments on challenging mathematical reasoning and agentic software engineering benchmarks. Our results show that SRL significantly outperforms strong baselines across both domains & 5.3)
Through detailed analysis, we show that granular guidance is vital to SRL's reward and its impact on model behavior. We observe that SRL induces flexible and sophisticated reasoning patterns, such as interleaved planning and verification, which improve solution quality without simply increasing output length (.2).
2 RELATED WORK
2.1 SFT (DISTILLATION) FOR LLM REASONING.
Distilling reasoning into smaller models via SFT on teacher-generated long Chain-of-Thought (CoT) rationales has proven highly effective for transferring complex problem-solvee skills (Li et al. 2023 Huang et al.] 2024 Min al. 22024]] Yeo al. 2025), as exemplified by the small models distilled from DeepSeck RI (Guo et al. 2025). Research indicates this process is surprisingly data-efficiem
with small, high-quality datasets often being sufficient (Ye et al. 2025 Muennighoff et al. 2025)
2

---

Published as a conference paper at ICLR 2026
Query:
Rollouts
Teacher Response
Teacher Response
[Stpt Strpli
Brspsl
Tokao taken
Model
Model
Model
(a) RL(VR)
(b) SFT
(c) SRL
Figure 2: Illustration of SRL as compared to RL(VR) and SFT. (a) RL(VR) takes query as input and performs k rollouts. The final answer correctness is used the reward. (b) SFT uses both query x and a complete teacher response y as input, training with a per-token loss to maximize the probability ply ply|x). (c) SRL also uses a query and teacher response. It breaks the response into step actions and, at each step, uses the previous steps as context. The model generates a next step action along with its step-wise inner thoughts, and the reward rı based the similarity between the model's and the teacher's action.
Given the success, research has focused the underlying factor for effective SFT distillation (Chen al. 2025a). Some emphasized the logical structure of the reasoning trace rather than its semantic correctness (Luo al.
2025 Stechly al. 2025), as models can learn from demonstrations with factual errors (Li et al., 2025a). Moreover, significant challenges remain in the suddent-teacher gap where the student fails to learn from overly complex data (Li et al. (0256), and the risk of teacher hacking, where the student overfits to teacher's s specific flaws ( (Tiapkin et al. (0225). Ultimately, distillation from a teacher model imposes performance ceiling on the student (Huang et al. 2024).
2.2
RL FOR LLM REASONING.
The development of DeepSeek-RI (Guo et al. 2025 showed the effectiveness of rule-based RL for enhancing the reasoning capabilities of LLMs. This approach utilizes a scalable reward system based final answer correctness, exemplified by the Group Relative Policy Optimization (GRPO)
algorithm (Shao et al. 2024) and parallel algorithms (Ahmadian et al.) 2024 Lambert et al. 2024 Xie al.) 2025). Building this foundation, subsequent research has introduced numerous algo- rithmic refinements. For example, Dr. GRPO (Liu et al. (2025) mitigates bias by removing variance normalization, while DAPO Yu al., 2025) introduces token-level loss and relaxes the policy update constraint by increasing the clipping threshold. Other notable advancements include modifica- tions to clipping methods, normalization techniques, the KL divergence loss, and dynamic sampling strategies (Chu et al. 2025b; Zhang & Zuo [2025] Zhang et al., 2025 Chen et al. (025b). Despite these algorithmic variations, these approaches primarily rely on the final outcome's reward signal. critical challenge arises when the rollouts fail to identify correct solution trajectory, particularly for difficult queries. DAPO YYu et al. (2025). for instance, addresses this by filtering out instructions that do not yield any successful rollouts.
3 PRELIMINARIES
A Large Language Model (LLM) is formally defined by probability distribution pe over sequences of tokens, parameterized by set of model weights Given an input prompt, represented as a token sequence x [x1, [_i,--.. --- xn], the model generates response sequence y [y1,... yml The response is produced autoregressively. where the generation of the token Yj at any step j is conditioned on the initial prompt x and preceding tokens in the generated sequence, (yı,..., -i- 1). The joint probability of the entire response sequence given the prompt x is thus factorized as: xo(y = I Po ( Xx .1..... .))
Supervised Fine-Tuning (SFT). SFT is typically employed to specialize LLM for downstream applications or domains. is also commonly used to establish cold start for subsequent RL training phases that requires certain reply format/pattern, such RL for reasoning (Deng et al. 2025) tool
3

---

Published as a conference paper at ICLR 2026
Specifically, the process utilizes dataset D {xx(") {(x()))())) where each x+) is imput prompt and y-(1) is the corresponding desired model output. The primary objective is to update the parameters 0 to maximize the conditional probability of generating the target response y(t) given the imput prompt x+) This goal is formally achieved by minimizing the negative log- likelihood loss function: (0) the model learns to produce responses that are closely aligned with the exact words demonstrated in the labeled training examples.
use (Feng et al. 2025).
Zi=1 log po (y ) over the entire dataset. By minimizing this loss,
Reinforcement Learning (RL). Recent literature on improving model reasoning capability has focused on RL with verifiable reward (RLVR), where the policy model receives reward signals purely based on the final answer correctness. Building on this principle, Group Relative Policy Optimization (GRPO) (Shao et al. 2024) involves sampling a group of G response trajectories, {o, (=1> from the previous policy model, Hold, for each input query x. The objective function for GRPO is:
lo:1
po( x. 0x,0))
A,,: clip( Po(or, x. 01,c1)
E
min
1 - + (
Panca (Oi X, (,0,
Poaa (Oi, X, 0c,ct)
i=1
3 DKL po Pret]
The hyperparameter e > defines the clipping range for the policy update ratio, and the coefficient modulates the influence of the KI KI--divegeence penalty against the policy update. The term Oold refers the policy from the previous iteration. The advantage function, Å = (F: wiF:-man (ee--mma)F)FFFaeee (Ft is defined as the group-level normalized reward.
key challenge for these RL algorithms emerges when input queries are either too easy or hard, resulting in uniform correctness within policy rollouts {o In such cases, the ad advantage estimate A. vanishes, yielding an uninformative policy gradient and preventing model updates. A common strategy to mitigate this is to dynamically sample the batches, filtering out samples and re-sampling until the data sample satisfies 0 < |o, is (is_corret((o)) < G Yu et al. 2025).
4 METHODOLOGGY
4.1 THE CHALLENGE OF HARD REASONING PROBLEM
RL with verifiable reward is prominent technique for enhancing the reasoning capabilities of LLMs. The strategy is to close the gap between a model's 's potential performance across multiple attempts (pass@ k)) (Brown al. 2024 Yue et al., 2025). However, this paradigm falters on problems where the model's pass @ k rate is already near zero. For this set of difficult problems. which we term D positive reward signals are too sparse for RLVR be effective (Xiong al. 2025). Moreover, simply penalizing incorrect outputs can be detrimental to model performance ( YYu et al. 2025 Xiong et al. 2025), creating significant challenge for improving model reasoning.
Formally, we define Daard {x(t), al)) N the set of problems (x, a) where policy model's = a) < e, where each solution attempt y is sampled from the policy (-(x and 0 is small constant.
success rate is low with samples: & - (Extnctttttt
Due to the scarcity of successful trajectories, standard RL with verifiable reward struggles on Drand- Such data is further difficult be learned by SFT, due its limited amount and complexity in teacher reasoning trajectories (Li et al. 2025b).
4.2 SUPERVISED REINFORCEMENT LEARNING (SRL)
To address the challenge of learning from Dhand, introduce Supervised Reinforcement Learning (SRL), a framework that decomposes complex problem-solving as sequential decision-making process, and thus can be easily learned on how to properly operate step-wise. Instead of generating a monolithic solution, the model leams to take actions similar to the expert while producing their own inner reasoning process in a sep-bby-ttep manner. The whole framework is illustrated in Figure
'Empirically, we found that providing the subsquent step title (e.g., "2. **Coprime Pairs**" in Figure as the additional context for the learner predict the rest of the step content further boost the performance.
4

---

Published as a conference paper at ICLR 2026
Expert Trajectory:
Context
Given a national number between and 1, as need to write it as fraction in lowest tomms ((-Frac(o1(D)) and calculate the procuct of the resulting nuneraton and dencrinator, Me need detersine how many such rational nurbers exist 2011).
primss.
khere product
Instruction:
1. "pprie Factoriaation 20t prime factors 291 2. 11,
Complete
partial solution. You should first draft thinking process inner monologue..
[top
13, 17, 19, These are distinct primees
Step2
2. "sCopriee Patesti;
Magrd(s, assiggeed Mie,
<thinks
</thinko
in Vi2na 2361)
csolutions
2. det coprine Pairs:
*ncounting Pairs bigens
r2
for Vtises b 291 with Aged(a, bjjd, each of the cistinct prime factors of 20t nust be partitioned between them. This gives two choices for each prime, resulting </solutions
256 coprine patrs.
Figure 3: Given a solution trajectory. we take each summarized step as an action to be learned and take the partial solution before the step the context of our newly created data. The model is then prompted to generate its thinking process followed by the action for the current step. A reward (r2 in the figure) then calculated based the similarity between the model's and the expert's s action.
Action-based problem formulation. Given expert solution trajectory y that leads correct final answer, we decompose y into a sequence tuples: stys,p,, } Each step represents a logical action: the concrete action to be operated. This formulation is domain- agnaii-aggostic; for instance, an action in mathematical reasoning could be an algebraic manipulation, while for software agent, it could command executed in a code repository.
Step-wise training data construction. To create training data for SRL, we leverage a powerful teacher model, 0 exxpert, to generate solution trajectories. From a single complete solution with N steps, we construct N 1 partial trajectories. For each step k E {1,..., N 1), ). we create new input prompt Xsep, = [x, Ysee, ........sssp.............. Ystepp.... step, Ysep,. This process transforms one expert solution into rich of training instances that teach the model to proceed correctly from various intermediate states.
where the model's task to predict the subsequent
Learning with sequence similarity reward with own inner monologue. Given partial context Xaep, containing the problem and partial solution, the policy model po is prompted to generate the subsequent action step with their own inner monologue Vitink. which is encapsulated by "<think:>" tags. We then provide dense reward based the quality the generated logical action Ysu,, The prediction can be formally specified as: y' ~ Pe (- x
[ Y'sep..
To guide training, we consider the reward function that measures the similarity between the generated 2M action: R(y's sep. Sstep,)) = T where
T (Total elements):
This is the total number of elements in both sequences combined. It is calculated as the sum of the lengths of the two sequences: T = |Sil | + |S2]-
M (Matched elements): The total count of elements found in all non-overlapping matching blocks between the two sequences. The algorithm first finds the longest contiguous matching subsequence and then recursively searches for more matches in the segments to the left and right of that block.
If we represent the set all such matching blocks as a list of tuples (i. n), where n is the length of the matching block, then M is the of all lengths M " (R.jjn)cMachhhnllell
Combining these definitions, we can calculate the similarity ratio E [0, as:
2 (sdn)
R
|S|++|S)
While the choice of similarity function be adapted to specific data distributions task require- for our implementation. Furthermore, a
ments, we employ Python's 's difflib. di.ffiib.equeenchhattce negative reward is assigned if the generated output y' fails to adhere to the required format. Hence,
5

---

Published as a conference paper at ICLR 2026
the final reward used is:
R(yaep, Nstep,) if y' follows format,
ryyuup, sepe)
otherwise.
The policy pe is then optimized using this reward signal with the GRPO objective function defined in Equation [ Notably, our reward is computed only on the logical action, not the internal monologue. This grants the model flexibility develop its internal reasoning style while ensuring its external actions align with the expert's strategy. This design provides dense, step-level feedback and allows for rapid reward caleulation, making the SRL framework both effective and scalable.
Dynamic sampling for SRL. As our reward signal r E [o, 1] [-1} is dense, we generalize the dynamic sampling strategy previously designed for outcome accuracy and implement it to filter samples with less meaningful updates. Specifically, a sample should be filtered out if its rollouts yield rewards with near-zero variance, providing a weak advantage and thus weak learning signal. With the sequence similarity reward in SRL, we retain sample if the standard deviation of the reward scores of its rollouts exceeds a threshold c > 0:
G (r(o))))
- F) >
G
where G is the number of generated rollouts, r(o,,y) is the sequence similarity reward for the i-th rollout o, given the expert trajectory y. and is the mean reward for the sample. To maintain consistent batch size of B, we continuously sample and filter until the batch is filled.
5
EXPERIMENTS
5.1 MAIN RESULTS: MATH REASONING
Setup. We finetue Qwen2.5-7B. S-TB-Instruct Yang et al. 2024) on the s/K-1. dataset (Muennighoff et al. 2025). This dataset contains 1,000 diverse and challenging questions, each accompanied by a detailed reasoning trace and final solution generated by DeepSeck R1. The solutions from DeepSeek RI are formatted with structured, numbered steps (e.g.. "1. Title of Step 1*). We leverage this structure to generate intermediate training targets by parsing these solutions and treating each complete step as a ground- ground-truth continuation. Any data points that do not adhere to this format are excluded. The detailed data preparation process can be found in Appendix B.3 We hold out 60 questions from the dataset to form our validation set.
Baselines. We benchmark our proposed methods against several baselines, all initialized from the Qwen2.5-7B-Inan model. These baselines include:
1. SFT on either the complete reasoning traces (R1 reasoning) or the final solutions from the s s1K11.1 dataset (R1 outline)
2. s1K-BB, the official distilled model released by the dataset's authors
3. RLVR, which we implement using the GRPO algorithm. To ensure fair comparison, we implement additional dynamic sampling as in Yu et al. (2025), which removes samples with all correct or incorrect rollouts.
4. SFT RLVR, the same RLVR algorithm applied after initial SFT phase
5. R
Xi et al. 2024). which applies a reverse curriculum RL learning strategy by start- ing near the end of the demonstration solution and working backward with step-by-step decomposition to make the sparse final reward easier to find
Our proposed method, SRL, is likewise evaluated both as standalone technique and in a sequential configuration where precedes RLVR (SRL then RLVR). All models are trained for up to 30 epochs, and for each method, we select the checkpoint with the best performance on the validation set.
Evaluation. We evaluate all models on the following four competition- mathematical reasoning benchmarks: AMC23, AIME24, AIME255 and Minerva Math (Lewkowycz et al. 2022). Our
We re-implement this method by reformulating the data from s/K-1.1.
Data available HuggingFace (htts://huggingface.cm con/atttttcc///ttstttssss AI-MO/an http//amoovvlllla wnw.liinnnaaaaana and math-ai/aimm25
AI AT-MO/aimo-validatiion

---

Published as a conference paper at ICLR 2026
evaluation protocol for all benchmarks follows the package released by Qwen2.5-Maie and report the accuracy of greedy sampling. The evaluation prompt can be viewed Appendix B.2.2 In addition, for AMC23, AIME24 and AIME25, report the average @3 score with temperature of 1.0 for all baselines to ensure more robust evaluation, referenceing the evaluation Yu et al. (025).
The performance results of models summarized in Table Consistent with the officially released S1K-7B model, our model trained with SFT on the same dataset exhibited a notable performance degradation. In contrast, methods based on RL maintained generalization on the evaluation benchmarks. Specifically, while RLVR maintained the performance, SRL provided substantial boost of 3.0% on average. Furthermore, applying RLVR after SRL training yielded 3.7% increase on average, leveraging only Ik training data.
Performance.
Comparatively, while the baseline R also use teacher demonstrations to structured a step-by-step learning, it exhibits distinct performance characteristics. As shown in Table [ R performs competi- tively on AMC23 but lags behind SRL on the more challenging AIME24 and AIME25 benchmarks (e.g., SRL achieves 16.7% vs. R 13.3% on AIME24). This divergence highlights critical differ- ence between SRL and R objective, which relies on a reverse curriculum to facilitate exploration towards a sparse outcome reward. In contrast, SRL employs dense, intrinsic supervision, rewarding the model for the quality its step-wise "actions" rather than just the final answer. Consequently. for complex reasoning tasks where outcome-bet based exploration is very challenging, SRL provides more robust learning signal, effectively teaching the model how to reason and act.
Table I:
Evaluation results across competition-level math benchmarks. We take Qwen2.5-7B-Instee as the base model and report the performance of different training schemes (SFT, RLVR via GRPO, and SRL) using the same of training data. The bold numbers indicate the best results among the open-source models and the underscored numbers represent the second- bestdd-best results.
AMC23
AIME24
AIME25
Model
Minerva Math
Average
Avg
Greedy
Avg
Greedy
Avg@32
Greedy
Base Model
Qwen2.5-7B-Instte
49.3
50.0
10.5
13.3
2.5
6.7
34.9
24.6
Training with SFT
SIK-7B
24.1
25.0
2.2
3.3
3.7
3.3
20.2
11.7
SFT (R1 reasoning)
26.8
40.0
3.9
10.0
5.4
10.0
20.2
16.6
SFT (R1 outline)
36.2
27.5
5.1
3.3
3.8
6.7
31.6
16.3
Truining with Outcome-Reware RL
RL(VR)
$2.0
47.5
11.1
10.0
7.4
10.0
33.8
24.5
SET (outline) RL(VR)
37.6
35.0
4.9
4.5
30.1
17.4
R
52.5
57.5
10.0
13.3
7.4
35.3
26.6
Truining SRL
SRI. SRL RLVR
$1.5
50.0
16.7
7.1
52.1
57.5
13.3
20.0
8.6
10.0
36.4
28.3
5.2 ANALYSIS: MATH REASONING
Effect of dynamic sampling in SRL. In Table we analyze the impact of the dynamic sampling component in SRL, based on thresholding the standard deviation of sequence similarity rewards within rollouts. For both models, train until the training reward converges and select checkpoint based on validation scores. Our results are consistent the findings of DAPO (Yu et al. 2025), which stated that removing samples that provide zero learning signal is critical in the effectiveness of the RI. training loop, showing non-trivial d performance improvement.
Disentangling the impact of guidance granularity in SRL. To isolate the effects of guidance granularity from the benefits of sequence similarity rewards, we evaluated our multi-step method against two simplified, one-step baselines:
Fnnwl- reward: This bascline utilizes RLVR trained with GRPO, rewarding the model solely based the correctness of the final answer.
"hatss//gituuh.co////enl hatp:///0000eeM////00///5/---a

---

Published as a conference paper at ICLR 2026
Table The effect of dynamic filtering on SRL. Filtering out samples with less meaningful updates provides non-trivial performance improvement. DS stands for dynamic sampling.
AMC23
AIME24
AIME25
Model
Minerva Math
Average
Avg@33 @2 32
Greedy
Avg@32
Greedy
Avg@32
Greedy
SRL w/out DS
48.5
52.5
11.1
13.3
6.8
6.7
33.8
24.7
SRL DS
51.5
50.0
13.2
16.7
7.1
13.3
36.4
27.6
Table 3: Model Performance of different reward functions and density. For sequence similarity reward, implement it with the entire expert output as one-step supervision. The model benefits from our multi-step decomposition on the small set challenging training data.
AMC23
AIME24
AIME25
Model
Minerva Math
Average
Avg 32
Greedy
Avg @3
Greedy
Avg@32
Greedy
Final Answer Reward
One-step (RLVR)
52.0
47.5
11.1
10.0
7.4
10.0
33.8
24.5
Sequence Similarity Reward
One-step
51.8
52.5
11.7
13.3
6.9
34.9
25.9
Multi-step (SRL)
51.5
50.0
16.7
7.1
13.3
36.4
27.6
Holistic sequence similarity reward: The model generates complete solution in single step. This entire solution then evaluated for similarity against the complete ground-truth trajectory.
As shown in Table 3 the results underscore the value of fine-grained guidance for sequence similarity reward. While the holistic sequence-similand reward improved performance average, providing granular, step-by-step guidance led to markedly superior performance across benchmarks.
Model size variation. To evaluate the scalability of our approach, apply method Qwen2.5- 3B-Instruct (Yang et al. 2024) using the identical data setup from Table As summarized in Table 4 method yields consistent performance boost, improving the average score base model by 3.1% and surpassing RL(VR). Notably, while the limited capacity of the smaller base model restricts improvements on extremely difficult benchmarks (c.g., AIME24 & AIME25), we observe significant gains on more tractable tasks like AMC23 and Minerva Math.
Table 4: Evaluation results across competition-levé math benchmarks using smaller base model. We take -wen Instruct as the base model in this table to test the generalization of our method.
AMC23
AIME24
AIME25
Model
Minerva Math
Average
Avg@32
Greedy
Avg@32
Greedy
Av
Greedy
Owen2.5-3B-Inseee
32.5
45.0
1.7
3.3
25.7
RL(VR)
35.0
42.5
SRI. RLVR
37.5
47.5
6.7
28.3
Interlcaved reasoning behaviors. Beyond achieving superior performance, models trained with SRL exhibit distinctive and adaptive reasoning trajectories. While conventional models typically generate a single. monolithic reasoning block at the outset, our SRL-finetuned models aadels--prttcularlly RLVVR-dynamicaly
those trained with RLVR- dynamically interleave reasoning steps throughout the generation process.
This behavior manifests in three primary modes: (1) Structured Planning, where the model outlines a comprehensive roadmap before execution; (2) Dynamic Adjustment, where the model inserts iterative reasoning blocks to refine its trajectory mid-solution; and (3) Reflective Verification, where the model pauses to evaluate its logic and verify the answer before committing to a final output. These emergent patterns illustrated in Example [ and Example (see Appendix), which showcase the model's transition from static generation to an integrated, think-and-aek reasoning process. For further qualitative analysis and studies, please refer Appendix
8

---

Published as a conference paper at ICLR 2026
Example Interleaved thinking and self-verilication
Question:
What is the least positive integer value of x for which the inequality 3æ >> 2xr + is true?
<think>
solve the inequality 3x 2x we can start by isolating æ. Let's subtract 2xr from both sides the incquality.
</think>
Subtract 2z from both sides:
3x
2x
2x
2r
This simplifies to:
>
<think>> We can verify this by substituting Ir = 2 into the original inequality to ensure it holds true. </think>
The inequality 1 tells that must be greater than 1. The smallest positive integer greater than 2. Therefore, least positive integer value of that satisfies the inequality Substitute 2 into the original inequality:
3(2)
2(2) +1
This simplifies to:
6 > 5
This is so is indeed solution. Therefore, the least positive integer value of æ for which the inequality 3x > 2x + 1 true is
2
5.3 EXTENSION: SOFTWARE ENGINEERING AGENTIC REASONING
Task. We extend SRL framework the domain of software engineering, training agents resolve real-world programming issues. These tasks are commonly evaluated on benchmarks like SWE-Bench (Jimenez with large codebases and reason deeply about code functionality.
2023). which requires agents to perform complex, multi-turn interactions
However, unlike in math domain, the direct application of online RL to software engineering is impeded by significant practical challenges. These include handling long context windows. high- latency environment feedback, and slow patch verification (Wei al. 2025) Golubev al., (2225). Consequently, these obstacles have hindered the development of stable and scalable end-to-end RL. methods, leading prevailing approach of collecting expert agent trajectories and distilling them into a policy via SFT (Pan et al. 2024 Yang al. 2025).
Setup. We apply SRL to further fine-tune Qwen2.5 55 CCoder-7B-Instrreee (Hui et al. 2024), model already specialized for coding tasks. We use dataset from Yang et al. (202), which consists of 5,000 expert agent trajectories. These trajectories were generated by 2-rerd---7-sonne--2222222 thropic 2025 and subsequently verified to ensure they produce correct code patches.
(An
Each trajectory is composed of multiple steps defined by the agent's interactions with the coding environment. As the example below illustrates, single step contains natural language reasoning followed by an executable action:
I'11 help you implement the necessary changes to fix the issue with the "OriginVal idater not preperly handling wildeard (9). allowed_origins.
the steps you outlined.
aa Step 1: Find and read relevant code
First. let's explore the repository structure to locate the "Originvalidator" class nentioned description. This wrapping feature from the listings package works.
long 1ine
autonatic
<function=bash>
sparameter /tasthed -type -name "x.py" grep -v "__pycache___" paraneee
c/function>
In line with our SRL formulation (Section 4.2) we define the "action" as the environment-consumee command (e.g., the bash call). Following this decomposition, process the full trajectories to create

---

Published as a conference paper at ICLR 2026
134k step-wise training instances. For validation, we hold out 30 full trajectories, from which curate a validation set of 650 step-wise instances.
Evaluation. We evaluate our model's patch generation performance by measuring its resolve rate (%) under two distinct configurations, following Wei et al. (2025): (1) Oracle file editing evaluation: The model is provided with the oracle code files repair. This configuration isolates and measures the model's 's core patch generation capability; (2) End-to-end evaluation: This setting uses the Agentless- mini agent scaffold (Wei et al. 2025) to first identify the file(s) modify and subsequently generate the patch. It tests the model's fault localization and code repair abilities in conjunction.
We compare our SRI-trained model against two crucial baselines: the original base model (Qwen2.5- Coder-Instruct) and SWE-Gym-7B (Pan et al. 2024). Since SWE-Gym-7B is an SFT-based model finetuned from the same base model, this provides a direct, fair comparison between SFT and our SRL training methodology. As shown in Table 5 SRL substantially outperforms both baselines. In the oracle setting. SRL achieves a 14.8% resolve rate, representing 74% relative improvement over the strong SWE-Gym-7B baseline. The performance gain is consistent when evaluating in the challenging end-to-end setting, where SRL can obtain twice the performance.
Table 5: Performance of SRL on SWE-Bench-Verifiet Results in the table are using greedy decoding.
Oracle File Edit
End-to-End
Qwen2.5- (reen25--oder--iie 5-Cooder-Insstuuc (Base) SWE-Gym-7B
5.8
3.2
8.4
4.2
SRL (ours)
14.8
8.6
5.4 DISCUSSION AND LIMITATIONS
Despite the strong performance of SRL, its effectiveness subject to several inherent constraints. Pri- marily, the method relies heavily on the availability of high-quality, decomposable expert trajectories, as it assumes well-defined "action" granularity where complex reasoning can cleanly partitioned into discrete steps. This dependency suggests that SRL may be less effective in domains where expert demonstrations are sparse or tasks that require holistic intuition rather than sequential chain-of-thought structure. Furthermore, the success of the self-reflective loop is fundamentally tied to the student model's initial proficiency; the model must already possess a baseline instruction- following capability generate rollout samples that are coherent enough to yield non-trivial reward signals. Without this initial competence, the learning process may suffer from excessively sparse rewards. Consequently, future research will focus adapting SRL to domains with ambiguous action definitions, exploring techniques to alleviate the dependency on highly structured expert data, and investigating the variance in performance when applying different similarity functions to guide the learning process.
CONCLUSION
In conclusion, we introduced Supervised Reinforcement Learning (SRL), a novel method designed to teach LLMs complex reasoning skills from expert demonstrations, particularly for problems that are too difficult for conventional RL. or SFT approaches. By breaking down expert solutions into manageable steps and leveraging a dense sequence similarity reward, SRL provides effective, granular guidance that bridges the gap between imitation learning and reinforcement learning. Our empirical results demonstrate that SRL not only significantly outperforms baseline methods in both mathematical reasoning and software engineering tasks but also enables a powerful curriculum learning strategy when combined with RLVR. This work establishes SRL as robust and generalizable technique for unlocking a model's 's potential to learn from challenging, multi-step problems, paving the way for training more capable and versatile Al agents.
7
ACKNOWLEDGMENTS
We thank Jinwei Xing, Jinsung Yoon, and members from Google Cloud AI Research for their valuable feedback during the preparation of the paper.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint MrXivv24022147400, 2024.
Anthropic. Introducing claude 3.7 sonnet, 2025., 2025, URL claude-3-7 3---nen
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21778 2024.
Xinghao Chen, Zhijing Sun, Wenjin Guo, Miaoran Zhang. Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, et al. Unveiling the key factors for distilling chain-of-thhough reasoning. arXiv preprint arXiv:2502.18001, 2025a.
Yi Chen, Yuying Ge, Rui Wang, Yixiao Ge, Junhao Cheng, Ying Shan, and Xihui Liu. Grpo- care: Consistency-aware reinforcement learning for multimodal reasoning. arXiv preprint arXiv:2506.16141, 2025b.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: comparative study of foundation model post-training. arXiv preprint arXiv:2501. 17161, 2025a.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint 3xXiv:2504002544 2025b.
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai- Wei Chang. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self self-improvement. arXiv preprint arXiv:2503. /7352, 2025.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in Ilms. arXiv preprint arXiv:2504. 1536, 2025.
Alexander Golubev, Maria Trofimova, Sergei Polezhaev, Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Sergey Abramov, Andrei Andriushchenko, Filipp Fisin, Sergei Skvortsov, and Boris Yangel. Training long-context multi-tur software engineering agents with reinforcement learning. arXiv preprint arXiv:2508. 03501, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
I-Hung Hsu, Zifeng Wang, Long Le, Lesly Miculicich, Nanyun Peng, Chen- Yu Lee, and Tomas Pfister. CaLM: Contrasting large and small language models to verify grounded generation. In Findings of the Association for Computational Linguistics: ACL 2024, 2024.
Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey-part 2: Surpassing ol prepveeew through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.1644 2024.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. /2/86, 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.0677 2023,
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Imss to reason and leverage search engines with reinforcement leamning In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
11

---

Published as a conference paper at ICLR 2026
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXhv:241..55124, 2024.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer. Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 2022.
Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, et al. Lims can easily leamn to reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374 2025a.
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. Sym- bolic chain-of-thoughh distillation: Small models also" think" step-by-step. arXiv preprint arXiv:23306. 14050, 2023.
Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. arXiv preprint wxxxi:2502.12144 2025b.
Zhiwei Li, Yong Hu, and Wenqing Wang. Encouraging good processes without the need for good answers: Reinforcement learning for Ilm agent planning, arXiv preprint arXiv:2508. 19598, 2025c.
Zichen Liu, Changyu Chen, Wenjun Li. Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl -lzero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
Yijia Luo, Yulin Song, Xingyao Zhang, Jiaheng Liu, Weixun Wang, GengRu Chen, Wenbo Su, and Bo Zheng. Deconstructing long -hain-of-thought. A structured reasoning optimization framework for long cot distillation. arXiv preprint arXiv:2503. 16385, 2025.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09411 2024.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès. and Tatsunori Hashimoto. s1: Simple test-time 19393
scaling, 2025. URL https::/aaxxv.org/aabb//////
Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engincering agents and verifiers with swe-gym. arXiv preprint 1rX1v:2412.21139, 2024.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru WANG, Xiusi Chen, Dilek Hakkani-Tine Tür, Gokhan Tur, and Heng Ji. TooIRL: Reward is all tool learning needs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JML Workshop and Conference Proceedings. 2011.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits mathemat- ical reasoning in open language models. arXiv preprint 24X::3240..03000, 2024.
Kaya Stechly, Karthik Valmeekam, Atharva Gundawar, Vardhan Palod, and Subbarao Kambhampati. Beyond semantics: The unreasonable effectiveness of reasonless intermediate tokens. arXiv preprint NXXv:2505.13775 2025.
12

---

Published as a conference paper at ICLR 2026
Zhen Tan, Jun Yan, I-Hung Hsu, Rujun Han, Zifeng Wang, Long Le, Yiwen Song, Yanfei Chen, Hamid Palangi, George Lee, Anand Rajan Iyer, Tianlong Chen, Huan Liu, Chen- Yu Lee, and Tomas Pfister. In prospect and retrospect: Reflective memory management for long-term personalized dialogue agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2025.
Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. CoRR, abs/2507.06200 2025.
Danil Tiapkin, Daniele Calandriello, Johan Ferret, Sarah Perrin, Nino Vieillard, Alexandre Ramé, and Mathieu Blondel. On teacher hacking in language model distillation. arXiv preprint arXiv:2502.02671, 2025.
Chaojie Wang. Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q Im- proving multi-step reasoning for Ilms with deliberative planning. arXiv preprint arXiv:2406. 14283, 2024.
Peng-Yuan Wang, Tian-Shuo Liu, Chenyang Wang. Yi-Di Wang, Shu Yan, Cheng- Xing Jia, Xu-Hui Liu, Xin-Wei Chen, Jia-Cheng Xu, Ziniu Li, and Yang Yu. A survey on large language models for mathematical reasoning. arXiv preprint arXiv:2506. 084466 2025.
Yuxiang Wei. Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing 1lm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:250211844, 2025.
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang, and Xuanjing Huang. Training large language models for reasoning through reverse curriculum reinforcement learning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. In Forty-first International Conference on Machine Learning, 2024.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong. Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502. 14768 2025.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, and Hanze Dong. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint 250: 1343, 2025.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge. Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 2024.
technical report. arXiv preprint 4rXiv:2407.1066)
John Yang, Kilian Leret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software 2025.
engineering agents. arXiv preprint arXiv:2504.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025.
Edward Yeo. Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long thain-of-thought reasoning in llms. arXiv preprint arXiv:2500.03333, 2025.
13

---

Published as a conference paper at ICLR 2026
Fan Yin, Zifeng Wang. I-Hung Hsu, Jun Yan, Ke Jiang, Yanfei Chen, Jindong Gu, Long Le, Kai- Wei Chang, Chen- Yu Lee, Hamid Palangi, and Tomas Pfister. Magnet: Multi-turn tool-use data synthesis and distillation via graph translation. In Proceedings of the 63rd Annual Meeting the Association for Computational Linguistics (Volume I: Long Papers), 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-souree Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity Ilms beyond the base model? arXiv preprint arXiv:2504.1338 2025.
Jixiao Zhang and Chunsheng Zuo. Grpo-lead: difficulty- aware reinforcement learning approach for concise mathematical reasoning in language models. arXiv preprint arXiv:2504. 2504,09696, 2025.
Xiaojiang Zhang, Jinghui Wang. Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, al. Srpo: cross-domain implementation of large-scale reinforcement learning on Ilm. arXiv preprint arXiv:2504. 00.55544.42220 14286, 2025.
14

---

Published as a conference paper at ICLR 2026
A ILLUSTRATION OF SRL ON SWE TASKS.
In Figure 4 illustrate how approach the SWE tasks with SRL. We take consecutive action-observation pairs from the expert trajectories in the given SFT data as context. We prompt the LLM to first think in monologues and then conclude with its action. Sequence similarity score is thus computed between model action and expert action in the trajectory.
Past Expert Action Mundiaansitr.ee wew.irreeaac........ adilor pprsmeter-cmmee communddrreeeeeeeeeeee /parmetters sporameted ooth /eetbbe//ee cyuceppy/pp pp ameter-
Past Observation auccersfully at: /teltbed/reprodu pypyy
Past Expert Action <funclion-bassh /testbee 6& python rearocuce pys parameter> s/tunction-
Past Observation importing generated persers run ppython setup.py antir regenarate.
1ssue
Context
Model Thoughts <think> The error message indicates that the generated parsers are missing outoated... </ /think>
Model Action <function-bashh: /ttesibed python setup.py py antrr fuunctoon>
Expert Action <function-bashh /estbed pip install -e e/function>
LLM
Reward similarity(model action, expert action)
Figure 4: Illustration of applying SRL to SWE tasks. We take two rounds of the past expert actions and corresponding observations context and prompt the LLM to think before reaches the action. Reward is then calculated via comparing model action with the expert action.
B EXPERIMENT DETAILS
B.1 COMPUTATION AND PARAMETER DETAILS
We use VeRL as the training framework for GRPO and implement SRL based on it. Table 6 and g detail the hyperparameters that used for SFT, GRPO and SRL. The batch size for GRPO is set to be smaller (128) than SRL (512) as the filter rate is high based on final answer correctness, for most of the questions are very difficult. We maintain smaller batch size avoid repeated sampling and achieve better performance. We further lay out the system prompts we used for SRI. Experiments were conducted on GPU clusters the similar level of NVIDIA H100 80GB GPU.
Table Supervised fine-tuning hyperparameters.
Data type
hf16
Learning rate
5e-6
Global batch size
64
Scheduler
Cosine
Warmup ratio
0.3
Num train epochs
15

---

Published as a conference paper at ICLR 2026
Table 7: GRPO hyperparameters.
Batch size
128
Max grad norm
1.0
Data type
bfl6
Learning rate
5e-7
Warmup ratio
0.0
Rollout temperature
1.0
Rollout number
KL loss coeff
Table 8: SRL hyperparameters.
Batch size
512
Max grad norm
1.0
Data type
bf16
Learning rate
5c-7
Warmup ratio
Rollout temperature
1.0
Rollout number
8
KL loss coeff
B.2 PROMPT DETAILS
B.2.1 SYSTEM PROMPT FOR SRL LEARNING
System Prompt for SRL
solving nathenatical problens. provide problen, which include partial solution. Your task to continue the solution by providing the next logical step.
helpful
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). generate solution.
response
<thinks Your thoughts or/and draft, like working through exercise scratch paper. casual and as long as you want until you are confident to generate correct solution. </think>
Provide only the single, next step to continue the solution. Do not solve the entire problen.
B.2.2 SYSTEM PROMPT FOR EVALUATION
System Prompt for Math Evaluation
Please reason step by step, and put your final answer within \boxed(). You can draft your thinking process inside <think> </thinks tags anywhere. This your "scratch paper".
B.3 TASK DECOMPOSITION DETAILS
We conduct the task decomposition for math via data re-writing process. We first decide format that is easy to parse trajectory into steps:
Template Format for Task Decomposition Math
SOLUTION:
AMEESEP T ((STEP ACTTON))
2. **((SE TITLE)}** ((STEP ACTION))
Then prompt Gemini-2.55fil (Team 2025) to conduct data conversion using the below prompt:
16

---

Published as a conference paper at ICLR 2026
Prompt Used for Data Rewrite in Math
BAR ROLE
exper Tutor Educational Content Fermatter. goal is problen and its solution, and rewrite it into strictly structured, easy-to-fallow, step-by-step guide.
##F INPUT DATA
I will provide you with:
The Math Question.
The Original Solution.
#d# OUTPUT FORMATTINE RULES
You must adhare following format strictly:
SOLUTION:
>>{[STEP TITLE))
((STEP ACTTON))
**{[STEP TITLE)]**
((STEP ACTION)) [Continue until solved]
**Step Ticles:** Nust be bolded (A*litle<*). They should be short, imperative verb phrases (e.g., Identify the variables," "Isolate x," "Substitute values").
**Step Actions:** The equations and operations to execute according the title
Please follow the original thoughts and solutions provided the "Original Solution"'. Do nat salve
the task by your
[[THG SHOT EXAMPLE PUT HERE]]
VA CURRENT TASK
*sQuestion:** CINSERT QUESTION HERE]
*ndriginal Solution:** [INSERT SOLUTION HERE]
For the data after conversion cannot pass our formatting check, we will discard them.
C SUPPLEMENTARY STUDIES
Remsuning Langth
00003
600
Figure 5: Reasoning length distribution for base model and model trained with SRL.
In this section, we provide further analysis to supplement our experimental results in Section 5.2 Specifically, we aim to illustrate the mechanisms behind the performance improvements of our SRL-finetuned model. While recent work often associates improved reasoning with longer generation sequences, we first demonstrate that our model's gains not result of increased verbosity, In addition, we show that the model develops a more sophisticated internal structure. Through the following examples of structured planning and post-hoc self-verificatin the model leverages specific reasoning patterns to navigate complex mroblem-solving tasks more effectively than the base model.
visualize how
17

---

Published as a conference paper at ICLR 2026
Reasoning length. We investigate whether the performance gains from SRL can be simply attributed to an increase in reasoning length. In Figure we plot the distribution of reasoning lengths (i.e. the number of words in the solution) for both the base model and SRL-finetuned model. We observe no significant difference between the two distributions. This finding suggests that the performance improvement stems from enhanced planning and higher-quality reasoning. rather than merely increased token budget at inference time.
Qualitative Case Studies. The first example demonstrates a priori structured planning, while the second showcases the model's ability to perform mid-stream or post-hoc self-verification.
Example 2: Planning
Josh decides to try flipping house. He buys a house for $80,000 then puts $50,000 repairs. This increased the value of the house by 150%. How much profit did he make? <think>>
Question:
To determine Josh's profit, need follow these steps:
Calculate the new value of the house after the repairs.
2. Subtract the initial cost of the house and the repair costs from the new value to find the profit. </think>
First, let's calculate the new value of the house after the repairs. The value of the house increased by
150% after the repairs.
initial cost of the house is $80,000, and repairs cost $50,000. The total cost hefore value increase is:
80, 000 + 50, 000 = 130, 000
value house increased by 150% initial value $80.000. So, increase value
150% x 80, 000 1.5 80, 000 = 120, 000
Therefore, the new value of the is:
80, 000 120, 000 = 200, 000
Now, subtract the total (initial cost of the house plus the repair costs) from the new value of the house to find the profit:
200, 000 130, 000 = 70, 000
So, Josh's profit is:
70000
Example 3: Self-verification
Question:
Which one of these constellations is not located along the Milky Way in the sky?
Answer Choices: (A) Perseus (B) Cygnus (C) Scorpius (D) Leo
First, recall which constellations are along the Milky Way. Perseus, Cygnus, and Scorpius are all well-known constellations that lie along the Milky Way. However, Leo is not one of them; it located in the northern sky and does not align with the Milky Way. Therefore, the correct answer is (D) Leo.
<think>
double-check, can recall general locations these constellations:
Perseus is in the northern sky and is part of the Perseus Arm of the Milky Way.
Cygnus is also in the northern sky and is part of the Orion Arm of the Milky Way.
Scorppiu in the southern sky and part of the Sagittarius Arm of the Milky Way. Leo the northern sky and not aligned with the Milky Way.
</think>
The answer is (D) Leo.
D LLM USAGE
Throughout the preparation of this manuscript, we utilized a LLM as an assistant with refinement of writing. Its role strictly limited to improving grammar, rephrasing sentences for clarity, and
18

---

Published as a conference paper at ICLR 2026
correcting spelling. All intellectual contributions, ideas, and conclusions presented in this paper entirely the work the authors.
19


Reviews:
[{"review_id": "c2MUXBVkxw", "reviewer": "Reviewer_WNpC", "summary": "This paper introduces Supervised Reinforcement Learning, a framework that bridges the gap between Reinforcement Learning with Verifiable Rewards, which suffers from sparse rewards on difficult problems, and Supervised Fine-Tuning, which often leads to overfitting through rigid token mimicry. SRL reformulates problem-solving as a sequence of logical actions, where the model generates an internal reasoning monologue followed by an action, receiving dense rewards based on action similarity to expert demonstrations. This allows flexibility in reasoning while providing granular feedback.", "questions": "1. For extremely difficult problems where the model fails to rollout any reasonable intermediate step actions (e.g., pass@k near zero even for steps), how does SRL enable effective learning? Could the dense rewards still provide guidance, or would additional techniques like bootstrapping be needed?\n\n2. The method relies on a strong teacher model (e.g., DeepSeek R1) to generate reliable expert trajectories. If no such trustworthy teacher is available (e.g., for novel domains), how could SRL be adapted—perhaps through self-generated or synthetic data, or iterative refinement?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a well-motivated and innovative approach to addressing limitations in LLM training for complex reasoning, with clear writing.\n\n2. The introduction of SRL creatively combines elements of imitation learning and RL by decomposing expert trajectories into step-wise actions and rewarding only the actions, allowing the model to develop its own internal monologues. This novel formulation addresses the sparse reward issue in RLVR and the rigid mimicry in SFT, drawing from prior work like GRPO but extending it to dense, sequence-similarity-based rewards for hard problems.\n\n3. By enabling effective learning on difficult datasets like s1K, SRL has broad implications for enhancing LLMs in domains requiring multi-step reasoning, such as math and software engineering. Its generalization to agentic tasks and potential as a cold-start for RLVR pipelines could influence future distillation and RL methods for smaller models.", "weaknesses": "1. The method lacks sufficient implementation details, raising concerns about reproducibility. For instance, the prompts or heuristics used to decompose expert reasoning traces into individual steps (e.g., parsing numbered sections from DeepSeek R1 outputs) are not provided, making it challenging to replicate the step-wise data construction process.\n\n2. Although SRL rewards only the action sequence to avoid token-by-token memorization, this still imposes constraints on the model's output. For example, it may force the student to mimic potentially suboptimal or erroneous intermediate actions from the teacher (e.g., steps involving reflection to correct mistakes), even if the student could compute them correctly without such detours. This could limit flexibility in cases where the teacher's reasoning includes unnecessary self-corrections.\n\n3. Experiments are limited to a single 7B model (Qwen2.5-7B-Instruct), without exploring scalability across different model sizes (e.g., 1B, 13B, or larger). This leaves uncertainty about whether SRL's benefits hold for smaller or larger models, potentially restricting insights into its robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "99r9Y4Ij8a", "reviewer": "Reviewer_VwHa", "summary": "This paper introduces Supervised Reinforcement Learning, a framework designed to enhance complex reasoning in LLMs by decomposing problems into step-wise actions. SRL combines dense, step-level rewards with flexibility in internal reasoning generation. The method is evaluated on mathematical reasoning and software engineering tasks, showing improvements over SFT and RL baselines. Key contributions include: (1) a step-wise reward signal to mitigate sparse rewards, (2) empirical gains on challenging benchmarks, and (3) a curriculum strategy (SRL→RLVR) for further refinement.", "questions": "None", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem formulation (e.g., action-based decomposition) is well-motivated and intuitive.\nResults on math benchmarks show consistent improvements, with ablations justifying key components.\nThe curriculum strategy (SRL→RLVR) and flexibility in internal monologue generation are valuable insights.", "weaknesses": "The methodology bears significant resemblance to R³ (arXiv:2402.05808), which also uses a reverse curriculum with intermediate state sampling and outcome supervision for step-wise learning. Though SRL uses action similarity rewards instead of final-answer rewards, the high-level strategy of \"starting from expert states and moving backward\" is not sufficiently differentiated.\nComparisons focus on SFT and RLVR but omit advanced RL methods (e.g., DAPO, Dr. GRPO) and direct comparisons to R³.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hByjm3p3hy", "reviewer": "Reviewer_CWA3", "summary": "This paper proposes Supervised Reinforcement Learning (SRL), a new framework that bridges the gap between Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) for complex reasoning tasks. \nThis approach provides richer learning signals than RLVR and avoids the rigid imitation of SFT. \nExperiments on mathematical reasoning and software engineering benchmarks show that SRL significantly outperforms both baselines, and a curriculum of SRL followed by RLVR achieves the best overall results.", "questions": "1. I am curious how robust your difflib-based action-similarity reward is to “near-miss” actions\n2. I think the framework genuinely novel and potentially a new training paradigm for leveraging reward signals. I encourage you to evaluate SRL beyond structured domains in your paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The novel SRL framework addresses RLVR's sparse rewards and SFT's rigid mimicry by decomposing expert trajectories into step-wise logical actions with dense sequence-similarity rewards, uniquely combining SFT’s supervision and RL’s strengths and showing superior performance on math reasoning benchmarks.\n2. SRL demonstrates robust cross-domain efficacy across different multi-step reasoning tasks, proving its versatility.\n3. SRL excels in data efficiency for hard problems, using limited expert data to avoid SFT's performance degradation and enabling a high-performing SRL→RLVR curriculum.", "weaknesses": "1. SRL relies on the decomposition of expert trajectories into logical actions without providing details on automation for unstructured trajectories. Therefore, I'm not sure if it is scalable for large-scale or unstructured tasks and whether it introduces subjective bias.\n2. The reward only considers action similarity, potentially leading to false positive rewards from flawed reasoning. This impact should be considered.\n3. The experiments are conducted solely on small to mid-sized models. It would strengthen the paper to include results or discussion on whether SRL’s advantages extend to larger model sizes, as this would provide a clearer picture of its scalability and general applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QyXoKrUDnG", "reviewer": "Reviewer_JSqS", "summary": "The SRL framework breaks down problems into a sequence of \"actions\". In each step, the model first generates an \"internal monologue\" and then executes an \"action\".\n\nSRL's reward mechanism abandons sparse final-answer rewards and rigid imitation, instead providing dense signals at every step based on the similarity between the model's \"action\" and the expert's \"action\".\n\nCrucially, SRL only rewards the \"action,\" maintaining flexibility for the \"internal monologue\". This design combines dense signals (addressing the RLVR problem) with flexible reasoning (avoiding SFT rigidity) to promote stronger reasoning abilities.\n\nExperiments show that SRL significantly outperforms SFT and RLVR on both mathematical reasoning and software engineering benchmarks. A curriculum learning strategy of \"cold-starting with SRL then fine-tuning with RLVR\" achieves the best results.", "questions": "1. Can you clarify the difference with works like $R^3$ in the paper?\n2. Can you include more backbone models to make the paper more solid?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. It finds a clever balance between SFT and RL. The core concept of \"rewarding actions, liberating reasoning\" is an important innovation for LLM reasoning training.\n2. Comprehensive experiments were conducted on difficult benchmarks in mathematics and software engineering. The superior performance of SRL → RLVR is highly persuasive.\n3. In-depth analysis (Section 5.2): High-quality ablation studies demonstrate that fine-grained guidance is key (Table 3). Behavioral analysis shows that the model exhibits more flexible, advanced reasoning patterns, rather than simply increasing output length (Figure 4).\n4. Strong generalization ability: Successfully transferred from mathematics to software engineering tasks (Table 4), demonstrating its potential as a general-purpose reasoning framework.", "weaknesses": "1. Dependence on Expert Data and \"Action\" Definition: SRL is highly dependent on high-quality, easily decomposable expert trajectories and the definition of \"actions.\"\n2. Rationality of the Reward Function: The reward function is based on syntax (e.g., difflib) rather than common measures like KL divergence. How can the rationality of this approach be proven?\n3. The primary weakness of this paper's methodology is its lack of novelty, as its core idea significantly overlaps with existing work: (1) Unoriginal Theoretical Foundation: Theoretically, the approach of using expert demonstrations to solve the sparse rewards problem is a classic paradigm in traditional reinforcement learning. Techniques such as Reverse Curriculum Learning and Learning from Demonstration have long been explored and validated in related fields. (2) Significant Overlap with Recent LLM Work: In the specific application of LLM reasoning, the proposed SRL framework is highly similar to the recent $R^3$ paper [1]. $R^3$ explicitly explored using expert demonstrations to train the model starting from intermediate states of the demonstration. Through a reverse curriculum, $R^3$ effectively converts sparse outcome rewards into \"approximately step-by-step supervisory signals\". The SRL paper similarly relies on decomposing expert trajectories into step-wise \"actions\" (what $R^3$ calls \"intermediate states\") and constructs partial trajectories for training. The core mechanisms of both methods are fundamentally alike. So, more discussion and comparison should be included.\n\n\n[1] Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LNadCgmEna", "reviewer": "Reviewer_iZLY", "summary": "The authors present a method for supervised or assisted RL for LLMs.\nThe motivation given for the approach is that for difficult tasks, where the LLM rarely reaches the right answer, naive use of RLVR does not help.\nThus, the authors propose to break the process of generating an answer using the LLM into multiple steps.\nAt each step the LLM is given the problem and a partial answer, and is required to output the only the next step (action) to solve the problem after thinking.\nThe reward for the computed next step is computed as the normalized edit distance between the generated action and the expert predicted step.\nThis assistance or supervision using the partially completed solutions and to only focus on generating the next step helps the LLM learn better.\n\nThe method requires a dataset or expert generations with an action-based formatting, where each example can be broken into a sequence of actions or steps.\nThe definition of action can vary with domain and datasets.\nFor this paper, the authors use step-wise structure of math solutions and environment commands for SWE tasks to distinguish individual actions.\nThe authors also use dynamic sampling to remove training examples with low variance in scores in all cases including baselines.\n\nThe authors show that the method performs well and provides a boost of 3% over RLVR on math tasks (3.7% for SRL+RLVR) on average. While naive RLVR does not yield any improvement, SRL does. SRL also results in 6.4% improvement in the oracle edit task and 4.4% in end-to-end task.\n\nFor qualitative analysis, the authors state that the method allows the agent multi-step thinking which manifests as, outlining all steps in a comprehensive initial plan, on-the fly adjustments, and reflective verification. However, only one representative example has been shown for this behavior. The overall answer lengths are also similar to the base-model.", "questions": "- Can the authors provide a comparison to some other self or expert supervised method that rewards intermediate steps?\n- Did the authors try other reward schemes than string edit distance (like LLMs)? The same information can be presented in different sentences, and sometimes a slight difference in the text can drastically change the meaning; therefore, the edit-distance heuristic may not always hold.\n- The temperature used (1.0) does not seem to be the default to my limited knowledge. If so, how do the average@32 scores change with temperature like 0.6/0.7?\n- What was the final prompt used for inference, because the one mentioned in Appendix B states only one thinking step and outputs only one reasoning step (seems to be for training)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is simple and easy to follow.\n- Based on the motivation, the approach of providing the supervision of first few steps and a dense reward makes sense.\n- The method provides a boost in performance for math and SWE tasks.", "weaknesses": "- Adding comparisons against other methods that use intermediate rewards can be helpful (maybe process reward model (PRM) methods, e.g. [1]).\n- Since the thinking patterns are listed as a contribution, the representative examples showing the interleaved thinking patterns and others can be expanded and analyzed.\n- Since the method heavily relies on the data being in a specific format, there should be some discussion of limitations and expansion to other domains.\n\n\n[1] Lu, J., Dou, Z., Wang, H., Cao, Z., Dai, J., Feng, Y. and Guo, Z., 2024. Autopsv: Automated process-supervised verifier. Advances in Neural Information Processing Systems, 37, pp.79935-79962.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)