BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning
Authors: Xuechen Zhang
Zijian Huang
Yingcong Li
Chenshun Ni
Jiasi Chen
Samet Oymak
Venue: NeurIPS 2025 poster ()
Keywords: small language models
reasoning
policy optimization
chain of thought
reinforcement Learning
GRPO
Primary area: deep_learning
TL;DR: We propose BREAD, a novel and effective variant of GRPO that bridges supervised learning and reinforcement learning by employing branch rollouts from expert traces.

Abstract: Small language models (SLMs) struggle to learn complex reasoning behaviors, especially when high-quality traces are scarce or difficult to learn from. A typical approach for training such models combines a supervised fine-tuning (SFT) stage, often to distill reasoning capabilities from a larger model, followed by a reinforcement learning (RL) stage such as Group Relative Policy Optimization (GRPO). In this paper, we investigate the fundamental limitations of this SFT + RL paradigm and propose methods to overcome them. Using a toy student-expert model over Markov chains, we demonstrate that the SFT + RL strategy can fail completely when (1) the expert's traces are too difficult for the small model to express, or (2) the small model's initialization achieves exponentially sparse rewards as task complexity grows. To address these, we introduce BREAD, a GRPO variant that bridges SFT and RL via partial expert guidance and branch rollouts. When self-generated traces fail, BREAD adaptively inserts short expert prefixes/hints, allowing the small model to complete the rest of the reasoning path, and ensuring that each update includes at least one successful trace. This mechanism both densifies the reward signal and induces a natural learning curriculum. BREAD requires fewer than 40\% of ground-truth traces, consistently outperforming standard GRPO while speeding up the training by about 3$\times$. Importantly, we find that BREAD helps the model solve problems that are otherwise unsolvable by the SFT + RL strategy, highlighting how branch rollouts and expert guidance can aid SLM reasoning.

BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning
Xuechen Zhang* University Michigan zxuechen@umich ch. edu
Zijian Huang* University of Michigan zij ainn@ummmccc ch. edu
Yingcong Li
University of Michigan yingcong@umico ch edu
Chenshun Ni University of Michigan ni chenshhumiiee ch. edu
Jiasi Chen University of Michigan jias fasi@umich edu
Samet Oymak University of Michigan oymakOumich ch. edu
Abstract
Small language models (SLMs) struggle to learn complex reasoning behaviors, especially when high-quality traces are scarce or difficult to learn from. The standard training approach combines supervised fine-tuning (SFT) stage, often to distill capabilities of a larger model, followed by reinforcement learning (RL) stage such Group Relative Policy Optimization (GRPO). In this paper, we investigate the fundamental limitations of this SFT + RL paradigm and propose methods to overcome them. Under a suitable theoretical model, we demonstrate that the SFT RL strategy can fail completely when (1) the expert's 's traces are too difficult for the small model express, or (2) the small model's initialization has exponentially small likelihood of success. To address these, we introduce BREAD: a GRPO variant that unifies the SFT and RI. stages partial expert guidance and branched rollouts. When self-generated traces fail, BREAD adaptively inserts short expert prefixes/hints, allowing the small model to complete the rest the reasoning path, and ensuring that each update includes at least one successful trace. This mechanism both densifies the reward signal and induces a natural learning curriculum. BREAD requires fewer than 40% of ground-trute traces, consistently outperforming standard GRPO while speeding up the training by about 3x. Importantly, demonstrate that BREAD helps the model solve problems that are otherwise unsolvable by the SFT + RL strategy, highlighting how branched rollouts and expert guidance substantially boost SLM reasoning.
1 Introduction
Over the past few years, we have witnessed significant push toward enhancing language model reasoning, which has led to highly capable frontier models such OpenAl ol [17], Gemini 2.5 [19], and DeepSeek R1 [13]. These models can generate longer chain-of- thothh (CoT) traces and utilize more test-time compute to tackle challenging tasks [24]. Despite these innovations, reasoning with small language models (SLM) remains a challenge. For instance, the large DeepSeek-RI [13] has 6711 BB parameters in total whereas the SFT-distilled model sizes range from 1.5B to 70B, and their performance substantially degrades at the 1.5B model (see Table in [13]).
This work studies optimization strategies to enhance SLMs, with emphasis on reasoning tasks. Two popular optimization strategies for training LLMs are supervised fine-tuning (SFT) and reinforcement learning (such as GRPO or proximal policy optimization). Often, an SFT phase is employed, followed by an RL phase. While this two-stage procedure has found success, for SL.Ms, long-context reasoning
These authors contributed equally to this work. Correspondance to (zxuechen.ziianh.oyyma @umicchedu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Supervised Fine-tuning
Reinforc ement Learning
SLM
Fine sunod SLM
O
Ory x
O
O x
SLM
SLM
O-Y X
2rg X
O- Week tumn marcle Sparso reward
Too do
-Expett lnstas Dense reward
Figure 1: High-level overview of approaches. In existing training methods like supervised fine- line-tuning (left), high-quality reasoning traces produced by LLMs are often too complex for SLMs to imitate, so they deliver little benefit and can even hurt SLM reasoning capability. Since the subsequent RL phase starts from this weak starting point, the two-stage SFT+RL procedure often fails. In reinforcement learning (center), when the initial policy generates incorrect traces, the rewards are sparse, causing slow ineffective learning. We propose BREAD (right), which uses part of an expert trace an anchor, then generates additional rollouts that branch from its intermediate episodes. These branched trajectories provide denser, higher-quality feedback, helping SLMs learn robust reasoning strategies. Each dot represents a single episode, and the yellow trajectory is the expert trace.
problems can pose unique challenges due to the misalignment between the expert and student models and potentially sparse rewards.
For example, consider scenario in which each token generated by an expert/teacher model requires the smaller student model produce K intermediate tokens to express it. In other words, the expert thinks and outputs K steps ahead, from the student's 's point of view. In practice, this situation can arise when the expert model a x K deeper/looped version of the student. Naturally, such expert traces might be too challenging for the small model to learn from². On the other hand, success of the RL phase often relies on good supervised initialization during the SFT phase. In Section 2.1, we provide a mathematical setting capturing this intuition and demonstrate that SFT RL can fail for small models (see Figure 3), especially on difficult problems.
To address the difficulties of small models in learning from complex traces during fine-tuning, we propose our algorithm, Branch Rollouts and Expert Anehors for Densified rewards, depicted in Figures and 2. BREAD gracefully integrates the SFT and RL phases by anchoring the optimization process with the expert traces, while allowing the small base model to acquire progressively more flexibility as it becomes a stronger problem solver. Assuming that an expert trace is available (c.g. by querying large expert model), BREAD updates the model with correct trace; however, traces are progressively more self- self-generated.
Contributions: Our specific contributions are as follows:
Methodology: We introduce BREAD: a GRPO variant that generalizes conventional SFT and RL phases through the use of branched rollouts to automatically adapt to the problem difficulty. BREAD induces a learning curriculum along the reasoning trace, thereby densifying the reward signal and tackling compositional tasks see--by-ttep We investigate the theoretical properties of BREAD in a suitable student-expert setting. this setting, we establish how the expert model's trace can uninformative to the student and how the subsequent RI. phase can fail due to sparse rewards. In contrast, we show that BREAD can solve the target prohlem efficiently (low training cost) and concisely (short trace length).
Empirical impact: Our experimental results show that BREAD matches surpasses SFT vanilla GRPO while using 20% of the correct trace tokens. By reducing both the number of rollouts and the total optimization steps, BREAD reduces the overall training compute by R 75% relative to vanilla GRPO. Additionally, we construct slice of difficult problems and demonstrate that BREAD. D, when trained on this set, achieves substantially higher accuracy compared to SFT+GRPO, corroborating its fundamental benefits in solving difficult problems that are otherwise unsolvable by SFT+GRPO. Finally, we provide an empirical study of how branched rollouts from expert hints (i.e., using only parts of the expert trace) densifies the reward signal.
The rest of the paper is organized follows: Section 5 dicusses the related work on language model reasoning and traditional RL methods. Section explains our algorithm BREAD, which also contains the observations inspiring our algorithm design. Section 4 presents and discusses our main experiment results to demonstrate the effectiveness of BREAD. Section concludes the paper and discusses the limitation and future directions to improve in this domain.
II practice, SFT+RL can work well for much of the dataset but might fail on a subset of difficult problems.
See Section 4.1 for empirical evidence and evaluations.

---

Step Regular rollout
Step 2 Episode anchor search (EAS)
If fail
shorten hint
Question
x
If all X
Question
Hin[(oouu-
If Fall X lengthen hint
04
Question
Hin[Drur
Question
Hunl
Quastion
Else
Else, terminate the search
X
Step 3 Policy update
Figure 2: Workflow of BREAD. (1) Regular rollout: Given a question Q, sample group of rollouts. If the sampled rollouts contain correct reasoning trace, use this group of rollouts do the policy updates. Otherwise, go step Episode anchor search: Starting with the whole expert trace (provided by the ground truth from correct responses generated by LLMs) as the search space potential suitable hints, construct hint using the first half of the expert trace, append to the question g, and sample group of rollouts. If all the rollouts are correct, shorten the hint to contain fewer episodes and repeat the process; if all rollouts are wrong, lengthen the hint; otherwise, use the current rollouts to do the policy update.
2 Proposed Method: BREAD
In this work, we propose the Branched Rollouts and Expert Anchors for Densified RL (BREAD) algorithm. We will first describe the algorithm at a high level, followed by a mathematical toy model example (Section 2.1), and the key observations (Section 3.1, Section 3.2) that support its design.
In BREAD, for each question q paired with the answer a, the workflow of BREAD shown in Figure and proceeds as follows:
Regular rollout:
Sample group
rollouts {o.
2. Episode anchor search: If the success rate of the initial group is low (e.g. lower than a threshold), do a binary search to find a short hint that contains the "Expert Anchor" in the expert solution. "Expert Anchor" means the success rate of new sampled output group {o resulting from the question appended with the hint from the expert trace (q, p). is within pre-defined range. Policy updates: Optimize the policy via the following objective:
JAREAD (0)
E (a.na) (m.0)
min (O),( clip -s,11), 3Dx (Follam)))) (1)
0
where
(o P. oist).
mean (R.))
(0)
(2)
Tna(Or,1g, P; ovec)'
std((R.))_.)
The details of the algorithm can be found in Algorithm 1. Next, we will discuss the key observations that motivate the design of BREAD.
2.1 Theoretical Insights into SFT, Reinforcement Learning, and BREAD
Challenges of SFT: It is known that a times deeper LLM can internally simulate L chain-offthhuuhh steps of a smaller LLM [31]. This implies that the expert model can generate dense reasoning trace which necessitates Lx longer simplified trace for the student model to digest via SFT. Otherwise, the student model may fail to learn from SET as we will elaborate further below. Recent work [22] makes related empirical observation that SLMs can struggle to learn from strong reasoners.
Insights from compositionality: Reasoning problems inherently compositional hence need a shain-of-thoughh [39] to solve such problems. Suppose the problem contains 7' steps/subtasks to solve, each with successful completion probability of for the base student model. Then, we receive a final trajectory-level correctness reward with probability of e" assuming steps are independently completed. In contrast, by branching out from the expert trace at step T т. BREAD only needs to complete the downstream steps, increasing the success to e". Notably. by increasing from 1 to T, BREAD facilitates curriculum learning by tackling individual problem steps one at time requiring only (11/c) samples to attain success per step. Below we further formalize this discussion under Markov chain model.
Markov chain model: Let us model the expert and student models through Markov chains follows:
Imagine a Markov chain over the States {0, 1, ..,, K). We envision that the student model is

---

1.0
Tist
166
WEAL
8.6
RROA
C-36
EARA Ca
02
2000
0000
6000
50000 13000
2000
4000
6000
5000
10000
Treining step
Training step
(a) Varying Initialization difficulty
(b) Varying K Number of states
Figure 3: We compare SFT, GRPO, and BREAD according to the Navigation Task described in Section 2.1. K number of states in the Markov chain whereas is the prohability of transition non-favorable states. Our toy model reveals settings where BREAD can succeed while SFT or GRPO completely fail.
forbidden from implementing certain state transitions. Specifically, its connections are local and consecutive state transitions obey [x++1 #r| &
where the agent reaches state I; at time t, and denotes its maximum allowable jump distance of the small model. We consider the learnability of the following navigation task by the student.
Navigation Task: Start a trace from State Obtain a reward of upon reaching State K.
Suppose the expert model solves the Navigation Task by generating the trace Te T 0, $2,.........
K] where 2t E {1,..., , K 1} for 1 & f & the student Markov chain can only learn from Te when the expert transitions (æ,, 1x+11 1) are learnable, ie.. |x, - 1++1| & If no transition satisfies this, the small model cannot learn from SFT.
2TT ",
During SFT,
Without good SFT-induced initialization, pure RL is known to suffer from sparse rewards [37]. In the context of the Navigation Task, once SFT fails, the student model can only improve through vanilla RL if it can generate a trace that successfully runs from 0 to K to earn nonzero reward. In contrast, the student employs BREAD, rather than starting from 0. it can create rollout starting from zT-1- Thus, it only needs to complete much easier task starting from Ty-1, carning reward by reaching the final state 2T K. In the context of compositionality, generation of traces from Tt-1 to corresponds to solving / f'th subtask.
To proceed, have experimented with the Navigation Task controlled by the parameters number of states K + 1, the student model's jump capacity d. and initialization quality Specifically, for each state i, the student model can jump neighbor j * i, | jl d with probability most e. Hence, larger the e, the more model struggles under trace length constraint. Figure demonstrates how optimization of SFT and GRPO can suffer as function of K and e respectively. Importantly, we also display the performance of BREAD which exhibits much more favorable performance. The reader is referred to the supplementary material for full experimental details. To proceed, we provide a concrete theoretical analysis of this discussion in the next section.
2.2 Formal Guarantees under Random Walk Model
Setting and assumptions: The expert trace starts at State O and ends State K by following the trajectory Zo 0, Er K. We assume that K/T > Ft+1 2 K/cT for some > 1 and all f, and that the interval [K KK/TT, cK/T] contains at least one positive integer. In words, the expert model has jump size of Q(K/T).
We assume that the student model initialized as a symmetric random walk. That means, it can only move to left or right neighbor (jump size d = 1 and 0.5) and it is allowed to visit negative states. Thus, optimizing the student model same as choosing optimal neighbor transition probabilities for random walk. In context, the ideal transition rule is always moving to the right (from i to 1).
Finally, we enforee maximum trace length of L and, during RL, the model only receives reward if the trace successfully reaches state K within L steps. With these, we have the following lemma that establishes that SFT on the expert trace followed by reinforcement learning cannot improve the student model with high probability unless L > ((K2).
Lemma (Failure SFT+RL) Suppose the student model a Markov chain with initialized as symmetric random walk. Suppose the expert trace (xx)²=1 always jumps two or more (i.e. K

---

c. T). Given the expert trace and at most N & (4 traces generated during the RL phase, SFT and GRPO training have no impact the student model with probability at least 2e KK K414
In words, this theorem states that the student is not able learn from expert due to limited expressivity and it is not benefiting from the RL phase due to never obtaining nonzero reward. This failure happens unless the trace is long in the sense that L (2(K²). Indeed, this is the time it takes for symmetric random walk (initial student model) to hit State K.
Our next result proves how BREAD can overcome this bottleneck and address this bottleneck while conforming to a shorter trace length. To this aim, examine a variation of BREAD that relies on a simple memorization of successful traces.
Xo
X2
Algorithm: BREAD variant. The algorithm starts from the right most end of the expert trajectory and incremen- tally learns the correct completion. Concretely,
Figure 4: Depiction of the trajectory stitching argument for T 3. Short subtraces (blue, red, green) are those generated each round. The dashed yellow line are the final trace that maps zo to J..-
During the first round, create IID rollouts of length at most L starting from the prefix (=1 Once a reward of is achieved, record the generated suffix S (1/))a/T
At round & T, create IID rollouts of length most L starting from the prefix =-= Terminate a rollout once it hits state Shit in S--1 and complete the trace by replaying S7-1 from Shir- Declare success if the total length is at most I. and set S, to be the union of generated rollout and S, continued from Shit-
This algorithm incrementally learns sub-trajectories and stitch them to eventually obtain a full trajectory ST that solves the entire problem. This is depicted in Figure 4. During inference, the model simply replays the final memorized trace Sr (yellow dashed curve in Figure 4).
We have the following theorem that establish the sample complexity of this BREAD variant.
Theorem (Success of BREAD) Suppose L > 322 /T and assume BREAD generates rollouts at each round & T & T. Then, with probability at least 1 - Te- BREAD succeeds in all T rounds in creating Sy while never violating the maximum trace length Thus, BREAD memorizes successful student trace ST with length at most L with the same probability.
A few remarks are in place: First, when LxcK2/T /T, BREAD succeeds by using total of Si( log (T)) traces over T rounds whereas SFT+RL from Lemma fails with probability at least 1 - 2c 2e-02( Q(() confirming our intuition on reward densification benefits. Secondly, suppose TocK i.e. the expert model has constant jump size. In this case, BREAD solves the problem in linear time as it requires L Z OKK). In contrast, SFT+RL requires र O(K2) to solve the problem. In the next section, we discuss how these theory and insights are in line with the SFT and RL performance on real reasoning tasks with state-of-the the-art models.
3 Empirical Insights into BREAD
3.1 Limitations of SFT and Reinforcement Learning for SLMs
RL-only evaluations: We begin by assessing how well RL works with SLMs in isolation. Experi- ments with vanilla GRPO [33]. displayed in Figure 5, show that it merely sharpens the capabilities model already exhibits. GRPO relies on sampling a mix of good and bad traces. But when a small base model fails to produce any high-quality trace-nearly -nearly half of the queries in our evaluations in Figure 5a-Icarning stalls due to lack of reward signals (Figure 5b). Related limitations are noted for RL with Verifiable Rewards of [42], which struggles to elicit fundamentally new reasoning patterns.
SFT-only evaluations: SFT can introduce new knowledge into the model. However, discussed earlier, traces generated by much stronger models can be too complex for SLMs to learn resulting in poor initialization for RL. To demonstrate this, we start from 1.een..5-1.B-IInstru and Qwen2.5- 3B-Instruct base models and fine-tune them with 1000 difficult questions, paired with reasoning traces generated by Gemini Thinking Experimental [1]] and Deepseek-R 1 [13], following [35]. These reasoning traces were previously shown to improve the reasoning capability of large models such as Owen2.5532B-Innste Instrrct and Qwen2.5-14B Dwven2.--4BBIInstruce by [24]. However, in our experiment results shown in Table 1, when SFT was performed on small models, accuracy actually decreases. For example,
5

---

bwcchbolvecco zone
MMA / NMW
9.5
100
3.45
0.4
05
S180
2100
Number of samples where all rollouts fail
(b) Validation accuracy over training steps
Figure 5: Issues with vanilla GRPO SLMs. Among 256 samples in batch, for nearly half of them, the base model's 8 rollouts produce no correct trace. The absence of reward inhibits further performance gains and validation accuracy stalls.
Increasing Curriculum Difficulty
0.7
Base
Along
GSMBK - MATH - AIME + IMD
Correstees Baite
Ru:40 % Esport timk)
data
0.5
hard
Along
expert
0.4
reasoning
trace
0.3
trace
0.2
casy
medium
0.0
0.2
0.6
0.0
0.8
(a) Illustration of hint ratio (how much of expert
(b) Accuracy for different ratio of hints, appended to the inference query.
trace is provided) and curriculum learning.
Figure 6: (a) Math henchmarks may show difficulty tiers, but ranking individual problems for curriculum training still challenging. Instead, BREAD can adaptively adjust the difficulty by automatically choosing the branching point. For easier questions, it would provide no shorter hints (smaller hint ratio). (b) Model accuracy increases as longer hint is appended the query. Dataset is first 200 questions NuminaMath-CC The blue dots are the base model (Qwen2.5 --3-5-3---sssttucc and the orange dots are RL finetuned using traces containing 40% hints.
accuracy dropped from 0.257 to 0.177 on the GPQA dataset when the 1.5B parameter model was fine-tuned on S1K traces, and dropped even further 0.121 when fine-tuned on the more verbose S1K-1.1 data. This corroborates our central intuition: when traces used for SFT exceed an SLM's learning eapacity, they can hurt the model performance rather than helping.
These findings-thal for small models, RL alone, or SFT + RL, are insufficient- motivate BREAD which uses expert traces directly in the RL phase instead of relying on SFT to learn them first.
Qwen- -.5t---rutt
Onvet-3B-Instruct
Dataset
Base
SFT (SIK)
SFT (SIK-IKIJ)
Base
SFT (SIK)
SFT (SIK-I.J)
MATH [15] GPQA [28]
0.494
0.426
0.408
0.624
0.616
0.630
0.258
0.177
0.121
0.369
0.247
0.288
Table 1: Accuracy of small models decrease after SFT. The base SLMs Qwen2.5- IS--trrtuttt and
Qwen2.5-33 B-3--ntrrutt the datasets used for SFT are SIK and SIK-1.1 [24], with responses generated by Gemini Flash Thinking API [12] and DeepSeck-R [13] respectively.
3.2 Why BREAD: Expert Traces Provide Critical Guidance and Curriculum for RL
While Section 3.1 shows that an SLM can struggle to imitate traces generated by powerful LLMs, we posit that the SLM can still understand them well enough to extract useful information, namely by using part of the complex traces as hints to lower the problem difficulty. To illustrate this, we define the "hint ratio" as the fraction of the expert trace used (in terms of number of episodes), as illustrated in Figure 6a. A higher hint ratio means more guidance for the model. In Figure 6b, we plot the model accuracy for different hint ratios, where the hint is appended to the inference query. We can see that our intuition is correct: even providing part of the hint in a very simple way (by appending to the inference query) helps improve accuracy. In contrast, Figure shows that SFT only on the same traces gets very limited improvement and can even hurt. BREAD is motivated by this observation and uses expert hints in a more sophisticated way, by integrating them directly into the RL create denser rewards.
We can also view BREAD through the lens of curriculum learning- As shown in Figure 6a, as we branch out from the expert trace earlier, solving the problem becomes more challenging. The Episode Anchor Search (EAS) step BREAD automatically adjusts the branching point, producing a self-paced curriculum. In this sense, BREAD serves as generalized saaicuulummeaarnnn framework

---

which (1) allows us to utilize SFT data within RL, (2) adapts the optimization to the problem difficulty. and (3) generates dense rewards by creating curriculum along the reasoning trace. By branching along the expert trace at adaptive cut- cut-points, BREAD automatically surfaces the most informative training samples and tunes their difficulty on the fly. This relieves from hand-crafting data-level schedule and lets the model discover its own curriculum.
final surprising observation from Figure 6b is that conditioning RL training on questions with partial hints not only improves performance on hinted queries (e.g., orange dots with hint ratio 0.3 and above), but also improves accuracy on questions without any hints (i.e., orange dot with hint ratio of 0). This highlights the model's 's ability to generalize from partial short traces to full long traces.
Algorithm BREAD: GRPO with Branched Rollouts and Expert Anchors (see Algorithm 2 in Appendix for full version.)
Require: Dataset D, current policy no, sampling number G, a list of keywords for splitting episades
[Wo, W2,.... wa]
1: procedure BREAD(D, na, [. w,,..., step 2... do
3: Sample a batch Dd from
4: Update the old policy model Told To
5:
for each question and expert solution pair (Q,S) in D3 do
6:
[R, R2. ...; Ra],p EAS(Q. S, ma, G, [wo, W2, w.]
Episode Anchor Search (EAS)
Add the p). R; the buffer. 8: end for
For each (Q, o, in the buffer, compute Ar,: for the t-th token of 10: end for
9:
t- Equation (2)
for iteration 1, ..., T do
12:
Update the policy model my by maxmizing BREAD objective 13: end for
Equation (1)
14: return Te
15:
end procedure
procedure EASIQ, Q, n. G. [wn, W2, W2,... wy])
17:
[R, Ra. ..., Ro] - TQ)
>- Sample I responses for question Q with current policy ceartte_oorret_ppobbbbie
18:
Peomect
Re.... Ra], S)
19:
compute correct probability based on responses and the gold solution
20:
if0 < Poonec then
21:
return RR, Ra, Rc], NA
22:
else
23:
[Co, [lo, ex] - split Settt_eppsdee(SS [wo, u2, W2,..., ws]) return BINARY SEARCH AND GENERATE( [eI,,,,, ex], K)
24:
25:
end if
26:
end procedure
4 Experiments
Baseline algorithms. We evaluate the effectiveness of our method BREAD (as described in Algorithm 1) on mathematical tasks, which can be easily adapted to other reasoning tasks such as coding, commonsense reasoning, etc. We compare with the following baselines:
GRPO: Standard GRPO.
SFT: Standard supervised fine-tuning the full training set. We denote SFT various data splits as SFT (X), c.g. SFT(full) for the whole dataset, SFT (diffificuut)) for the hardest subset, SFT (FT(ronom)) a size- matched random subset, and SFT (Feeececed)) for the EAS-chosen subset. The precise discussion detailed with the experiments results.
SFT + CRPO: This means SFT is run followed by GRPO. In other words, GRPO continues from the final checkpoint of the same SFT run. We denote GRPO fine-tuning initialized from SFT model trained on data GRPO.
split X as SFT
CRPO w/ Expert Trace: During GRPO training, the entire expert trace is injected an additional rollout for all queries. This is strong baseline contains the expert. Details in Appendix C.
Training settings. We adopt the verl framework [34] for training. We utilize the Adam optimizer [20] with constant learning rate of x 10-6, For rollout, the prompt batch size is 256 and we sample 8 responses for each prompt. For training, the mini-batch size is 64. To find the appropriate branching point during the Episode Anchor Search (EAS) step of BREAD (Section 1), we first split the expert
7

---

trace into sentences (on easier datasets like MATH [15], where the expert traces are generally short) or into paragraphs (on harder datasets like NuminaMth-Cor [21] and OpenRI-Math-251 [9]). Then, we aggregate the split partitions into K 10 episodes evenly, and proceed with binary search. See details in Appendix B.3.
4.1 Main Results
B BREAD outperforms all baselines in terms of test accuracy. Figure 7 plots the training curves of all methods on the NuminaMath-Cor benchmark, starting from the Qwen-2.5-23 -.--3Bt-nstt base model. In Figure 7a, we visualize the test accuracy against training steps for BREAD and the baseline methods. BREAD outperforms all the baselines. Let us discuss each bascline in turn. BREAD improves final accuracy by more than 15% over vanilla GRPO. SFT can offer a stronger starting point for RL, which we can see by comparing the SFT and SFT+GRPO curves, but BREAD is still better. (Note that training SFT for more iterations does not further improve performance, see Appendix B. SFT (Difficult) represents SFT trained on the hardest questions from NuminaMath- CoT, obtained by sorting samples by solution length. Intuitively, problems that require longer solutions are typically more complex and involve more reasoning steps. Comparing the SFT (Diffi fifffi) and SFT (Difficult) +RPPO curves, we can see that SFT with too complex expert traces can even hurt the performance of SLM, which further hurts the later GRPO stage. Finally, from the GRPO w/ Expert trace curve, we see that adding the expert trace as an extra rollout during GRPO mitigates sparse rewards and shows noticeable performance improvement, but it is still worse than BREAD. We suspect this is because there is a distribution gap between small and large model's reasoning traces, which makes SLM imitation of large models hard, while BREAD can reduce the gap during learning by letting the SLM figure out the reasoning steps by itself more. See further discussion below on "hard questions" for elaboration. We also provide more results among different base models and datasets in Table 2 and Appendix B.
BREAD reduces training time. BREAD is also markedly more training-efficient. It can reach the accuracy of the best baseline, SFT+G in just 25% of the training steps (Figure 7a), translating to roughly a 75% reduction in total compute FLOPs (Figure 7b). We estimated the FLOPs based on the common cost evaluation method used in recent scaling-law studies [35, 16, 30J, by counting a forward pass as 2N and a backward pass as 4ND. D. Here N is the number of model parameters and D is the total token count processed in that pass. For calculation details, see Appendix A.1. Note that we estimate D as the average length of all expert traces in the training set, but according our measurements, BREAD's actual generation length is always shorter. Therefore for BREAD, its actual token count, and thus its FL.OP cost, is even lower than the estimated values reported Figure 7b. We also discuss the potential of reducing the number of rollout needed reduce training cost. in Appendix B.2.
Test ancursry
0.55
0.55
APTICES CPCC
Tost
0.50
3.50
0.45
1.00
150
260
250
360
0.0
0.5
1.0
2.0
2.5
Training step
Estimate PLOPp (80
349
(a) Test accuracy over training steps
(b) Test accuracy vs estimated FLOPs
Figure 7: Test accuracy over training steps (left) / /LLPPs (right). BREAD, which adaptively uses hints from expert traces during GRPO, significantly improves SLM reasoning ability compared to all baselines. The gray dashed line (max accuracy of the best baseline) demonstrates that BREAD can speed up convergence speed by about 3x. Both figures share the same legend.
BREAD succeeds on the hard questions, where other baselines fail. To understand the gains of BREAD, we train and evaluate each method exclusively on very difficult problems. The goal is to investigate whether expert hints in BREAD can help SLMs learn new information particularly from these hard questions. We conduct the experiment on the NuminaMatthCCo dataset and the Breen255-3--Instraee
as the base model. To build the hard dataset, we first run ordinary SFT and from the training dataset, we select the 500 questions for which three independent generations produce no correct trace (pass @3 0). These tasks are unsolved by the small model, and their expert traces
8

---

1.0
5.2
11.00
2.8-
5.10
6.08
Salve SHOE
0.05
0.00
5
1.0
155
20 25
40
1.0
1.5
20
25
30
35
00
1reating stap
Touining step
(a) Test accuracy over training steps
(b) Ratio of questions whose rollouts are all wrong
Figure 8: (a) Test accuracy very hard questions over training steps. BREAD outperforms other baselines significantly while the traditional methods (SFT, CRPO, SFT+GRPO) learn littl. (b) Proportion of test questions for which every rollout fails (solve-none ratio). In vanilla GRPO, the ratio stays persistently high, signalling that training stalls under sparse rewards. In BREAD, the solve-none ratio for regular rollout starts similarly high, but EAS injects expert traces and densifies the reward. This rate continuously decreases, confirming that the model is learning.
proved too complex for SFT learn. The 500 samples were split into 80/20 train/test subsets. We then again used (wwen.55-3B-Instrrr
as the base model and trained each method on this hard subset. The results are shown Figure 8a. All baselines show little or no improvement on the test set after training on the hard questions. In contrast, BREAD achieves a clear upward performance with continued training, demonstrating that partial expert guidance and branched rollout strategy can provide learnable information, even when standard SFT and vanilla GRPO fail. We argue that BREAD succeeds because it adaptively reduce problem difficulty and densifies the rewards. As shown in Figure 8b, BREAD sharply lowers the solve-none ratio, getting more informative samples and richer feedback. This enables BREAD to learn effectively even from very hard questions and complex reasoning traces.
BREAD improves sample efficiency during training. SLMs distilled via large-scale SFT can achieve strong reasoning capability, such as SeepSeek-R1-Disce [13]. However, the distillation pipeline prohibitively expensive. For example, this model is trained with 800k samples from both reasoning and non-reasoning domains curated with DeepSeek-R1, containing 671 billion parameters. A single forward pass through such a huge model already costs more FLOPs than 25 RL training steps with 8 rollouts. The pipeline also needs expensive sample filtering and trace post-procsssing, like the heavyweight data-collection procedure in [24, 22]. What makes things worse is that, as illustrated in Table 1, each target model requires training with different samples, multiplying the cost.
In contrast, BREAD is far more sample-efficient, achieving comparable gains with a small fraction of the expert trace and without any heavyweight sample selection stage. To show this, we created two trace-budget-matcher baselines, SFT (selected) and SFT (random). For fair comparison, we first record the expert traces actually requested by BREAD via Episode Anchor Search (EAS). On NuminaMath- Co',, this corresponded to 36.7% of samples, and on the easier Math [15] dataset the fraction falls to 19.15 99.1% (during 300 training steps of Qwen-2.5 2.5-3BB-IInnttte fine-tune base models with the same number of traces To create SFT (selected), we select the exact subset chosen by BREAD. To create SFT (random), we
We then supervised
equally sized randomly picked subset. Finally, we created SFT (full), which uses all the expert traces, take a high cost. Each of the SFT (x) phases followed by GRPO phase. As the results in Table 2 show, BREAD outperforms nearly all the SFT (X) + GRPP baselines in terms of accuracy, and can even approach the performance of the expensive DeepScek-RR -1- R1-Distill model. Notably, while DeepSeek-RI. RI--istill gains from vast and diverse training data, BREAD achieves nearly comparable accuracy without requiring this data. We also observe that small models do not necessarily benefit. from SFT with expert traces, evidenced by the Drren-2.511.5--Insttte
run on NuninaMath-Coi where SFT (full) +CRPO has relatively low accuracy. Also, which samples are most useful for SFT is uncertain: on the MATH dataset, SFT (selected) +GRPO has higher accuracy than SFT (random) + +RPO with Qwen-3B-Instree but is the opposite for
5 Related Work
Supervised fine-tuning (SFT) and Reinforcement Learning (RL). Recently, there a debate about whether SFT or RL can truly improve the reasoning ability of language models (LMs). With the emergence of [33, 13, 36), more and more reinforcement finetuned reasoning models demonstrate the importance of RL. in reasoning tasks. [6] prove that RL can enhance the reasoning ability of LMs

---

Modd
Dataset
GRPO
SFT(hil)+CRPO SPTirandom)+GPP
BREAD
DeepSock-Disill
Qwen- 1.5B Instract
0.590
0,706
0,818
Owen-3B-Instuud
MATH
0.681
0.846
0.235
0.794
0.843
Qwen-1 1.3B-Instract
NeminaMath-Com
0.347
0.242
0.356
0:234
0.361
0.368
Qwen- 3B Instruct
NuminaMath Col
0,425
0,537
0,519
0,502
0.647
Table 2: BREAD outperforms other baselines and nearly reaches the accuracy of DeepSeek-R
which
has the benefit of vast training data, There is no MeesSee--R-Dishtiie for Qwen-3B provided by [13], so its cells are left blank.
while SFT can only force LMs to memorize knowledge. However, [42] argues that the base model already has the reasoning ability while Reinforvement Learning with Verifiable Rewards (RLVR) barely increases the probability of correct reasoning trace. Furthermore, distillation works such as [13, 24] prove that SFT can help SLMs acquire reasoning capabiliity comparable to the expert/teacher models. While the current popular pipeline to train a reasoning LM is SFT followed by RL., we argue the need for stronger integration of SFT and RL because, for hard questions, the expert solution might not be suitable for base models to learn while RL can struggle to discover even single correct trace.
Efficient RL and Reasoning, While reasoning LMs become more powerful, computation is more demanding during the training and deployment procedure. Therefore, researchers recently paid more attention to efficient reasoning in both directions. Pivotal Token Search (PTS) [1] accelerates the Direct Preference Optimization (DPO) [27] training by identifying tokens a language model generation that significantly impact the probability of success for the reasoning task. Following the
memory efficiency but training time inefficiency introduced by GRPO [33, 13], many uollow-up works improve the training convergence speed, including DAPO [41], CPPO [23], PODS [40] and DUMP [38]. [36, 43, 2] saves the token usage during inference by training with one or multiple levels of length penalty. Meanwhile, meta reinforcement fine-tuning (MRT) [26] makes inference token wasted less in meaningless reasoning steps by making the success rate steadily increase with the number of reasoning episodes. There are also orthogonal approaches to efficiency e.g. by forming cascades of small and large language models [5, 44, 14]. Compared with these previous works, we not only decrease the supervised signals during training by only introducing an expert solution when the current model cannot solve the current task with a probability relatively high enough, but guide the model with the expert hint to increase the RL training efficiency. which can provide denser reward for speeding up the RL training procedure.
Related classical RL methods. DAgger [29] intermittently injects expert actions to correct agent behavior. Instead, BREAD adaptively inserts expert hints only when the agent fails and allows the model to complete the remaining of the reasoning trace, allowing for natural curriculum. Go- Explore [8] trains an agent that can solve all Atari games by branching from intermediate states, while BREAD branches out from expert traces. Methods like Hindsight Experience Replay (HER) [3] and potential- based reward shaping (PBRS) [25] densifies learning signals under sparse rewards through goal relabeling external shaping functions, but BREAD can achieve a similar effect through hint-based rollout initialization. Finally, while prior works like Kickstarting [32] and Expert Iteration [4] explore teacher-student transfer via full trajectories or alternating control, BREAD explicitly focuses on sparse expert intervention with minimal demonstrations. Some earlier work, such as Model-Based Policy Optimization (MBPO) [18], propose automatically update the predictive model during policy training and only learn through short rollouts, but does not include the hint from expert traces, which will lead to the improvement limitation in LLM post-training domain.
6 Discussion and Limitations
Our work introduces BREAD as a principled algorithm to densify the reward signal using branched rollouts from the expert trace. BREAD significantly enhances sample complexity, training time, and eventual accuracy over employing GRPO. It is also theoretically well motivated and allows the model to overcome fundamental bottlenecks of SFT+GRPO. BREAD has also few limitations: Firstly, we focus on SLMs and assume that there is a strong expert/teacher model to provide high-quality traces. This assumption can be weakened as we can use BREAD whenever successful traces are obtained and use standard RL otherwise. Secondly, it is possible that SLM may fail to obtain a reward signal even from partial traces of the teacher. In the extreme scenario, the student model can't 't even conclude even if the full solution by the expert is presented as it can't 't follow the argument at all. This shortcoming could potentially be addressed by training or prompting the expert model to generate simpler more digestible traces. We leave these as future directions.
10

---

Acknowledgements
This work is supported by the National Science Foundation grants CCF-2046816, CCF-2403075, CCF-2212426, the Office of Naval Research grant N000142412288 and an Adobe Data Science Research Award. The computational aspects of the research is generously supported by computational resources provided by the Amazon Research Award Foundation Model Development.
References
[1] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08900 2024.
[2] Pranjal Aggarwal and Sean Welleck. LI: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697, 2025.
[3] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[4] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
[5] Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05777 2023.
[6] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoe Le, Sergey Levine, and Ma. Sft memorizes, el generalizes: comparative study of foundation model post-training. arXiv preprint arXiv:2501.171611 2025.
[7] Rick Durrett. Probability: theory and examples, volume 49, Cambridge university press, 2019.
[8] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):500-588 2021.
[9] Hugging Face Open rl: A fully open reproduction of deepseek-rl, January 2025.
[10] William Feller, An introduction to probability theory and its applications, Volume 2, volume 2. John Wiley & Sons, 1991.
[1]] Google, Gemini 2.0 flash thinking mode
2024. https:
/ //cloud. //cloud.googl. wwt. cww//ootte/ htr//vereeexaaae htt.teeea//geeeeaaeeeeaaaaaaaa/e
[12] Google. Gemini 2.0 flash thinking mode (gemini -2. 1-fllash-thinking-eeee 111111199) De- cember 2024. Accessed 15 May 2025. "Last updated 14 May 2025" on the page.
[13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[14] Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations, 2024.
[15] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
[16] Jordan Hoffmann, Scbastian Borgcaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15555 2022.
[17] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Bl- Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1672 2024.

---

[18] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32. 2019.
[19] Koray Kavukcuoglu. Gemini 2.5: Our most intelligent ai model. http:://bloo.goolll technology/gooe htog//ooolleeoooo deepmind/ htt///eminiii. modell seeeeltthikingg updates popdates-mach-20-00 March 2025. Accessed 15 May 2025.
[20] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.
[21] Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau. Guillaume Lample. and Stanislas Polu. Numinamath. [https: / http///ggggnnggc.eeeee wow.AI--O///umaaaaaaaaaaaaa-aa-a project- - nititinmminna//// aimo- progoopporresss-e wwtnggrss--pizzeelooo heriize/bbbbbnne http://////oo/mm//////////////// dataset. pdf), 2024.
(https://gi com/
[22] Yuetai Li, Xiang Yue, Zhangehen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ra- masubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. arXiv preprint arXiv:2502. 121443, 2025.
[23] Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-based reasoning models. arXiv preprint 2:X:222503.22342, 2025.
[24] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501.19393 2025.
[25] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. In lcml, volume 99, pages 278-287. Citeseer, 1999.
[26] Yuxiao Qu Matthew YR Yang. Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine-tuning, arXiv preprint arXiv:2503.07572 2025.
[27] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-5377 2023.
[28] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. In First Conference on Language Modeling, 2024.
[29] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings. 2011.
[30] Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.00445 2023.
[31] Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint 250. 2025.
[32] Simon Schmitr, Jonathan Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Woj- ciech M Czarnecki, Joel 7. Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning. arXiv preprint
2018.
12

---

[33] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint #xXv:24402033300, 2024.
[34] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu, Hybridflow: flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
[35] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint urXiv:2408.000 03314, 2024.
[36] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint 1rXiv:2501.12599 2025.
[37] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817, 2017.
[38] Zhenting Wang, Guofeng Cui, Kun Wan, and Wentian Zhao. Dump: Automated distribution- level curriculum learning for rl-based Ilm post-training. arXiv preprint MrXiv:2504.09710 2025.
[39] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. ohain-of-thouglee prompting elicits reasoning in large language models. Advances neural information processing systems, 35:24824-24837, 2022.
[40] Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not all rollouts are useful: Down- sampling rollouts in Ilm reinforcement learning. arXiv preprint arXiv:2504. 138/8, 2025.
[41] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning 2025.
system scale. arXiv preprint ar 2rXiv:2503.14413
[42] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprini arXiv:2504. 3837, 2025.
[43] Xuechen Zhang, Zijian Huang, Chenchun Ni, Ziyang Xiong, Jiasi Chen, and Samet Oymak. Making small language models efficient reasoners: Intervention, supervision, reinforcement. arXiv preprint PxXi:2505.009611 2025.
[44] Xuechen Zhang, Zijian Huang, Ege Onur Taga, Carlee Joe-Wong, Samet Oymak, and Jiasi Chen. Efficient contextual IIm cascades through budget-constrained policy learning. Conference on Neural Information Processing Systems, 2024.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] is perfectly acceptable to answer "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We list our contribution in the end of the introduction section Section 1, which
are supported by later methodology Section I and experiment sections Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We discuss the limitation in the last section (Section 6).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sele--peeiiiainnn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: In the Section 2.1, we define our theretical model and proves our claims in details.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe the main steps to reproduce our experiments including algorithm, dataset and base models Section and Section We also provided details about algorithms and hyperparameters in the Appendix.
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include our code the spplementary materials and will also provide checkpoints to make sure experiments are reproducible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /hips.cc/pppii wttpccccppccccpeeececeecccc hwwppuuguuuuuu...popppppppeaaaee gwt////ideeeeeee/ege//ee/eeeeee/ iot.e//mmmmiiiiiioooooo..ooo.o.. ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experiment details are stated in Section 4 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We use several large datasets with several base models to demonstrate the generalization ability our methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide the computation resources needed in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:://narrp.....pbbb...
wttpiiss..cpppppp wttptttttiiiiillleeeeeeeeaaaaaee ines?
Answer:
[Yes]
Justification: This paper does not contain anything that harmful potentially.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impact which is also potential positive ssocietal impacts in Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
18

---

Justification:
This paper does not have negative impacts potentially.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We correctly cite all codebase and datasets that used during our experiments in Section and Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www. datasatsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We include our code for data generation, model training and inference and result aggregation in the supplementary zip file.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
19

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not relate to human subject study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not relate to human subject study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The development and design of our method does not involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M policy (https //tt://ne.....e ce/ cc/Conferencess/222 /LLM) for what should or should be described.
20

---

The Supplementary Material is organized as follows:
In Appendix A, we explain the calculation of estimated FLOPs (Appendix A.I) and addi- tional details about the training and evaluation of BREAD (Appendix A.2).
In Appendix B, show that SFT further cannot improve the performance of models (Appendix B.1), BREAD can reduce the number of rollouts required during the sampling stage training (Appendix B.2), and the distribution of the step number of expert solutions (Appendix B.3).
In Appendix C, we explain the baseline GRPO w/ Expert Trace in detail and compare to RL with SFT loss.
In Appendix D, present detailed formulation of the Markov model introduced in Sections 2.1 and 2.2, and provide concise proof of related theorems.
Algorithm 2 BREAD: GRPO with Branch Rollouts and Expert Anchors (full version)
Require: Dataset D, current policy ng. sampling number G, list of keywords for splitting episodes
[Wo W2, .... w.]
1: procedure BREAD(D, na, G, [wo, W2, ..., wrl
2:
for step 1, ..., do
3:
Sample a batch D, from D
4:
Update the old policy model Told ne for each question and expert solution pair (Q, S) in D3 do
5:
6:
[Ri R2, ..., Ra] p EAS(Q, S, Tu, G, [Wp, w2,..., wJ] Episode Anchor Search
(EAS)
7:
Add the (Q,p), R2 the buffer. end for
8:
For each (Q,p), o, in the buffer, compute Ai, for the t-th token of og 10: end for
9:
Equation (2)
for iteration 1, T'. do
12:
Update the policy model by maxmizing the BREAD objective end for return Th
t Equation (1)
14:
end procedure
16:
procedure EAS(Q, G. [W, 02, W2...., w)))
[R1, R2,... RG] - (Q) >- Sample responses for question Q with current policy m
18:
Poonect wwwpute_mmooetoccce wwwrectrttccc.. prcccct__oobbbbbit((A.. R2, R2...., R)
19:
t> compute correct probability based on responses and the gold solution
20:
if Pronect then
21:
return [Rg R2g ..., Rgl, NA
22:
else
23:
[co, CI..... ex] ] - split Settt__ppioee((SS [wp, W2, W2.,... wr]) return BINARY SEARCH AND GENERATE(Q, G. [2,, e2.. ex],
24:
25:
end
26:
end procedure
27: procedure BINARY_ SEARCH C.
e2, G.. ex] I,, R)
28:
M L+R
29:
e2, [R, Ro. Rol e,.., ewl)
30:
Promrct - cwmputee_coocccccccccc
conpute_correct. prarrecttroobbbiltty((B,, Re,..., Ra),S)
31:
if Prommct then
32:
return RR. R2,.., Ra],St.M
else if Peonrct then
34:
== M, R = R, M = L+R
return binabynnaaaa cearry__eerhhhee wtw.aaaachaannneeeeeteeeeeeeeeee S. ex]
ex,
36: else
L L M,M 2
38:
return Dinary_searcoe and gonerate(Q.S,....
Q, G. ex R)
end if
40:
end procedure
21

---

A Experiment details
A.1 FLOPs
We compute the estimate FLOPS following [35, 16, 30J. The supervised finetuning, which include one forward and one backward phase, people use common approximation 6 [16], and for inference, which include only one forward phase, people always use 2ND [30]. Here N represents model parameters, D is the total token count processed in that pass. So a forward phase takes 2N D while a backward phase take D. For estimation, we define the average length of a single question in one inference time as D sample so D D sample X Prolloou for RL. and D sample for SFT. For estimation, we define the average length of a single question in one inference time as Dammpl so D Dample Thallout for RL and D D sample for SFT. GRPO with eight rollouts, Mmllout 8 including eight forward and eight backward phase needs x 8 N rollout BREAD will take more forward pass to do the binary search, includes including eight forward and nine backward phase, so we use x 8 N Drolouu + + N Drolout We estimate D with the average length of all expert trace in the training dataset. We truly record the additional number of generation D mnditionnl = Deallout X Radiitiioaa rellout- Notably, the generation length of BREAD is always shorter than expert trace and vanilla GRPO. So our BREAD can reduce more computation resource compare to 75% shown in Figure 7b.
6 8 N pollout + 4N additionll- GRPO w/ Expert trace
A.2 Experiment Setting Details for BREAD
In addition to Section 4, we list additional details of our experiments here. During training. set max_ fexx_prompt__eneee = 2048 for MATH experiments and max_proopt__eeeeeeee NuminaMath-CCT experiments. For both training and evaluation, we max max_response__eme _eeeethh_hhhheeeeeee 4096 for MATH experiments and max response_lengee ments. During training, we set the number of rollouts as 8. We used a low-variance KL divergence and set the coefficient for KI. divergence as 0.001. We set the temperature at 0.6 for both training and evaluation.
= 4096 for
= 8192 for NuminaMath-CofT experi-
For the implementation details of the episode splitting for the expert trace during training, we split the expert trace with sentences separated by " " "o "\n'' for MATH and paragraphs separated by "An\n" for NuminaMath-Can -ooT. Note that the expert traces can be either human provided solutions or correct solutions provided by larger expert models, and the splitting method here also be changed to split according to specific keyword list. After splitting, we aggregate the traces into 10 episodes evenly for all expert traces. The reason why implement this way is that want to make sure that the number of episodes of the expert traces is not too large, which can guarantee that the EAS step does not take too much time.
During training, assume we have a question Q and an expert hint p ( or without p during inference), the template of the prompt is as follows:
{ content'; '<lim_starrs (syytemm Ynou are a helpful assistant. You first thinks about the reasoning process in the mind and then provides Show your
the user with the answer <limmend/>nee / work in <think> <\think> tags. And return the final answer within [boxed(]: im end haslslimssttttt sessst//asssstaattt AnLet me solve this step by step. \ 'role': 'user'}
For the hardware requirements, all of experiments are done with L40S 40GB GPUs except the training starting from Qwen2.5-3B 5-3-5---r-B--t as the base model, which requires 8 80G H100 GPUs.
B
Additional Experiments
B.1 SFT training saturates and does not help after a while
Supplement to Figure 7a. As shown in Figure 9, training SFT for more iterations does not further improve test accuracy. other words, we already use the model that SFT can achieve. For a fair comparison, we therefore use the 300-step checkpoint as the starting point for SFT+GRPO.
22

---

0.51
SPT
Test accuracy
0.52
0.51
0.49
50
100
150
200
250
300
350
400
Thiining
Figure 9: Test accuracy SFT over training steps. The accuracy doesn't continuously increase after the number of training step we use.
B.2 BREAD reduce the number of rollouts needed
Another straightforward way to reduce training cost is to reduce the number of rollouts, since FLOPs scale linearly with that count. However, for vanilla GRPO, fewer rollouts lead to lower accuracy, whereas BREAD maintains its performance even as the number rollouts decreases. The result is shown in Figure 10. As have shown in the main body, when the task is difficult that the base policy rarely produces correct traces, vanilla GRPO fails. To study rollout efficiency under conditions where GRPO can learn, moved to the MATH dataset with 2.wen-2.5-3B-Insttuue base model. With eight rollouts, GRPO does increase accuracy. However, performance will decrease the rollout budget is reduced from to 5. In contrast, BREAD reaches the same accuracy with just five rollouts, cutting training FLOPs while preserving performance.
0.69
0.60
Teat sccuracy
0.64
BREAD(5 rollouts)
GRPO(5 rollouts)
0.62
CRPO(8 rollouts)
Estimate
Figure 10:
Test accuracy over training steps with different number of rollouts. The gray dashed line shows the final accuracy of vanilla GRPO with rollouts. The total training step is 500.
B.3 Episode details
As described in Appendix A.2, we plot the distribution of the number of steps for our two datasets before episode aggregation. As shown in Figure 11, we can see that that most expert traces contain less than 20 steps, while few of them contain much more steps, which may slow down the training sif there ised if there is no episode aggregation.
B.4 BREAD can reduce the number of rollouts needed
We also conduct experiments with LLaMA3.22 Inntrruutt model. The results are shown in Table These experiments support that the method generalizes beyond the Qwen family and is not model- specific, and the effectiveness more significant when the task harder.
B.5 BREAD compare to other methods.
Regarding recent works, we would like to highlight that our focus is on challenging tasks where existing methods like DAPO [41] are not specifically designed to perform well. So, these tasks are
23

---

Distribution Solution Steps
Distribution
4000
150000
Feegueryy
125000
200000
2000
75000
50000
25000
50
100
200
100
Number of Solution Stops
Number Solution Staps
(a) Distribution plot of MATH solution step num- bers.
(b) Distribution plot of NuminaMath-Cof solution step numbers.
Figure 11: Distribution plot of MATH and NuninaMath-CoT solution step numbers
Method
MATH (Full)
MATH (Hard Subset)
GRPO SFT+GRPO
43.6%
0.0%
45.9%
3.0%
BREAD
48.2%
6.0%
Table 3: Performance of LLaMA3.2-3B Instructt model the full MATH dataset and its hard subset (constructed following Figure 8).
typically not included in the evaluation benchmarks of DAPO and similar methods. For example we train and evaluate DAPO on the same difficult dataset used in Figure 8, under the same settings. We observed that DAPO shows little improvement. This aligns with its design: DAPO's dynamic sampling strategy tends to filter out hard questions rather than try to solve them, making less useful for the types of tasks target with BREAD. The results shown in Table 4.
Method
Accuracy
GRPO
4 %
SFT+GRPO
1 %
BREAD
14%
DAPO
5%
Table 4: BREAD compare to DAPO.
C
GRPO w/ Expert Trace Details and its Connection with RL with SFT Loss
The baseline GRPO w/ Expert Trace (GRPO-ET) generally follows the same procedure as the standard GRPO. Instead, it enforces one of the G rollouts as the expert trace. Therefore, the objective function is
(())OSEr()) = E (g.S.e) --
logl
min
clipp)) =.) Acs)) BDuu (TT ||
loyl
min (ra,(0)As,,,, clip (0), - + As..) 0Dr. DDK. (FTol
(3)
G 0s
where
(i.e(0) ne (e Oiee) (o, lq, oc.ce)
A, R, mean (R), Re)
ÅS Rs mean R2 [ Rs)
Rs)
( Rs)
(4)
24

---

All notations in Equations (3) and (4) are the same as [13], while S represents the expert trace, and all variables whose subscript contains S represent the corresponding variables.
Here, we can see a clear connection between GRPO w/ Expert Trace and GRPO (rollout number - 1) with an SFT loss. Suppose that we want to deploy GRPO while adding the SFT entropy loss to the standard GRPO loss, the objective function is
SPT
(g (g. >
losl
min (1)
-DDi (sra (Tol(/ee))
G
0
[osl Σ
log (S$ 9. see)
(5)
where
Ta Tu(u occce) (0)
mean 3 Rs)
A
(6)
(o, oicce)
std (H
| ) Ris)
There are 3 main difference between Equation (3) and Equation (5):
1. A coefficient for the expert trace loss Glosl
2. A KL. divergence term -K (TTr)
3. An Advantage term min (ra,(()) clip (r,,,(0), -,, ++) As.)
Suppose that Ti, + é, because Ås,ı O, min (rr: (- As.))
(0) Ås,t, which is exactly the entropy loss with a coefficient As, if we replace (oralorely otet) in (0))()) with the one hot embedding of the expert trace tokens. Therefore, GRPO w/ Expert Trace can not only have better training consistency, but can also assign different credits according to the token probability ratio between the old and the current policy.
Algorithm 3 GRPO w/ Expert Trace (GRPO-ET)
Require: Dataset D. current policy sampling number G SEXPERT_TRACER ns. G)
I: procedure GRPO_
step = 1, .... do 3: Sample a batch 2D from D
4: Update the old policy model Tald TT Sample G 1 outputs for G (-u(-) for each question Q D3
5:
6:
Combine the expert trace with the sampled outputs to construct [oo S
7:
Compute rewards { (ra) rs} for each sampled output o, and expert trace
8:
t- is generally because of solution correctness
For each o and expert trace compute Å,, for the f-th token of o, and S. for iteration = 1 T do
> Equation (4)
10:
11:
Update the policy model by maximizing the GRPO-ET objective end for end for
Equation (3)
12:
14:
return me
15:
end procedure
D Supporting Material for Section 2.1: Proofs and Experimental Details
D.I Experimental Details
Recall that we consider Markov chain with K + states (indexed by 0, 2, K) and (K +
1) x (K + 1) transition matrix. We assume that, for the expert/large model, (i j) transitions learnable in the transition matrix. However, for small/student model, not all j) transitions
25

---

are learnable and we denote the learnable state transitions by P [K 1] x [ + [K] {0, ..],
1] where
We first introduce the following definition of pretrained small model utilized in our experiments.
Definition (Pretrained Small Model) A pretrained small model is defined as a Markov model
M (S, P), where {0, 2......, K} is the state space and P IR (1Xx+11XKKKK (X)x) is the transition matrix. The model satisfies the following properties:
Let d 2 1 denote the maximum allowed jump between states. The of learnable state transitions P satisfies:
(i.j) P and only if - 5l d.
For some < 1/d, the transition probabilities satisfy:
O(de),
ifi= J. if. E P and + J. if (i.j) P.
P.y :- P(i j)
(
In the following, we describe the BREAD algorithm for the Markov model. Our algorithm is in line with the theory presented in Section 2.2 and incrementally learns solution by optimizing the local transition dynamics starting from the right end of the expert trace.
Algorithm 4 BREAD Markoy
Require: An SFT trajectory (.o. .rr) initial small model M([K), P), reward threshold Fimed. maximal trajectory length Tim
1: for episode T, l..... do
2: r--
3: while Pawedd do
4: Sample N trajectories from .M starting from state ou with maximal length Tixx
Update the current reward: Tnew
Update the transition matrix: P Paew
7: end while
8: end for
Experimental settings for Figure 3: We conduct experiments by fine-tume tie-ttunig a Markov chain model (ef. Definition 1) using three different methods: SFT, GRPO, and BREAD. To introduce additional randomness in our experiments, for each state i E [K]. we randomly select one of its connected states (e-g., among - d,...,;,4
d states) and assign it the highest transition probability of 1 (ce), instead of always assigning the highest probability to Pi - i). Due to symmetricity, this is not expected change the fundamental behavior of the algorithms but provides more variability. In both Figs. 3a and 3b, d = 2, I Tmax 2 K and SFT is infeasible with expert jump size of 3 as we set T' K /3. In Fig. 3a, fix K 30 and vary E (0.01, 0.025, 0.05). In contrast, in Fig. 3b, the e = 0.05 is remained unchanged and the number of states varies in K {30, 50, 1003. Each experiments is trained for 10000 iterations with 1000 trajectories sampled with each iteration.
D.2 Proofs in Section 2.2
D.2.1 Proof of Lemma 1
Since the expert model always jumps two or more whereas the student is restricted to a jump size of at most one, the SFT phase (specifically maximizing likelioe using expert trace) will have no impact on the student model. This is because the supervised loss, induced by the negative log- likelihood, is not optimizable beyond infinity due to () transition probabilities assigned between 3; x++1- To conclude the proof, we show that, no trajectory sampled during the RI. phase will receive a reward with high probability. Since the SFT phase has no impact on the base student model, the student still follows a symmetric random walk. Let (x:) denote this random walk which is sum of IID Rademacher variables. The classical concentration bounds on the hitting-time of random
walks state [7, 10]
(P3 x K for <
2e k2/21.
26

---

Thus, with access to N & /4/ samples, applying union bound, we guarantee that no trace generated by the student model receives reward.
D.2.2 Proof of Theorem
We first recall the following lemma on random walk hitting times.
Lemma (Lower bound on random walk hitting time, [10] If X, is a symmetric random walk initialized at 0, we have that Pemax X; K) 2 0.8K/vI
To proceed, the proof will be done inductively. Suppose for all rounds i < т, the followings hold:
Each round requires at most traces achieve success with probability at least e-
All sub-traces generated by the small model round i (starting from the expert's xT-;) that are stitched to S (trace constructed so far) is at most length L/T.
To complete the induction, consider the new round r and prove the two statements above. At round T, we start from State zリ-z and aim to reach a State within S,-1- This process will follow symmetric random walk initialized Ty--- By design, Ty-7+1 is within S,-1- As result, the trajectory length of the subtask is upper bounded by the time it takes for the random walk to reach from ØT-r to --- Consequently, the total trajectory length upper bounded by the all subtasks. Since each subtask is independent and identically distributed, we focus on analyzing single subtask with maximal trace length L/T. Recall from the assumptions in Section 2.2 that jer-r- 2T -T-r+1| & C K/T. /T"" To this aim, we apply Lemma 2 which ensures that, probability of
hitting within L/T trace length is least 0.8 -.
0.8
0.8 e 'LT
Since L 5c K²/KT
VL/T
probability failure for each trace is & 1/e. Over IID traces, probability of success at any round least 1 e-t. Aggregating these events, find that the probability of success over T rounds is at least 1 - Te- given t BREAD rollouts at each round.
27


Reviews:
[{"review_id": "goOt2mmkeA", "reviewer": "Reviewer_YNri", "summary": "This work introduces new GRPO-like algorithms for training Small Reasoning models. The authors leverage the observation that small models required longer traces to match larger models, hence, SFT might hurt performance. Moreover, because GRPO relies on having a good initialization from SFT, subsequent RL training can more likely fail.  Hence, the authors designed BREAD, an algorithm that leverages the expert demonstrations (reasoning traces) to adaptively choose how much of the expert trace to prepend to achieve a threshold success rate of the sampled group. This allows the model to obtain a kind of curriculum that improves RL training and bypasses the SFT training phase. The empirical evaluation successfully shows that their algorithm improves over all baselines.", "questions": "- I appreciate the toy model provided that serves to motivate the algorithm design. I’m mostly interested in understanding how much we can trust it. Where do you get the motivation of modelling the limited capacity of small models by limiting the learnable graphs in the Markov chain? \n- Can we apply BREAD for general LLM reasoning or does it only apply to SLMs? Do you expect that without the bottleneck of the model capacity, BREAD could still provide an advantage over GRPO?\n- As I understand it, the expert guidance is incorporated by modifying the search process. Effectively changing the policy \\pi that is being executed, doesn’t this introduce off-policy issues?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is very well written and provides a clear exposition of the motivation and design of BREAD;\n- The authors include a reasonable set of baselines, including naive variations of GRPO, and show that their algorithm improves over all of them.\n- The authors provide experiments that allows to further understand the impact of their algorithm on compute, sample efficiency and effect of the hardness of the tasks/questions\n\nWeaknesses:\n- The toy model builds a nice intuition for the problem motivation but it may be too limited. Though I understand this may not be the main contribution of this work.\n- In terms of the BREAD objective, I believe there might be a problem with off-policy. See questions.", "was_revised": true, "final_justification": "The authors addressed most of my concerns effectively and provided further insight about the method. \nOnly the off-policy issues of the algorithm remained unresolved but I believe they do not reduce the merit of the current method. Moreover, the authors have provided further baselines and evidence of the empirical performance of their method during the rebuttal period that strengthens their evaluation.", "rebuttal": "Thank you for your encouraging feedback on our manuscript. We’re grateful for your recognition of our approach and your constructive comments. Below, we respond to your questions and concerns. We would be grateful to address further questions during the discussion period.\n\n**Q1: What motivates modeling the limited capacity of small models by limiting the learnable graphs in the Markov chain?**\n\nA: Our motivation comes from three insights: (1) Recent studies[1] show that an L-times deeper LLM can simulate L chain-of-thought steps of a smaller model. This implies that for a student model with limited depth/capacity, a dense reasoning trace from a stronger expert must be broken down into shorter, more manageable segments to enable learning via SFT. (2) Compositional nature of reasoning: Reasoning tasks are inherently compositional and typically require multi-step chain-of-thought processes to arrive at a correct solution [2]. Small models often struggle to learn such long trajectories end-to-end. (3) Markov chains are more amenable to theoretical analysis while being representative of sequential/language data [3-7].\n\n**Q2: Can BREAD generalize beyond SLMs?**\n\nA: Yes, BREAD is a general method that is not limited by model size. We thank the reviewers for raising this point, and we are open to demonstrate BREAD’s scalability to larger models as a future direction. This highlights the method’s potential applicability across a wide range of settings, making it even more broadly useful.\n\nBREAD targets the fundamental limitations of SFT and RL. SFT often struggles to teach hard-to-imitate traces, and standard RL often suffers from sparse rewards. Although these problems are more noticeable in small models, high-capacity models can also struggle with similar difficulties, particularly when dealing with complex reasoning tasks where high-quality traces are hard to generate and the RL reward is sparse due to low probability of success. For example, Qwen2.5-7B-MATH, the accuracy drops from 13.3% to 6.67% on AIME from 15.1% to 5.6% on GPQA after SFT with S1K dataset. Qwen2.5-32B-Instruct achieves only 10% accuracy on AIME, which improves to 37% after fine-tuning on the S1K dataset. However, even at 37% accuracy, reward signals in RL remain sparse(19.5% of samples all 8 rollouts fail). BREAD addresses this by adapting problem difficulty through a curriculum learning that interpolates between expert (offline) traces and self-generated (online) rollouts. We therefore believe BREAD can help in large-model settings, particularly for tasks involving long reasoning chains or limited supervision. But because of the computational source constraint, it’s hard for us to show results on larger models. We will continue this as future work. \n\n**Q3: Does BREAD introduce off-policy learning issues?**\n\nA: The expert traces are used to guide the trajectory generation, not to replace or bypass the model’s policy. In detail, BREAD samples rollouts by interpolating between the expert traces and the self-generated traces to adapt difficulty. However, the policy gradient update is always computed using the log-probabilities of the model’s own trace(online part). This ensures that the learning signals remain on-policy, even though the rollout paths may be partially guided by the expert. Therefore, BREAD benefits from expert traces without breaking the assumptions of policy gradient methods.\n\n[1] Saunshi, Nikunj, et al. \"Reasoning with latent thoughts: On the power of looped transformers.\" arXiv preprint arXiv:2502.17416 (2025).\n\n[2] Wei, Jason, et al. \"Chain-of-thought prompting elicits reasoning in large language models.\" Advances in neural information processing systems 35 (2022): 24824-24837.\n\n[3] Zekri, Oussama, et al. \"Large language models as markov chains.\" arXiv preprint arXiv:2410.02724 (2024).\n\n[4] Makkuva, Ashok Vardhan, et al. \"Attention with markov: A curious case of single-layer transformers.\" The Thirteenth International Conference on Learning Representations. 2025.\n\n[5] Edelman, Ezra, et al. \"The evolution of statistical induction heads: In-context learning markov chains.\" Advances in neural information processing systems 37 (2024): 64273-64311.\n\n[6] ldiz, M. Emrullah, et al. \"From self-attention to markov models: Unveiling the dynamics of generative transformers.\" arXiv preprint arXiv:2402.13512 (2024).\n\n[7] Kim, Hyunsu, et al. \"Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo.\" arXiv preprint arXiv:2503.00699 (2025)."}, {"review_id": "ovLXkSFRyo", "reviewer": "Reviewer_vjLr", "summary": "This paper proposes BREAD: a method for improving GRPO training of small language models on hard reasoning problems using backplay.", "questions": "- Is there a correlation between a model's averaged pass@1 accuracy on a question and the amount of hint that needs to be shown before reaching the correct answer? Could this be leveraged to warm-start the binary search?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper is clear and easy to follow\n- The method demonstrates strong results for SLMs on hard problem sets\n\n**Weaknesses**:\n\n- BREAD is essentially an application of backplay [1,2] to reasoning problems. Despite this, no references on backplay are cited or discussed.\n- Evaluation is done only on the Qwen2.5 model series. A more thorough evaluation investigating what characteristics of a model necessitate backplay supervision would be interesting.\n\nMissing references:\n- [1] Salimans, Tim and Richard J. Chen. “Learning Montezuma's Revenge from a Single Demonstration.” ArXiv abs/1812.03381 (2018): n. pag.\n- [2] Resnick, Cinjon et al. “Backplay: \"Man muss immer umkehren\".” ArXiv abs/1807.06919 (2018): n. pag.", "was_revised": true, "final_justification": "While BREAD demonstrates some improvement in the reasoning abilities of small models, the improvement is not significant (>5%) and has limited novelty (drawing heavily from prior ideas in backplay).", "rebuttal": "Thank you for the detailed and constructive feedback. Here, we answer the questions and address the concerns. We would be grateful to address further questions during the discussion period.\n\n**W1: lack of references of backplay.**\n\nA: Thank you for pointing this out. We agree that BREAD shares conceptual similarities with Backplay [1,2]. Both methods aim to mitigate the challenge of sparse rewards by starting training closer to successful endpoints and gradually increasing difficulty. We will cite these works and include a discussion in the Related Work section.\n\nAlso, we would like to emphasize several key differences between BREAD and classical Backplay. While Backplay typically replays from fixed intermediate states and often requires delicate tuning, BREAD dynamically selects the hint ratio for each question during training, enabling adaptive difficulty adjustment without additional tuning overhead. This forms a curriculum tailored to each query, and offers significant computational efficiency.\n\nMoreover, unlike Backplay, BREAD does not require expert traces for all training samples. This is important as generating expert traces is often prohibitively expensive for reasoning tasks(e.g., requiring human annotation or large model inference). \n\n**W2: What characteristics of a model make BREAD necessary or particularly beneficial?**\n\nA: Thank you for this question. We provide a general discussion of this topic in Section 5 (Discussion and Limitations), and we will expand on it further in the revised version. \n\nBREAD is motivated by the observation that certain model characteristics make it challenging to learn complex reasoning behaviors through standard SFT or the model’s initialization has a very small likelihood of success so RL alone fails. In such cases, curriculum-based or backplay-style learning becomes particularly helpful. In particular, we believe BREAD is most beneficial under the following conditions:\n\n1. Limited Model Scale: If the base model is strong enough to generate at least one correct rollout per query, BREAD becomes identical to the vanilla GRPO. However, for smaller or weaker models that cannot do so reliably, BREAD offers a strict advantage. So SLMs, that struggle to imitate or generate correct rollouts, are a key beneficiary. \n\n2. Weak Prior Reasoning Ability: Models that lack training on structured reasoning tasks have difficulty in producing stable and correct reasoning traces during RL exploration. BREAD helps stabilize training by anchoring trajectories to expert traces. For instance, in our experiments, vanilla GRPO based on Qwen2.5-1.5B-Instruct shows limited improvement and is much worse than BREAD (Table 2). Whereas DeepScaleR-1.5B-Preview[5] achieves stronger results using DeepSeek-Distill-1.5b[3], which has the same architecture, but was obtained after careful supervised training. BREAD mitigates the need for this expensive supervised training stage. So BREAD is especially helpful when the base model starts from a weaker reasoning prior.\n\n3. Hint-Responsiveness: BREAD is most effective when the student model can utilize partial hints to guide its reasoning. However, in some extreme cases, a model may fail to benefit from expert traces even when provided in full, due to its inability to understand and follow complex traces. For instance, Llama‑3.2‑1B‑Instruct[6] achieves 6.8% accuracy on MATH‑500, and this improves marginally to 7.3% with hints. Since the model cannot meaningfully use the hints, BREAD provides limited benefit (achieves accuracy 8.9%) in such scenarios. \n\n**Q1: Can we use the model's average accuracy on a question to warm-start or guide the branching point (e.g., binary search) more efficiently?**\n\nA: Thank you for raising this excellent suggestion. We do expect a correlation: questions with high pass@1 accuracy likely require little or no hinting, while harder questions benefit from longer anchored prefixes (i.e., branching closer to the answer). Our algorithm design reflects this intuition. However, in our current implementation, we prioritize sample efficiency and we only do binary search for queries where the model fails to generate any correct rollout (pass@1 = 0). This helps avoid generating expensive expert traces for easier questions. We discuss how expensive it is to generate the trace in paragraph \"BREAD improves sample efficiency during training\". But we agree that incorporating accuracy to warm-start the binary search could improve efficiency if we want to binary search on queries with different accuracy, and we see this as a promising direction for future work.\n\n[1] Salimans, Tim, and Richard Chen. \"Learning montezuma's revenge from a single demonstration.\" arXiv preprint arXiv:1812.03381 (2018).\n\n[2] Resnick, Cinjon, et al. \"Backplay:\" man muss immer umkehren\".\" arXiv preprint arXiv:1807.06919 (2018).\n\n[3] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).\n\n[4] Liu, Aixin, et al. \"Deepseek-v3 technical report.\" arXiv preprint arXiv:2412.19437 (2024).\n\n[5] Luo, Michael, et al. DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL. Notion Blog, 2025, https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2.\n\n[6] Meta AI. Llama‑3.2‑1B‑Instruct. Meta Llama 3.2 model card, Hugging Face, 25 Sept. 2024."}, {"review_id": "dPfdCY3qQs", "reviewer": "Reviewer_mfDd", "summary": "This paper proposes a reinforcement learning (RL)-based approach to address the challenges of insufficient exploration by small models on difficult tasks and distributional shift caused by instruction tuning. The authors leverage partial expert demonstrations as starting points to guide model continuation and optimize exploration. Experimental results indicate that the proposed method improves the performance of reasoning tasks on the Qwen series models. However, the paper exhibits significant shortcomings in terms of methodological novelty, experimental design, and generalizability.", "questions": "Please address all the issues mentioned in the Weaknesses section as thoroughly as possible.", "limitations": "Some of the limitations faced by this paper have been discussed in the Weaknesses section.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 1, "strengths_and_weaknesses": "### Strengths\n1. The problem addressed in this paper is important. On challenging tasks, models often fail to explore correct solution paths, making training difficult.\n2. The paper is clearly written and easy to understand.\n\n### Weaknesses\n\n1. Lack of originality in the core idea: The novelty of the proposed method is questionable. The two challenges identified by the authors—difficulty in exploring correct paths on hard problems and distributional shift caused by SFT—are not unique to small models; these issues also exist for larger models.\nMoreover, the idea of using partial demonstrations as starting points to facilitate exploration has already been proposed in recent works, such as R^3 [1], which use expert demonstrations to construct curricula for better exploration. \nEarlier work can be traced back to OpenAI’s RL research [3]. R^3 also utilizes demonstration-based continuation and validates its effectiveness on mathematical tasks. The main differences between this paper and existing work appear to be the choice of RL algorithm (GRPO vs. PPO) and the focus on small models in this paper. Additionally, relevant prior work is not adequately cited. The authors should clarify the novelty of their proposed approach.\n\n2. Experiments are conducted only on the Qwen series models, which have been shown to exhibit the “performance improvement under random rewards” phenomenon. For example, Shao et al. [4] demonstrated that Qwen models show significantly greater gains under spurious rewards compared to other models, raising concerns about the generalizability of the results. The authors should include evaluations on additional small models to validate the universality of their method.\n\n3. The two core challenges identified by the authors do not appear to be specific to small models; larger models are also affected. I am curious about the effectiveness of the proposed approach on larger models.\n\n4. Lack of comparison with other methods: The experimental section does not include comparisons with other approaches aimed at improving SLM reasoning capabilities, such as R^3. The authors should compare their method against these baselines to better highlight its advantages.\n\nReferences:\n\n[1] Xi, Z., Chen, W., Hong, B., Jin, S., Zheng, R., He, W., Ding, Y., Liu, S., Guo, X., Wang, J., & Guo, H. (2024, July). Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning. In International Conference on Machine Learning (pp. 54030-54048). PMLR.\n\n[3] Florensa, Carlos, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. \"Reverse curriculum generation for reinforcement learning.\" In Conference on robot learning, pp. 482-495. PMLR, 2017.\n\n[4] Shao, Rulin, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du et al. \"Spurious Rewards: Rethinking Training Signals in RLVR.\" arXiv preprint arXiv:2506.10947 (2025).", "was_revised": true, "final_justification": "Some of my concerns have been addressed—for example, the authors have added comparisons with other similar methods and included results based on the Llama series.\n\nHowever, some of my questions remain unresolved. The authors also acknowledge that the two core findings of the paper are not unique to small models. While the proposed method is novel when applied to small models, there are still similar approaches in the broader landscape of larger models, which affects the originality of the work. I hope that, when resources permit in the future, the authors can further apply their proposed method to larger models to explore its advantages compared to existing methods in those settings.\n\nAdditionally, I do not agree with the authors' justification: “It is worth mentioning that the Qwen model family serves as a well-established testbed for mathematical reasoning tasks and methods, as evidenced by its exclusive use in multiple notable papers [6-10].” My reasons are as follows:\n\nDeepSeek-r1 has also released distilled versions based on other models, and did not solely rely on Qwen for experiments.\n\nRegarding the papers [7-10] cited by the authors, upon my thorough review, I found that none of them have been published in any conferences or journals (only one is reported on a Workshop); they are only available on Arxiv. This means that these papers have not undergone rigorous peer review, so the fact that they exclusively used the Qwen model family for experiments does not justify that it is reasonable to do so.\n\nThe existence of the Spurious Reward issue in the Qwen model family raises concerns about works that only use Qwen models for experiments, and this should be taken seriously.\n\nThe additional experimental results are promising. I will increase my score. However, I believe it is necessary for the authors to include these experiments in the revised version and to provide a more thorough justification for the differences between the proposed method and existing work. Otherwise, readers may be confused about the novelty of this paper.", "rebuttal": "Thank you for your detailed and constructive feedback. Below, we address the comments point by point. We would also be grateful to hear any additional feedback or questions.\n\n**W1.1 & W3: The issues BREAD target also exist for larger models.**\n\nA: Thank you for pointing this out. We agree that the two core issues identified in our paper are not exclusive to small models. However, this is not a weakness of the paper, instead, it highlights the method’s potential applicability across a wide range of settings, making it more broadly useful. Due to limited computational resources, we were not able to run experiments on large models. However, we are open to demonstrate BREAD’s scalability to larger models as a promising future direction.\n\nMeanwhile, we remark that these issues are more acute in small models. Also, BREAD benefits from a strong expert/teacher model to provide high-quality traces (human experts may not readily be available). For this reason, our experiments prioritize SLMs, where the impact of BREAD is most clearly observable. Overall, our focus on SLMs is not a limitation, but a significant contribution in itself for several reasons. SLMs are increasingly important for cost-sensitive deployments, such as on-device reasoning and agentic AI [5]. Most reasoning benchmarks show a large performance gap between SLMs and LLMs. Many pipelines that work well for large models fail to scale down to smaller ones. So, improving reasoning in SLMs remains a central open problem.\n\n**W1.2: Novelty concerns and comparison with suggested papers.**\n\nA: Thank you for pointing out connections to previous papers. We will cite the papers and include a discussion in our revised version. We will also add R^3 as a baseline. The results and the setting are provided in the response to W4.\n\nWe agree that BREAD shares conceptual similarities with [1,3]. All methods aim to mitigate the challenge of sparse rewards by starting training closer to successful endpoints.\n\nBut there are several key differences between BREAD and previous papers: \n\n(1) Dynamic vs. fixed hint: Previous works use a manually defined schedule to adjust the prefix (hint) ratio during training, which requires careful tuning. In contrast, BREAD dynamically selects the prefix length for each question at each iteration, using a binary search to automatically adapt the difficulty.\n\n(2) Use expert traces only when needed: Previous works require full expert traces for all training examples, whereas BREAD adaptively applies hints only when needed, significantly reducing the cost of expert trace generation by human experts or large models.\n\n(3) Cold-start capability: R^3 relies on an initial SFT warm-up phase before applying reverse RL, whereas BREAD can improve performance without requiring any prior SFT initialization. We also provide both empirical and theoretical results showing that SFT initialization can fail, and this phase can be expensive.\n\n(4) Theoretical grounding: Complementing our empirical results and previous works, we provide a Markov chain-based theoretical model to explain why small models could struggle with long reasoning traces, why GRPO without hints can perform poorly, and how BREAD mitigates the sparse reward issues.\n\nThese differences show the novelty and broader applicability of BREAD, especially in settings where expert traces are costly and model capacity is limited.\n\n**W2: Shao et al. raise concerns about the generalizability of the results evaluated on Qwen family models.**\n\nA: Thank you for the question. The main message in Shao et al. [4] is not contradictory with ours: They argue that pure RLVR cannot improve the models’ ability, but makes the latent ability in the base models show up. That’s exactly why we should utilize expert traces in BREAD so that new knowledge can be infused during RL training to truly improve the models’ reasoning ability. Notably, [4] states a maximum accuracy of 78.5% on MATH using Qwen2.5-7B, whereas our paper achieves 84.3% with BREAD using the smaller Qwen2.5-3B under the same number of iterations. So it also doesn’t contradict BREAD’s performance benefits, as BREAD goes strictly beyond RLVR with both spurious and true labels.\n\nAdditionally, we evaluate BREAD by training LLama3-1B on GSM8k dataset. BREAD achieves meaningful improvements over vanilla GRPO and noticeably outperforms SFT+GRPO, while also reducing the cost of expert trace generation by selectively using hints rather than full traces for every example.\n\n|GRPO|SFT+GRPO|BREAD|\n|-|-|-|\n|9.0%|36.6%|39.7%|\n\nIt is worth mentioning that the Qwen model family serves as a well-established testbed for mathematical reasoning tasks and methods, as evidenced by its exclusive use in multiple notable papers [6-10]. However, we are grateful for the reviewer's recommendation to use additional models and will provide additional evaluations in the final version.\n\n**W4: Lack of comparison with other methods.**\n\nA: Thank you for the suggestion. We ran additional experiments for R^3 as the baseline on MATH. For fair comparison, we deploy the same R^3 on GRPO and train for 300 steps in total for both methods.\n\nBREAD demonstrates clear improvements in both accuracy and sample efficiency. Unlike R^3, which requires full expert traces for every example, BREAD learns effectively using only a small fraction of expert traces. This is important because collecting full reasoning traces is prohibitively expensive. It requires either human labeling or generation by large models such as DeepSeek-R1, whose single forward pass already costs more FLOPs than 25 RL training steps with 8 rollouts. BREAD’s ability to reduce this cost while improving performance highlights its practical value.\n\n\n|Model|Method|Acc|Expert Trace Usage|\n|-|-|-|-|\nQwen-1.5B-Instruct|GRPO|59.0%|/|\nQwen-1.5B-Instruct|BREAD|78.8%|27.8%|\nQwen-1.5B-Instruct|R^3| 70.2%|100%|\nQwen-3B-Instruct|GRPO|68.1%|/|\nQwen-3B-Instruct|BREAD|84.3%|19.1%|\nQwen-3B-Instruct|R^3|74.6%|100%|\n\n\n[1] Xi, Zhiheng, et al. \"Training large language models for reasoning through reverse curriculum reinforcement learning.\" arXiv preprint arXiv:2402.05808 (2024).\n\n[3] Florensa, Carlos, et al. \"Reverse curriculum generation for reinforcement learning.\" Conference on robot learning. PMLR, 2017.\n\n[4] Shao, Rulin, et al. \"Spurious rewards: Rethinking training signals in rlvr.\" arXiv preprint arXiv:2506.10947 (2025).\n\n[5] Belcak, Peter, et al. \"Small Language Models are the Future of Agentic AI.\" arXiv preprint arXiv:2506.02153 (2025).\n\n[6] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).\n\n[7] Cetin, Edoardo, Tianyu Zhao, and Yujin Tang. \"Reinforcement Learning Teachers of Test Time Scaling.\" arXiv preprint arXiv:2506.08388 (2025).\n\n[8] Dong, Qingxiu, et al. \"Reinforcement Pre-Training.\" arXiv preprint arXiv:2506.08007 (2025).\n\n[9] Muennighoff, Niklas, et al. \"s1: Simple test-time scaling.\" arXiv preprint arXiv:2501.19393 (2025).\n\n[10] Aggarwal, Pranjal, and Sean Welleck. \"L1: Controlling how long a reasoning model thinks with reinforcement learning.\" arXiv preprint arXiv:2503.04697 (2025)."}, {"review_id": "TPPDMj9Ztp", "reviewer": "Reviewer_iFM6", "summary": "This paper introduces Branched Rollouts from Expert Anchors, a novel approach that combines supervised fine-tuning and reinforcement learning to enhance reasoning in small language models. The authors identify the limitations of traditional SFT + RL training, particularly for SLMs struggling to learn from complex expert traces and sparse rewards. BREAD addresses these issues by incorporating partial expert guidance and branched rollouts, allowing for more efficient learning. The method reduces the need for extensive expert traces, accelerates training, and improves model performance, especially on challenging tasks.", "questions": "1. Have the authors tested whether models with 7B parameters or larger, when using the same SFT and RL methods as described in this paper to improve model COT capabilities, experience the issues of insufficient RL rewards or performance degradation caused by the SFT-only method mentioned in the paper?\n\n2. Does the length of the COT reasoning path affect the training algorithm proposed in this paper? Typically, how many reasoning steps are involved in the expert traces constructed in this paper?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The BREAD framework effectively bridges the gap between supervised fine-tuning and reinforcement learning , addressing the issues of sparse rewards and weak initialization in small language models , ensuring more efficient learning.\n2. BREAD demonstrates substantial improvements in training efficiency, reducing the number of required rollouts and thus lowering computational costs while achieving higher accuracy than traditional GRPO and SFT methods.\n3. BREAD outperforms existing methods, including GRPO and SFT+RL, on challenging tasks, especially those that are unsolvable using standard SFT+RL, by providing expert guidance and adaptive curriculum learning.\n4. The paper offers a thorough theoretical analysis, formalizing the limitations of SFT+RL and justifying the design of BREAD, supported by mathematical insights into how expert traces can enhance model performance.\n5. Experimental results consistently show that BREAD not only achieves superior accuracy but also accelerates training time.\n\nWeaknesses:\n\n1. Section 3.2 on the Deficiencies of SFT-only: a) The authors have not provided sufficient evidence to demonstrate that the performance degradation observed with SFT-only is due to the model's insufficient learning capability rather than biases present in the SFT data. b) The authors have concluded that SFT-only leads to performance degradation based on a limited set of SFT datasets (only two) and a narrow range of SFT methods (two methods). I believe these experiments are insufficient to robustly prove the claim that SFT-only causes performance degradation.\n\n2. Bias in the Datasets Used in Table 2: The datasets used in Table 2 are biased, as they are all mathematics-related, which may not adequately demonstrate the generalizability of the proposed method. Validating the method on other reasoning datasets would provide a better assessment of its broader applicability.", "was_revised": true, "final_justification": "After considering the rebuttal and discussions, I believe the authors have made a strong effort to address the key concerns, and my overall impression of the paper has improved. I will keep the score.", "rebuttal": "We appreciate your positive evaluation of the manuscript and your thoughtful suggestions. Thank you for acknowledging our approach—please find our responses to your questions and concerns below. We would be grateful to address further questions during the discussion period.\n\n**W1: More evidence to conclude that performance degradation with SFT-only arises from model limitations.**\n\nA: In addition to empirical results on real datasets, we provide theoretical insights to illustrate the limitations of SFT as presented in Section 3.1. It is known that an L times deeper LLM can internally simulate L chain-of-thought steps of a smaller LLM [1]. So the expert model can generate a dense reasoning trace which necessitates an L times longer simplified trace for the student model to digest via SFT. Translating this to a toy student-expert model over Markov chains, we show how the student model cannot benefit at all from the expert model’s trace. As for more empirical evidence, some recent papers also have similar observations [2,3]. \n\nWe also would like to clarify that our intent in Section 3.2 is not to claim that SFT-only universally leads to performance degradation. Rather, we aim to show that the limitations of SFT can cause SFT + RL strategies to fail, particularly in complex reasoning tasks involving long traces. We highlight a specific failure mode that can arise even when using widely used SFT datasets, and our results serve to motivate the need for the proposed method BREAD. We also acknowledge the reviewer's concern and will clarify that the *strict degradation* SFT in Table 1 may stem from the distributional difference between S1K and MATH/GPQA datasets. Note that, we pick the s1k dataset because the s1 paper [8] shows that SFT with s1k achieves very strong results on the 32B model. So it is naturally surprising that our experiments demonstrate that the same training data can in fact hurt the performance of small models. We speculate that this is because the s1k dataset is too hard for small models (besides the distribution shift) but more digestible for large models. Furthermore, NuminaMath evals on Table 2 indicate that SFT can hurt performance even on the same distribution. We will carefully clarify these points in the final revision.\n\n**W2: All datasets are all mathematics-related.**\n\nA: Thank you for the thoughtful feedback. Mathematical reasoning tasks are widely used in prior works [3-9] as a standard benchmark for evaluating reasoning capabilities. Also, these tasks strike a critical balance, they are sufficiently challenging, yet solvable for SLMs with the correct training strategy. This makes them particularly well-suited for studying GRPO variants like BREAD.\n\nAdditionally, the availability of reliable expert traces in this domain allows us to conduct meaningful and controlled experiments, which may not be feasible in other domains.\n\nTherefore, we believe that math benchmarks serve as a strong and appropriate testbed for analyzing the effectiveness of our method. But we really appreciate the reviewer’s concern and would welcome suggestions on alternative reasoning datasets. We are open to expanding our evaluation in future versions.\n\n**Q1: Did you observe sparse rewards in RL or performance degradation from SFT with larger models?**\n\nA: Yes. The spare reward issue can happen even with Qwen2.5-32B-Instruct, it achieves only 10% accuracy on AIME, which improves to 37% after fine-tuning on the S1K dataset. However, even at 37% accuracy, reward signals in RL remain sparse(19.5% of samples all 8 rollouts fail). BREAD targets the fundamental limitations of SFT and RL that are not constrained by the model size. Although these problems are more noticeable in small models, high-capacity models can also struggle with similar difficulties, particularly when dealing with complex reasoning tasks where high-quality traces are hard to generate and the RL reward is sparse due to low probability of success.\n\n**Q2: How many reasoning steps are involved in the expert traces constructed in this paper?**\n\nA: We provide statistics on reasoning trace length in Figure 10 (Appendix B.3: Episode Details). For the MATH dataset, the expert traces have an average of 8.42 steps, with a minimum of 1 and a maximum of 228, and a standard deviation of 11.46. For the NuminaMath-CoT dataset, the average is 7.08 steps, with a minimum of 1, maximum of 97, and a standard deviation of 5.40. While reasoning trace length varies widely, BREAD is designed to be robust to this variability. BREAD dynamically creates a learning curriculum by adaptively inserting short expert prefixes/hints. \n\n[1] Saunshi, Nikunj, et al. \"Reasoning with latent thoughts: On the power of looped transformers.\" arXiv preprint arXiv:2502.17416 (2025).\n\n[2] Chu, Tianzhe, et al. \"Sft memorizes, rl generalizes: A comparative study of foundation model post-training.\" arXiv preprint arXiv:2501.17161 (2025).\n\n[3] Li, Yuetai, et al. \"Small models struggle to learn from strong reasoners.\" arXiv preprint arXiv:2502.12143 (2025).\n\n[4] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025).\n\n[5] Qu, Yuxiao, et al. \"Optimizing test-time compute via meta reinforcement fine-tuning.\" arXiv preprint arXiv:2503.07572 (2025).\n\n[6] Cetin, Edoardo, Tianyu Zhao, and Yujin Tang. \"Reinforcement Learning Teachers of Test Time Scaling.\" arXiv preprint arXiv:2506.08388 (2025).\n\n[7] Dong, Qingxiu, et al. \"Reinforcement Pre-Training.\" arXiv preprint arXiv:2506.08007 (2025).\n\n[8] Muennighoff, Niklas, et al. \"s1: Simple test-time scaling.\" arXiv preprint arXiv:2501.19393 (2025).\n\n[9] Aggarwal, Pranjal, and Sean Welleck. \"L1: Controlling how long a reasoning model thinks with reinforcement learning.\" arXiv preprint arXiv:2503.04697 (2025)."}, {"review_id": "0dNWZzaWZA", "reviewer": "Reviewer_KHRX", "summary": "The paper explores reasoning with (relatively) small language models. The authors propose BREAD, a method that combines RL and SFT via a simple strategy: if the policy model cannot solve the given question, it is given a hint via a partial expert trajectory on this prompt. More precisely, BREAD does a binary search to figure out the length of the partial trajectory that leads to the student model to achieve pass rates within a pre-defined range. The authors show that this simple procedure outperforms combinations of supervised finetuning (SFT) and reinforcement learning (GRPO).", "questions": "**Q1.** When you do policy updates, do you train on all tokens, including the expert tokens? Or only the completion tokens generated by the student? Is it true that the method is no longer a policy gradient method, as the trajectories are off-policy?\n\n**Q2.** Do I understand correctly that all plots in Fig 6 and Fig 7 are test pass rates, where the test set is separate from the train set, and you are not training on the test set?\n\n**Q3.** Same question for Table 2: for these results, do you use any transcripts on MATH and Numina? What are the questions that you distill on?\n\n**Q4.** Do I understand correctly that in your SFT experiments you use a much lower amount of SFT compared to the industry methods such as DeepSeek-Distill? What is the size of the \"full\" dataset in your experiments?", "limitations": "Limitations adequately addressed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths \n\n**S1.** The paper is clear and easy to follow.\n\n**S2.** The proposed method is very simple and makes intuitive sense. It is easy to implement, and it has a very clear intuition. It can probably also be generalized to other kinds of \"hints\" beyond partial trajectories.\n\n**S3.** Section 3.1 presents a toy model where the improvement from BREAD can be proved theoretically.\n\n**S4.** The empirical evaluation shows consistent improvements relative to other combinations of SFT and GRPO.\n\n# Weaknesses\n\n**W1.** If I understand correctly, the policy is updated by training on all the tokens, including the expert tokens which were not generated by the policy. If this is correct, I believe that the policy update is no longer a policy gradient update, since the trajectories can be significantly off-policy. \n\n**W2 (minor).** Please say explicitly in the Policy Update section of Section 3 that the update (1) is the standard GRPO update. Otherwise, it is not clear where this objective comes from.\n\n**W3.** The presentation often does not provide all the details of the results. For example, in Section 3.2 RL-only paragraph, it is unclear what model is used, what dataset, etc. Without these details, the information on how many problems are unsolved in each batch seems pointless. In the SFT-only section, it is stated that the SFT is done on \"1000 difficult questions\". What are those questions? Also, it is unclear what are S1K and S1K-1.1. \n\n**W4.** Similarly, it is unclear what exactly is shown in Figure 5b. It is said that RL is using 40% hints, but then there is a full range of values for the \"\"Hint ratio\" in the plot, and there is an improvement for 0% hint ratio. Please clarify what exactly is shown.\n\n**W5.** In an ideal world, it would be great to see a comparison with GRPO and possibly newer methods like DAPO [1] in a standardized setting such that would be possible to directly compare to existing results in the literature (e.g. Qwen2.5-32B on AIME). I realize that this may be computationally challenging, and possibly 32B is already not a \"small\" LM. I note that the authors provide a comparison to DeepSeek-Distill in Table 2.\n\n[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale", "was_revised": true, "final_justification": "After reading the rebuttal and the other reviews, I maintain my score and recommend accepting the paper.", "rebuttal": "Thank you for your kind and constructive feedback. We’re pleased that you found our approach valuable and have addressed your questions and concerns point by point below. We would be grateful to address further questions during the discussion period.\n\n**W1: Does BREAD introduce off-policy learning issues?**\n\nThe expert traces are used to guide the trajectory generation, not to replace or bypass the model’s policy. In detail, BREAD samples rollouts by interpolating between the expert traces and the self-generated traces to adapt difficulty. However, the policy gradient update is always computed using the log-probabilities of the model’s own trace(online part). This ensures that the learning signals remain on-policy, even though the rollout paths may be partially guided by the expert. Therefore, BREAD benefits from expert traces without breaking the assumptions of policy gradient methods.\n\n**W2: Please say explicitly in the Policy Update section of Section 3 that the update (1) is the standard GRPO update.**\n\nA: Thank you for the suggestion. We will clarify in the next version.\n\n**W3,4: The presentation often does not provide all the details of the results.**\n\nA: Thank you for the suggestion. We will make this clearer in the next version. For the details of results in Section 3.2, the information is provided in the caption of Table 1: we use Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct as our base SLMs, and the S1K and S1K-1.1 datasets are generated following the procedure in [2]. Additional experimental details can be found in Appendix A. \n\nThe hint ratio in the x-axis in Figure 5b refers to the proportion of hint tokens appended to the query during **inference**, which is distinct from the 40% hint ratio (orange squares) used during RL **training**. A higher hint ratio means more guidance for the model. Currently, this setting is explained in Lines 180–185 and 198–201. Our results show that conditioning RL training on partially hinted questions not only improves performance on hinted evaluation queries (e.g., those with 40% hints), but also leads to improved accuracy on fully unhinted queries.  This highlights the model’s ability to generalize from partial short traces to full long traces.\n\n**W5: Compare to other methods in a standardized setting.**\n\nA:  Thank you for the suggestion. We agree that standardized comparisons are valuable. In our work, we include vanilla GRPO and DeepSeek-Distill as baselines to enable such comparisons. Regarding recent works, we would like to highlight that our focus is on challenging tasks and SLMs where existing methods like DAPO[1] are not specifically designed to perform well. So, these settings are typically not included in the evaluation benchmarks of DAPO and similar methods. For example, DAPO use Qwen2.5-32B instead of smaller models on those challenging tasks,\n\nFor example, we train and evaluate DAPO on the same difficult dataset used in Figure 7, under the same settings. We observed that DAPO shows little improvement. This aligns with its design: DAPO’s dynamic sampling strategy tends to filter out hard questions rather than try to solve them, making it less useful for the types of tasks we target with BREAD.\n\n|Method|Acc|\n|-|-|\nGRPO|4%|\nSFT+GRPO|1%|\nBREAD|14%|\nDAPO|5% |\n\n**Q1,2: Are we training on the test set?**\n\nA: No, we do not train on the test set. In Figure 6, we evaluate using the official test subset provided by the Math/Numina-CoT benchmark. In Figure 7, we create an 80/20 train/test split from the dataset to ensure a clear separation between training and evaluation data. And in Table 2, we train on the training set and evaluate using the test subset provided by the Math/Numina-CoT benchmark. \n\n**Q3: Training data for SFT**\n\nA: Yes, we use a much lower amount of data. For SFT(full), we use the training subset of MATH(about 7500 samples)/NuminaMath-CoT to do SFT. In contrast, DeepSeek-Distill uses a much larger dataset. They use approximately 800k samples, including both reasoning and non-reasoning samples. \n\n[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale\n\n[2] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025."}]

Author rebuttal:

Meta-review: The authors propose a method named BREAD to densify the reward signal when learning to improve SLM's reasoning with the help of a larger expert model. The method is well motivated. When self-generated traces of SLM fail, BREAD adaptively inserts short expert prefixes/hints and hence densify the reward signal. A toy example illustrates and justifies the idea well, and the experimental results demonstrate the effectiveness of BREAD. Some issues (e.g. experiments on models other than Qwen; off-policy vs on-policy) were raised by the reviewers, and most of them were well addressed during the rebuttal. The reviewers are all positive on the contribution of the paper. Overall, it is a well-motivated, simple&effective, well-justified solution to the investigated problem. I hence recommend acceptance. In the meantime, I would suggest the authors to further strength the paper based on the rebuttal.
Decision: Accept (poster)